You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 9 Next »

This article provides a tutorial on deploying, running and scaling Joget on Azure Kubernetes Service (AKS). AKS is a managed Kubernetes service offered by Azure.

If you are not familiar with Kubernetes, refer to Joget on Kubernetes for a quick introduction.

Deploy Joget on Azure Kubernetes Service

1. Create Kubernetes cluster in AKS

This guide will go through creation process with the Azure portal, if you want to create a cluster through Azure CLI please refer to the article Azure CLI.

From the Azure portal, go to the Kubernetes services then Create a Kubernetes cluster.

In the Basics page, choose the Subscription, Resource Group and input the Kubernetes cluster name. Adjust the other configuration settings as desired, or leave as default.

In the Node pools tab, you can configure to add node pools into the cluster. Read on multiple node pools in AKS. For this guide, we will use a single node configuration.

For other tab options - Access, Networking, Integrations, Advanced and Tags, you can leave the default options or make adjustments/changes as necessary. After that, you can click on the Review + create and deploy the Kubernetes cluster. 


When the resource has completed their deployment, you can then connect to the cluster (read here) using Azure CLI/Azure Cloud Shell.

2.Deploy MySQL Database

Once we have a running cluster, you will need to deploy a database to be used by the Joget platform. You can pretty much follow the same method of deploying MySQL DB as in the Joget Kubernetes page.

  • Create persistent storage using PersistentVolume and PersistentVolumeClaim
kubectl apply -f https://k8s.io/examples/application/mysql/mysql-pv.yaml
  • Deploy the MySQL image
kubectl apply -f https://k8s.io/examples/application/mysql/mysql-deployment.yaml
  • Inspect the deployment
kubectl describe deployment mysql
kubectl get pods -l app=mysql
kubectl describe pvc mysql-pv-claim

You need to modify the original yaml files for production usage (eg. using different version of MySQL image and setting up secret instead of plain password in the yaml).

3.Deploy shared storage in AKS

If you are running a multiple node Kubernetes cluster, you will need to allocate shared persistent storage with read write access by multiple nodes. In Azure, you can set up Azure NFS volume to be used in the Azure Kubernetes cluster. Refer to the official documentation here for detailed info and steps. You can also read more on other options for storage in Azure Kubernetes here.

  • Create an Azure Ubuntu VM at the same Virtual Network as the AKS cluster.
  • Setup NFS server into the VM.

From the link, you can use this script to set up the NFS server (edit the variables as necessary especially the AKS_SUBNET).

#!/bin/bash
# This script should be executed on Linux Ubuntu Virtual Machine

EXPORT_DIRECTORY=${1:-/export/data}
DATA_DIRECTORY=${2:-/data}
AKS_SUBNET=${3:-*}

echo "Updating packages"
apt-get -y update

echo "Installing NFS kernel server"

apt-get -y install nfs-kernel-server

echo "Making data directory ${DATA_DIRECTORY}"
mkdir -p ${DATA_DIRECTORY}

echo "Making new directory to be exported and linked to data directory: ${EXPORT_DIRECTORY}"
mkdir -p ${EXPORT_DIRECTORY}

echo "Mount binding ${DATA_DIRECTORY} to ${EXPORT_DIRECTORY}"
mount --bind ${DATA_DIRECTORY} ${EXPORT_DIRECTORY}

echo "Giving 777 permissions to ${EXPORT_DIRECTORY} directory"
chmod 777 ${EXPORT_DIRECTORY}

parentdir="$(dirname "$EXPORT_DIRECTORY")"
echo "Giving 777 permissions to parent: ${parentdir} directory"
chmod 777 $parentdir

echo "Appending bound directories into fstab"
echo "${DATA_DIRECTORY}    ${EXPORT_DIRECTORY}   none    bind  0  0" >> /etc/fstab

echo "Appending localhost and Kubernetes subnet address ${AKS_SUBNET} to exports configuration file"
echo "/export        ${AKS_SUBNET}(rw,async,insecure,fsid=0,crossmnt,no_subtree_check)" >> /etc/exports
echo "/export        localhost(rw,async,insecure,fsid=0,crossmnt,no_subtree_check)" >> /etc/exports

nohup service nfs-kernel-server restart

After the NFS server has been set up, you can then create the PersistentVolume and PersistentVolumeClaim.

Example azurenfsstorage.yaml;

apiVersion: v1
kind: PersistentVolume
metadata:
  name: aks-nfs
  labels:
    type: nfs
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: NFS_INTERNAL_IP
    path: NFS_EXPORT_FILE_PATH
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: aks-nfs
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: ""
  resources:
    requests:
      storage: 1Gi
  selector:
    matchLabels:
      type: nfs

Replace the values for NFS_INTERNAL_IP, NFS_NAME and NFS_EXPORT_FILE_PATH with the actual settings from your NFS Server.

kubectl apply -f azurenfsstorage.yaml

4.Deploy Joget DX

With the prerequisite database and persistent storage available, you can now deploy Joget. You can apply the example joget-dx7-tomcat9-aks.yaml file to deploy.

Example joget-dx7-tomcat9-aks.yaml;

apiVersion: apps/v1
kind: Deployment
metadata:
  name: joget-dx7-tomcat9
  labels:
    app: joget-dx7-tomcat9
spec:
  replicas: 1
  selector:
    matchLabels:
      app: joget-dx7-tomcat9
  template:
    metadata:
      labels:
        app: joget-dx7-tomcat9
    spec:
      initContainers:
       - name: init-volume
         image: busybox:1.28
         command: ['sh', '-c', 'chmod -f -R g+w /opt/joget/wflow; exit 0']
         volumeMounts:
           - name: joget-dx7-tomcat9-volume
             mountPath: "/opt/joget/wflow"
      volumes:
        - name: joget-dx7-tomcat9-volume
          persistentVolumeClaim:
            claimName: aks-nfs
      securityContext:
        runAsUser: 1000
        fsGroup: 0
      containers:
        - name: joget-dx7-tomcat9
          image: jogetworkflow/joget-dx7-tomcat9:latest
          ports:
            - containerPort: 8080
            - containerPort: 9080
          volumeMounts:
            - name: joget-dx7-tomcat9-volume
              mountPath: /opt/joget/wflow
          env:
            - name: KUBERNETES_NAMESPACE
              valueFrom:
                fieldRef:
                    fieldPath: metadata.namespace
---
apiVersion: v1
kind: Service
metadata:
  name: joget-dx7-tomcat9
  labels:
    app: joget-dx7-tomcat9
spec:
  ports:
  - name: http
    port: 8080
    targetPort: 8080
  - name: https
    port: 9080
    targetPort: 9080  
  selector:
    app: joget-dx7-tomcat9
  type: ClusterIP
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: joget-dx7-tomcat9-clusterrolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: view
subjects:
  - kind: ServiceAccount
    name: default
    namespace: default

You can then check the deployment progress from the Azure portal. (Or use kubectl commands eg. kubectl get deployment joget-dx7-tomcat9)

5.Deploy Ingress for external connections

You can then expose the application for external access through Ingress. You can read more regarding Ingress in Kubernetes here. In this guide, we will use Nginx Ingress Controller as an example to access Joget.

Deploy Nginx Ingress Controller to AKS cluster

You can refer to the AKS documentation regarding creating ingress-nginx and also the nginx-ingress document.

There are 2 known methods of deploying the Nginx Ingress Controller to the AKS cluster;

  1. Deploy through Helm
  2. Use yaml file from the Nginx Ingress Controller Github

Install using Helm

Using Azure CLI/Cloud shell, set up the Helm for Nginx Ingress

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

helm install ingress-nginx ingress-nginx/ingress-nginx --create-namespace --namespace nginx-ingress

Install using yaml file

You can use kubectl apply command.

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.4.0/deploy/static/provider/cloud/deploy.yaml

After the Ingress Controller has been deployed, we can then apply the Ingress yaml so that we can access the Joget application externally.

Example joget-ingress.yaml;

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: joget-dx7-tomcat9-ingress
  annotations:
    nginx.ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
  ingressClassName: nginx
  rules:
    - http:
        paths:
          - path: /jw
            pathType: Prefix
            backend:
                service:
                  name: joget-dx7-tomcat9
                  port:
                    number: 8080

After the Ingress deployment is completed, you can get the public IP from the Kubernetes resources > Services and Ingresses pane in the Azure portal (eg. http://<external-ip>/jw).

Setup Database

To complete the Joget deployment, you need to perform a one-time Database Setup. Key in the MySQL service name and the Database Username and Password. Click on Save.



Once the setup is complete, click on Done and you will be brought to the Joget App Center.

6.Setup cert-manager for TLS termination

Before starting the TLS setup, you need to set ‘enable-underscores-in-headers’ as true for Ingress by using configmap.

Example ingress-configmap.yaml;

apiVersion: v1
kind: ConfigMap
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
data:
  enable-underscores-in-headers: "true"
  allow-snippet-annotations: "true"

Update the Ingress configuration with kubectl apply -f ingress-configmap.yaml

Install cert-manager into the cluster

Similar to installing the ingress controller, you can install cert-manager either through Helm or through yaml file. Refer to the cert-manager official documentation here for detail. For this guide we will be using the yaml file method.

**Before going further with these steps, make sure that you have set up DNS to the public IP of the ingress that has been generated by AKS earlier.

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.10.0/cert-manager.yaml

Configure Let’s Encrypt issuer

Example stagingissuer.yaml file;

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    # The ACME server URL
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: [update email here]
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-staging
    # Enable the HTTP-01 challenge provider
    solvers:
      - http01:
          ingress:
            class: nginx
kubectl apply -f stagingissuer.yaml

You can check on the status of the issuer resource after you have deployed it

kubectl describe issuer letsencrypt-staging

Deploy/Update the Ingress with TLS configuration

As we have previously deploy the Ingress without TLS configuration, we can update the Ingress yaml file to include the TLS configuration.

Example Ingress yaml with TLS;

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: joget-dx7-tomcat9-ingress
  annotations:
    nginx.ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    cert-manager.io/cluster-issuer: "letsencrypt-staging"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - exampledomain.com
    secretName : aks-jogetworkflow
  rules:
    - host: exampledomain.com
      http:
        paths:
          - path: /jw
            pathType: Prefix
            backend:
              service:
                name: joget-dx7-tomcat9
                port:
                  number: 9080

This staging procedure is to ensure that the certificate is generated correctly before we setup the Issuer with Let’s Encrypt production.

kubectl get certificate
[ ~/jogetaks ]$ kubectl get certificate
NAME                READY   SECRET              AGE
aks-jogetworkflow   True    aks-jogetworkflow   30s
kubectl describe certificate aks-jogetworkflow


If the certificate is generated correctly then we can set up the production Issuer.

Example productionissuer.yaml file;

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: [update email here]
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-prod
    # Enable the HTTP-01 challenge provider
    solvers:
    - http01:
        ingress:
          class:  nginx

Update the ingress yaml file with the production annotation.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: joget-dx7-tomcat9-ingress
  annotations:
    nginx.ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - exampledomain.com
    secretName : aks-jogetworkflow
  rules:
    - host: exampledomain.com
    - http:
        paths:
          - path: /jw
            pathType: Prefix
            backend:
              service:
                name: joget-dx7-tomcat9
                port:
                  number: 9080

After applying the updated ingress yaml, you need to delete the previous secret so that the new certificate can be generated for the production.

kubectl delete secret aks-jogetworkflow

Then run back the describe command to check on the cert status

kubectl describe certificate aks-jogetworkflow

After the new certificate has been issued, you can then access the Joget domain with https to ensure that everything is working properly.

7.Scale Deployment

While you can set the nodes or pods to autoscale in AKS (read here), you can also scale the number of nodes or pods manually. To scale the number of pods running Joget, you can use the kubectl command.

kubectl scale –replicas=3 deployment/joget-dx7-tomcat9

Adjust the replica number as you desired and the desired number of pods will initialize and startup.

As for the node, you can scale the node count of the node pool from the Azure portal. Go to the Cluster in the Kubernetes service (in this guide example jogetakscluster) > Settings > Node pools. Select the node pool and then click on the Scale node pool. Choose Manual as the Scale method and input the desired node count (maximum available resource is based on the VM size that you have chosen).

8.Additional Note :- Configure Joget in AKS with Azure Database for MySQL

This additional note is to guide on how to configure Joget in AKS connecting to the Azure Database for MySQL. It is based on the Azure guide here which has been modified to use the Joget application. This guide uses the Azure portal as to assist in visual guidance.

Assuming that a resource group has been created, then from the Azure portal, go to the Virtual network services then Create virtual network and also the subnets for the MySQL and AKS resources.

Based on the Microsoft recommendations of using Azure CNI to setup the configuration of AKS and Azure DB. You can read more on the AKS networking best practises here.

Then add the IP address space for the Virtual network.

After that create 2 subnets for the MySQL resource and also the AKS cluster.

We can then create the Virtual network resource.

8.1 Deploy Azure MySQL Flexible Server

Search for resource Azure Database for MySQL flexible servers. Then click on Create → Flexible server.

In the Basics tab, we configured as below since we are testing the resource. Modify as needed (also note in the screenshot the MySQL version is 5.7, we have tested with version 8 also).

Then in the Networking tab, we need configure the resource to use the Virtual network that we created earlier.

For Security and Tags tabs we can leave as default or make changes as necessary. After done we can create the resource.

For testing purpose, after the MySQL resource has been created, we turned off the require_secure_transport parameter. This is so that we will be able to intialise through the Joget setup page. Should you need this parameter to be enabled, you can then edit the app_datasource-<profile>.properties file. Example of the workflowUrl parameter with the require_secure_transport parameter turned on;

workflowUrl=jdbc\:mysql\://<azuredburlhere>\:3306/jwdb?characterEncoding\=UTF-8&useSSL\=true&allowPublicKeyRetrieval\=true


8.2 Deploy AKS Cluster and Joget

As to deploying the AKS cluster and Joget itself, the steps are similar as above in this KB page. The only different part is when setting up the AKS cluster, in the Networking tab, we need to specify to use Azure CNI and associate the virtual network and subnet that we have created earlier.

After AKS and Joget have been deployed, we will be able to do the DB setup on Joget.

  • No labels