App Considerations

1. App Form Data
2. Shared App Form Data
3. App Process Data
4. Plugins
5. Integration with External Systems

App Form Data

This section is applicable to you if you intend to migrate the form data stored in the server, otherwise, you can move on to the next section.

As part of application building best practises, your app table name should have is own prefix.

For example, in the screenshot below. The "Expenses Claim" app is making use of "j_expense" as the prefix in its forms' table naming.

Figure 1: Form tables

Migrating app form data out would be quite straightforward since it resides in its own set of tables. Depending on the database systems that you are using, you can make use the appropriate tools/clients to export the data out. 

Figure 2: Export App

When exporting app, we can export its dependant plugins as well as form data too. By default, only the app design is exported out.

However, it is important to note that although there's ability to export form data in Export App feature, the intent of this feature is to export/import setup data, not production data. If the resultant export package size file size is too big, importing it later on will result in error / request time out issues.

Shared App Form Data

This section is applicable to you if you intend to migrate the form data stored in the server, otherwise, you can move on to the next section.

Referring to Figure 1 above, we can make use of the Form builder > Advanced Tools > Table > Table Usage In Other Apps to check if tables are being shared with other apps.

If there are shared form data tables, then we will need to assess how we can separate the data or continue to access the shared data upon separation. There are few things to consider based on the assumption that the data is vital to the app that we are migrating out:-

1. How are we making use of the shared tables? As a lookup or we write into them too?
2. Can we continue to provide access from the original server?
3. At which layer should the access be created? Joget app server or DB server level?
4. If DB level, we can consider making use of JDBC related plugins(i.e. JDBC Options Binder / JDBC Form Binder / JDBC Datalist Binder).
5. If Joget app level, we can consider making use of JSON API or provision new ones using API Builder where we can exert more controls in API management.

App Process Data

This section is applicable to you if your app is making use of processes (Process Builder), otherwise, you can move on to the next section.

Due to the complexity and structure of the data of the workflow engine (shk* tables), as of now, we do not recommend migrating process data out in parts.

This is because data of process instances are stored in set of tables shared by all other apps in the same Joget server. Process data contains not just instances from the latest version of the app that we are migrating out, it may also contain all the process instance data of the same app but from earlier versions. The current server and target server may also contain different set of versions of the same app too. Therefore, migrating the data out in parts will likely to affect the normal functionality of the workflow engine.

We would recommend to leave the process instances' data behind. For process instances that are still in transit (open.running), here's the approach that we recommend.

1. Migrate the form data to the new server.
2. Compile a list of process instances that are still running. It should also contain the pending activity and latest set of workflow variable data.
3. Start a new process instance with reference to the form record ID and set of workflow variable data, and resumes at the last pending activity.

This way, end users should be able to seamlessly continue with their process flow.

Having explained about the complexity of migrating workflow engine data out, there is an alternative.

In case you find it important to preserve the audit trail of the part / completed process data, you can enable Process Data Collector at the app level. Process related data of the app would be written into set of tables (app_report_* tables) that can be easily digested for analytical use such as by using SLA Report Menu. However, this plugin would only work on new process data as they are being executed after it is enabled, and not retrospectively on past process data.

Plugins

This section is applicable to you if your app is making use plugins, otherwise, you can move on to the next section.

Referring to Figure 1 above, plugins that are used by the app can be exported alongside with the app itself. You should check if the new / target server has the same plugins but of different versions and check for its compatibilty.

Integration with External Systems

This section is applicable to you if your app is making external calls / relying on external systems, otherwise, you can move on to the next section.

Make a list of all external touch points that your app is using.

Figure 3: Service Calls in App's Performance

The server calls in app's performance tab is a good place to refresh your memory as it will be able to help to list down calls being made outside of the Joget server over a period of time.

For each of them, do check for permission in the external systems to see if you need to whitelist / permit your new server in order to continue to function normally. If the new server is located in environment that is significantly different with current server (i.e geographically far away), it would be good to test its network / throughput to ensure that the app in the new server would be able to function within expected response times and not get bogged down by slow external integrations.

Platform Considerations

Joget Server Version

When we are migrating the app out from one to another, it is good to maintain the exact same version from the original server and the one you are migrating to. This is so that we do not introduce too much of moving variables, in case the app we migrated to, fails to function as expected.

Bear in mind that if you intend to upgrade your Joget platform, do it first at the environment where the staging server is, or upgrade the staging server where the said apps have been confirmed working before by following the Upgrade Guide. This way, the only change made is to upgrade the Joget platform version. It would be easier to find out cause of any problems as there are not many variables changed.

User Management

Typically, we will continue to serve the same set of users in the new server as we scale out.

If you are making use of an identity management system (i.e. LDAP), then it should be relatively easy to simply point to the same server.

Otherwise, you may need to duplicate the existing user directory date (tables with prefix of dir_ in the database) to the new server.

Migration of Running Process Instances

Before We Begin

Before attempting anything, always perform a backup in any (app or database) servers that you will be working on. This is so that there's always something to fallback on, just in case!

Here's the checklist that we would recommend before the migration.

1. The target server has the same app, same process(es), and have been tested to be working end to end (with all integrations, and plugins fitted in). This is so that discrepency in app/process design can be ruled out in case of any problem arised later on.
2. The target server has the same user management, so that the same user assigned, in current server, can be found in target server. This is so that problem that may arise when assigning an user can be ruled out.

Performing the Migration

Copy Form Data

Migrate non-process data over first by copying them over. Remember to copy and not to move, so that the original server remains fully functional and intact.

1. Make a list of process instances that are still running.
2. Copy these data over to the target server.
   1. Form Data from the database
   2. File Upload from wflow/app_formuploads
3. Now that form data and its uploaded files are available in the target server, we can migrate process instances over to the target server.

To verify the data that you have copied them over, make use of CRUD Menu or List Menu in the app to browse through the data imported in.

Migrate Process Instances

Then, perform the following steps, starting with just 1 process instance. The steps below can be performed using the utility app available for download in this article.

1. Based on the list compiled in step 1, start new process instance in the new server using the same user with reference to the form record ID. Here's a sample of a process instance that is still running.
   
   Figure 4: Process Instance View
   
   In Figure 4 above, there are 3 vital information we need, and use them to call JSON API in the target server by using JSON API#web/json/workflow/process/start/(*:processDefId). We will also need to copy any workflow variable data over to the new instance too. We can pass on var_* parameters to set the workflow variables' values accordingly.
   
   At this point of time, the new process instance would be waiting at its very first activity.
   
   IMPORTANT NOTE: If you have tools that run right at the start of the process flow, this may cause manipulation of form data state. Take note of potential changes that may occur and devise a way to preserve original data/state.
2. Once a new process instance is started, note down the process instance ID generated, and we will then need to make use of JSON API#web/json/monitoring/activity/start/(*:processId)/(*:activityDefId) to resume the activity where it supposed to be, in the original server.
   
   
   Figure 5: Process Instance View Highligting Open Activity Instances
   
   NOTE: Depending on how many pending activities that we may have in the specific process instance, we may need to make multiple calls to this JSON API. Take note that in the first of such call, we will need to set the "abortCurrent" parameter value to "true" so that it aborts the first default pending activity instance. Subsequently, set "abortCurrent" parameter to "false".
   
   IMPORTANT NOTE: The assignee of the resumed activity instance may not be correct if its participant mapping relies on performer of past activities. This is because past activities' data is NOT available in the target server.
3. In the target server, attempt to continue with the process assignment, as usual, through Datalist Inbox Menu or Inbox Menu in your app, ideally, to the very end, to validate that it is working as expected.

Once we have validated and refined the steps to migrate a process instance, we can then move on to migrate more process instances over.

Utility App for Migration of Running Process Instances

We have created an utility app to perform the 3 steps listed above. It has been tested using MySQL database and on Joget DX 7.

Please refer to the diagram below to understand on how to use it.

Figure 6: Utility App to Migrate Process Instances

1. Install this app in the current server. Publish the app.
2. Go to setup menu to key in parameters required of the target server.
   
   Figure 7: Utility App Setup Screen
   
   
3. At the target server, login as admin, navigate to System Settings > General Settings > API Domain / IP Whitelist and key in the domain / IP of the current / old server. This is so that the target server can accept JSON API calls from current server. We have set up the app.
4. Go to List of Running Process to see the list of running process instances. Select and click Migrate to start migration.
   IMPORTANT NOTE: If your app has User Notification Plugin or similar notification plugin enabled, or other process tool that will immediately run after process starts, it may be a good idea not to have them enabled while migration is taking place.
5. Observe current server log files and target server's Running Processes to verify its execution status / result.
   An entry will be added to Logs menu if it is successful.
   
   Figure 8: Utility App Logs
   
   In the List of Running Process menu, the status will change to "Migrated" with process instance ID furnished based on the data from logs.
   
   Figure 9: Utility App List of Running Process with New Process Instance Information
   
   

APP_migrateProcess-2-20210518184056.jwa

Note: This migration utility app is tested on Joget v6 Enterprise 6.0.31 and Joget DX Enterprise 7.0.16 and with process migrated into a Joget server runing Joget DX 7.0.16.