...

Fundamentals

Please refer below section on workflow framework basics.

Overview

Insights has growing need of a customizable framework, which would help running offline tasks with in a defined order. Workflow framework is fully configurable, extensible, and scalable. Workflow has independent design and any component can make use of this feature by registering itself to workflow and schedule jobs. Workflow as a whole comprises of below entities that together manages the workflow framework.

Entities

• Workflow Tasks: In workflow, a work is divided into smaller dependent tasks. In another words a task is a small unit of given work.

• Rabbit MQ Channels: Each workflow task has its own rabbit MQ channel and each task uses these channels for intra-task communication.

  • Sample MQ Message Format : {"executionId":1602843384824,"workflowId":"Report_1602842568","currentTaskId":373,"nextTaskId":-1,"sequence":1,"isWorkflowTaskRetry":false}

• Workflow Task Sequence: Each given task has a unique sequence number which ensures that all tasks are executed sequentially. When a task completes its execution then it notifies its next task using MQ channel.

• Workflow Lifecycle Stages:

  • NOT_STARTED: Default status when any new workflow is created.

  • IN_PROGRESS: Status is changed to IN_PROGRESS when task is currently being executed.

  • COMPLETED: Status is changed to COMPLETED when all the tasks of the workflow are completed its execution without any error and exception.

  • ERROR: Status is changed to ERROR when any workflow task fails to complete its execution due to error or exception.

  • ABORTED: Any workflow when it is retrying and at a certain point of time it exceeds the maxretrycount then status of that workflow is changed to ABORTED.

  • RESTART: When any workflow is in ABORTED state then there is flexibility to RESTART that workflow again after fixing possible errors.

  • TASK_INITIALIZE_ERROR: When any workflow task fails to publish message to its next task after successfully completing current task due to issue in MQ.

...

• Workflow Error Handling Phases: Workflow is said to be failed when it is unable to execute its task or goal which may be due to infrastructure failures like MQ, DB, and NEO4J down. In such scenarios workflow comes with built-in error handling mechanism. Workflow is capable for retry in below cases.

  • Workflow is failed while publishing message first time to MQ. In this scenario workflow do not have any execution history as it got failed in initial phase.

  • Workflow is failed in middle task where workflow has execution history available and in this case at least one of the tasks in ERROR state.

  • Workflow is failed with at least one task is completed but none of the tasks are in ERROR state.

• Workflow Thread-Pool: Workflow framework comes with multi-threading architecture where each workflow task is further broken into small chunks and each such chunk is executed on its own thread. Workflow uses refined custom thread-pool and its parameters are kept configurable as per need. Based on load and other factors such as CPU, memory and others user can easily optimize workflow thread-pool. Below are the list of parameters which user can configure.

  • corePoolSize: Number of thread that initially JVM creates for tasks , for example corePoolSize is 5 then JVM keeps creating threads up to 5 these means if 6^th task come then JVM creates 5 tasks and start them immediate and put 1 task in waiting queue

  • maximumPoolSize: maxPoolSize defines the maximum number of threads that can ever be created.

  • keepAliveTime: when the number of threads is greater than the core, this is the maximum time that idle threads will wait for new tasks before terminating.

  • waitingQueueSize: The queue to use for holding tasks before they are executed.

Prerequisites:

Overview:

Workflow framework comes with build-in reporting tool. To configure reports refer this prerequisites section.

Pre-Setup:

Server-Config : refer below example properties in server-config.json, user can change it according to need.

...

• Workflow Engine Configuration:

  • Create new folder “workflowjar” in INSIGHTS_HOME.

  • Place latest PlatfromReport and PlatfromWorkflow artifacts into workflowjar folder.

  • Run the OS specific workflow service script present in insights docroot/nexus repository to register a service.

  • Use OS specific start / stop scripts to start / stop workflow engine.

• Directory Structure For Report PDF:

  • Create folders exactly as follows and add required files in it.

    • C:\Insights\InSights_Windows\Server2\INSIGHTS_HOME\assessmentReportPdfTemplate.

    • C:\Insights\InSights_Windows\Server2\INSIGHTS_HOME\assessmentReportPdfTemplate\reportTemplates.

    • C:\Insights\InSights_Windows\Server2\INSIGHTS_HOME\assessmentReportPdfTemplate\reportTemplates\REPORT_SONAR_JENKINS_PROD(Name of the report template same as there in DB)

    • Add the required JSON files which is mentioned in the Report Template in the folder along with the HTML file and the CSS files.

Report Configuration:

Step By Step Report Configuration:

Navigate to Report Management and click "Report Management" under the Admin section.

...

1. Edit the report by clicking edit button as shown above. In edit section only modifying task, reoccurrence flag, mailing details is allowed.

2. Delete any report only if report doesn’t have any execution history. If it has execution history then report cannot be deleted.

3. Mark any NOT_STARTED / RESTART / TASK_INITILIZE _ERROR report to run immediate in turn which is picked by workflow immediate job scheduler.

4. Restart any ABORT state report to run again, report is turned into ABORT state when any report has crossed its maxretrycount. In such cases it is possible to select those reports in ABORT state and Restart them after fixing possible issues.

5. Download PDF button gets enabled for the report which completes its execution at least once, for the reports which are in ERROR state but has completed at least once execution successfully the PDF gets downloaded for that execution. For the reports those are completed their execution without any errors and in COMPLETED state, latest PDF gets downloaded for such reports.

6. Refresh the landing page.

7. Activate / Deactivate report whenever required.

8. Detailed execution history of the each task execution of selected report.

Workflow Frequencies:

Miscellaneous:

Workflow State / Flow Diagram:

...