Correlation

Traceability is one of the important aspects of DevOps and it is vital in order to find the bottlenecks inside a pipeline. It helps consumers to take informed decisions on where is the bottleneck inside a pipeline so as to take the necessary corrective actions on the same. Traceability is one of the unique features of Insights due to the way data is stored inside Neo4j. Co-Relation between the different data collected is done by the engine based on the relations defined inside Insights. Co-Relation between the data is possible only if the various tools are integrated inside the pipeline and Insights does not help in the tools integration. This section explains how to configure the co-relation functionality inside Insights and what are the various aspects related to the same.

One of the same dashboard which is based on the co-relations is show in the below figure :

Sample Implementation of Correlation : Traceability Dashboard

The above Dashboard basically depicts Pipeline as following

For a single epic in Jira with Key PS-5

It has 21 issues
It has 82 commits in Git
It has 64 Continuous Integration Jobs in Jenkins which resulted in One Sonarqube executions.
It has 53 Code quality executions,
312 artifacts generated in Nexus
50 Rundeck deployments

User can click on individual tile to drill down further. For example clicking on tile with issueKey : PS-77 will bring out pipeline related to it. Below summary changes as user drill down on specific tiles.

Dashboard can bring various textural inferences (Summary) like below but not limited to

Bifurcation of Jira issues in Epic, Stories, Defects, tasks, subtask etc.
Total number of contribution developers
Build success and failure count
Deployment success and failure
Code quality parameters
Artifacts details

Along with inferences, most importantly it shows average handover time between various tools. If some of the handover time is out of threshold user can take actions based on inference to improve it.

The Neo4j Cypher query output in Graph database looks like below

For the Correlation to be working inside Insights following are the steps which should be followed.

Prerequisite for establishing co-relation:

The DevOps Pipeline should be established and there should be tools integration between a minimum of two tools (For eg: Jira and Git Integrated and developers mentioning the user story ID in their Git Commit Message)
The Nodes which are getting correlated in Neo4j should have a common field value.
These fields should be indexed inside Neo4j to ensure optimal performance. (For Configuring Indexes in Neo4j please refer to the Neo4j Documentation https://neo4j.com/docs/cypher-manual/3.5/schema/index/#query-schema-index-introduction)
Define required intervals on which the correlation module within the engine should be performed (Changes to be made inside the server-config.json)
Define the co-relation using the Insights UI and get it stored in Databse.

The details of configuration of the JSON file is given in the below section :

server-config.json structure

JSON Attribute

Description

correlationWindow

Denotes maximum time (in hours) for a node to picked up by co-relation engine. 48 - the most optimal time. Please refrain from changing it, since this will increase the load on engine and Neo4j

Eg: 48

correlationFrequency

Denotes minimum time (in hours) for a node to be picked up by co-relation engine. 3 - the most optimal time. Please refrain from changing it since this will increase the load on engine and Neo4j

Eg: 3

batchSize

Denotes the batch size of the message.

Eg: 2000

Step 1: Changes in the Server-Config file

Correlation by default is disabled inside Insights. So this will have to be enabled for the relationship to be created.
Open server-config.json file and add the following at the end of the file.

"correlations" : {

"correlationWindow" : 48,

"correlationFrequency" : 3 ,

"batchSize" : 2000

}

Step 2: Do’s and Don’t’s to build Correlation.

To Create co-relation from Insights UI please refer below mentioned link:

To create co-relation.

The below section talks in detail about the correlation_configuration table. It is strongly recommended to create co-relations from the User Interface.

The source value of the key “fields” should contain same value as of the destination.

JIRA -> GIT

Jira → Git relationship is only possible if the commit message has the Jirakey.

In the correlation_configuration table DO NOT CHANGE, key :“fields” value: “jiraKeys” ,Key:”fields” value “key ”

NEO4J the git property name should be “message”.

correlation_configuration structure

Component	Significance
relation_name	This property depicts the correlation's name which is entered by the user ,and will be created in Neo4j.
Destination	destination_toolName - This property depicts the destination tool name which will be selected by the user from the drop down menu of the UI. destination_labelName - This property depicts the destination label name which will be selected by the user from the drop down menu of the UI. Properties are fetched from Neo4j of the destination tool , based on the label name & category. destination_toolcategory- The tool category is auto populated, when it's respective tool is selected.This parameter is also important to fetch the properties from Neo4j. destination_fields - This field depicts the properties selected for the destination tool to make the correlation.
Source	source_toolName - This property depicts the source tool name which will be selected by the user from the drop down menu of the UI. source_labelName - This property depicts the source label name which will be selected by the user from the drop down menu of the UI. Properties are fetched from Neo4j of the source tool , based on the label name & category. source_toolcategory- The tool category is auto populated, when it's respective tool is selected.This parameter is also important to fetch the properties from Neo4j. source_fields - This field depicts the properties selected for the source tool to make the correlation.
enable_correlation	This field depicts the status of the flag, whether the correlation is enabled/disabled. Enabling the correlation - The engine will pick the correlation for execution. Disabling the correlation - The engine will not pick the correlation for execution.

Step 3: Indexing in Neo4j

All the fields required for the correlation needs to be indexed in Neo4j which includes all the fields mentioned in the correlation_configuration table and correlation fields in label DATA - uuid, toolName, correlationTime, maxCorrelationTime, inSightsTime, inSightsTimeX. jiraKeys, jiraKeyProcessed.

Syntax to index the field in as follows:

create index on TOOLNAME(fieldname)

Mandatory fields

Label- DATA:uuid, toolName, correlationTime, maxCorrelationTime, inSightsTime, inSightsTimeX. jiraKeys, jiraKeyProcessed and all the fields mentioned in while creating the relation from UI.

Example:

create index on :scm(jiraKeys)

create index on :JIRA(key)

create index on :DATA(uuid)

create index on : JENKINS(scmCommitId)

create index on : JENKINS(buildNumber)

Step 4: Start latest version of engine.

Start the engine.
The configured relationships will be created in the Neo4j. The time would be based on the number of nodes available for the engine to process.
The newly created relationship along with the necessary properties should show up something like the below mentioned diagram inside neo4j.

Co-Relation Builder