| Anchor | ||||
|---|---|---|---|---|
|
Traceability is one of the important aspects of DevOps and it is vital in order to find the bottlenecks inside a pipeline. It helps consumers to take informed decisions on where is the bottleneck inside a pipeline so as to take the necessary corrective actions on the same. Traceability is one of the unique features of Insights due to the way data is stored inside Neo4j. Co-Relation between the different data collected is done by the engine based on the relations defined inside Insights. Co-Relation between the data is possible only if the various tools are integrated inside the pipeline and Insights does not help in the tools integration. This section explains how to configure the co-relation functionality inside Insights and what are the various aspects related to the same.
One of the same dashboard which is based on the co-relations is show in the below figure :
Sample Implementation of Correlation : Traceability Dashboard
The above Dashboard basically depicts the following
For a single user story in Jira with Key IS -10
- It has 4 commits in Bitbucket
- It has 4 Continuous Integration Jobs in Jenkins which resulted in One Sonarqube executions.
- It has 3 Rundeck deployments and had 49 test case executions and 2 Load and Performance runs
The data for the above mentioned dashboard is stored in Neo4j as depicted in the below mentioned screenshot. The query spawns across several nodes connected by six relationships as shown below
For the Correlation to be working inside Insights following are the steps which should be followed.
Prerequisite for establishing co-relation:
- The DevOps Pipeline should be established and there should be tools integration between a minimum of two tools (For eg: Jira and Git Integrated and developers mentioning the user story ID in their Git Commit Message)
- The Nodes which are getting correlated in Neo4j should have a common field value.
- These fields should be indexed inside Neo4j to ensure optimal performance. (For Configuring Indexes in Neo4j please refer to the Neo4j Documentation https://neo4j.com/docs/cypher-manual/3.5/schema/index/#query-schema-index-introduction)
- Define required intervals on which the correlation module within the engine should be performed (Changes to be made inside the server-config.json)
- Define the co-relation using the Insights UI and get it stored in Databse.
The details of configuration of the JSON file is given in the below section :
server-config.json structure
| JSON Attribute | Description |
|---|---|
| correlationWindow | Denotes maximum time (in hours) for a node to picked up by co-relation engine. 48 - the most optimal time. Please refrain from changing it, since this will increase the load on engine and Neo4j Eg: 48 |
| correlationFrequency | Denotes minimum time (in hours) for a node to be picked up by co-relation engine. 3 - the most optimal time. Please refrain from changing it since this will increase the load on engine and Neo4j Eg: 3 |
| batchSize | Denotes the batch size of the message. Eg: 2000 |
Step 1: Stop all the running Agents
- All the agents running to be stopped.
Step 2: Stop the PlatformEngine
- Check if all the queues in the RabbitMq is empty and stop the Platform Engine. This is to ensure that there is no more data to be consumed by the engine.
Step 3: Changes in the Server-Config file
- Correlation by default is disabled inside Insights. So this will have to be enabled for the relationship to be created.
- Open server-config.json file and add the following at the end of the file.
| Panel |
|---|
"correlations" : { "correlationWindow" : 48, "correlationFrequency" : 3 , "batchSize" : 2000 } |
Step 4: Do’s and Don’t’s to build Correlation.
| Info |
|---|
To Create co-relation from Insights UI please refer below mentioned link: The below section talks in detail about the correlation_configuration table and is meant for advanced users who do not want to use the User Interface. It is strongly recommended to create co-relations from the User Interface. |
The source value of the key “fields” should contain same value as of the destination.
| Note | ||
|---|---|---|
| ||
Jira → Git relationship is only possible if the commit message has the Jirakey. In the correlation_configuration table DO NOT CHANGE, key :“fields” value: “jiraKeys” ,Key:”fields” value “key ” NEO4J the git property name should be “message”. |
correlation_configuration structure
| Panel | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
|
Step 5: Indexing in Neo4j
All the fields required for the correlation needs to be indexed in Neo4j which includes all the fields mentioned in the co-relation.json file and correlation fields in label DATA - uuid, toolName, correlationTime, maxCorrelationTime, inSightsTime, inSightsTimeX. jiraKeys, jiraKeyProcessed.
Syntax to index the field in as follows:
create index on TOOLNAME(fieldname)
| Note | ||
|---|---|---|
| ||
Label- DATA:uuid, toolName, correlationTime, maxCorrelationTime, inSightsTime, inSightsTimeX. jiraKeys, jiraKeyProcessed and all the fields mentioned in while creating the co-relation.json filerelation from UI. |
Example:
create index on :scm(jiraKeys)
create index on :JIRA(key)
create index on :DATA(uuid)
create index on : JENKINS(scmCommitId)
create index on : JENKINS(buildNumber)
Step 6: Start latest version of engine.
- Start the engine.
- The configured relationships will be created in the Neo4j. The time would be based on the number of nodes available for the engine to process.
- The newly created relationship along with the necessary properties should show up something like the below mentioned diagram inside neo4j.
Step 7: Start all the Agents
- Once all the expected relationships are created in neo4j, start all the agents.
...