Anchor | ||||
---|---|---|---|---|
|
Traceability is one of the important aspects of DevOps and it is vital in order to find the bottlenecks inside a pipeline. It helps consumers to take informed decisions on where is the bottleneck inside a pipeline so as to take the necessary corrective actions on the same. Traceability is one of the unique features of Insights due to the way data is stored inside Neo4j. Co-Relation between the different data collected is done by the engine based on the relations defined inside Insights. Co-Relation between the data is possible only if the various tools are integrated inside the pipeline and Insights does not help in the tools integration. This section explains how to configure the co-relation functionality inside Insights and what are the various aspects related to the same.
One of the same dashboard which is based on the co-relations is show in the below figure :
Sample Implementation of Correlation : Traceability Dashboard
The above Dashboard basically depicts the Pipeline as following
For a single user story in Jira epic in Jira with Key IS -10 PS-5
- It has 21 issues
- It has 4 82 commits in Bitbucketin Git
- It has 4 64 Continuous Integration Jobs in Jenkins which resulted in One Sonarqube executions.
- It has 3 Rundeck deployments and had 49 test case executions and 2 Load and Performance runs
- 53 Code quality executions,
- 312 artifacts generated in Nexus
- 50 Rundeck deployments
User can click on individual tile to drill down further. For example clicking on tile with issueKey : PS-77 will bring out pipeline related to it. Below summary changes as user drill down on specific tiles.
Dashboard can bring various textural inferences (Summary) like below but not limited to
- Bifurcation of Jira issues in Epic, Stories, Defects, tasks, subtask etc.
- Total number of contribution developers
- Build success and failure count
- Deployment success and failure
- Code quality parameters
- Artifacts details
Along with inferences, most importantly it shows average handover time between various tools. If some of the handover time is out of threshold user can take actions based on inference to improve it.
The Neo4j Cypher query output in Graph database looks like below
For the Correlation to be working inside Insights following are the steps which should be followed.
Prerequisite for establishing co-relation:
- The DevOps Pipeline should be established and there should be tools integration between a minimum of two tools (For eg: Jira and Git Integrated and developers mentioning the user story ID in their Git Commit Message)
- The Nodes which are getting correlated in Neo4j should have a common field value.
- These fields should be indexed inside Neo4j to ensure optimal performance. (For Configuring Indexes in Neo4j please refer to the Neo4j Documentation https://neo4j.com/docs/cypher-manual/3.5/schema/index/#query-schema-index-introduction)
- Define required intervals on which the correlation module within the engine should be performed (Changes to be made inside the server-config.json)
- Define the co-relation using the Insights UI and get it stored in Databse.
The details of configuration of the JSON file is given in the below section :
server-config.json structure
JSON Attribute | Description |
---|---|
correlationWindow | Denotes maximum time (in hours) for a node to picked up by co-relation engine. 48 - the most optimal time. Please refrain from changing it, since this will increase the load on engine and Neo4j Eg: 48 |
correlationFrequency | Denotes minimum time (in hours) for a node to be picked up by co-relation engine. 3 - the most optimal time. Please refrain from changing it since this will increase the load on engine and Neo4j Eg: 3 |
batchSize | Denotes the batch size of the message. Eg: 2000 |
Step 1:
Stop all the running Agents- All the agents running to be stopped.
Step 2: Stop the PlatformEngine
- Check if all the queues in the RabbitMq is empty and stop the Platform Engine. This is to ensure that there is no more data to be consumed by the engine.
Changes in the Server-Config file
- Correlation by default is disabled inside Insights. So this will have to be enabled for the relationship to be created.
- Open server-config.json file and add the following at the end of the file.
Panel |
---|
"correlations" : { "correlationWindow" : 48, "correlationFrequency" : 3 , "batchSize" : 2000 } |
Step
42: Do’s and Don’t’s to build Correlation.
Info |
---|
To Create co-relation from Insights UI please refer below mentioned link: The below section talks in detail about the correlation_configuration table and is meant for advanced users who do not want to use the User Interface. It is strongly recommended to create co-relations from the User Interface. |
The source value of the key “fields” should contain same value as of the destination.
Note | ||
---|---|---|
| ||
Jira → Git relationship is only possible if the commit message has the Jirakey. In the correlation_configuration table DO NOT CHANGE, key :“fields” value: “jiraKeys” ,Key:”fields” value “key ” NEO4J the git property name should be “message”. |
correlation_configuration structure
Panel | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|
|
Step
53: Indexing in Neo4j
All the fields required for the correlation needs to be indexed in Neo4j which includes all the fields mentioned in the co-relation.json file correlation_configuration table and correlation fields in label DATA - uuid, toolName, correlationTime, maxCorrelationTime, inSightsTime, inSightsTimeX. jiraKeys, jiraKeyProcessed.
Syntax to index the field in as follows:
create index on TOOLNAME(fieldname)
Note | ||
---|---|---|
| ||
Label- DATA:uuid, toolName, correlationTime, maxCorrelationTime, inSightsTime, inSightsTimeX. jiraKeys, jiraKeyProcessed and all the fields mentioned in while creating the co-relation.json filerelation from UI. |
Example:
create index on :scm(jiraKeys)
create index on :JIRA(key)
create index on :DATA(uuid)
create index on : JENKINS(scmCommitId)
create index on : JENKINS(buildNumber)
Step
64: Start latest version of engine.
- Start the engine.
- The configured relationships will be created in the Neo4j. The time would be based on the number of nodes available for the engine to process.
- The newly created relationship along with the necessary properties should show up something like the below mentioned diagram inside neo4j.