Traceability is one of the important aspects of DevOps and it is vital in order to find the bottlenecks inside a pipeline. It helps consumers to take informed decisions on where is the bottleneck inside a pipeline so as to take the necessary corrective actions on the same. Traceability is one of the unique features of Insights due to the way data is stored inside Neo4j. Co-Relation between the different data collected is done by the engine based on the relations defined inside Insights. Co-Relation between the data is possible only if the various tools are integrated inside the pipeline and Insights does not help in the tools integration. This section explains how to configure the co-relation functionality inside Insights and what are the various aspects related to the same.
One of the same dashboard which is based on the co-relations is show in the below figure :
Sample Implementation of Correlation : Traceability Dashboard
Prerequisite for establishing co-relation:
- To establishing correlation two tools should have at least one common field value.
- All the common fields and mandatory fields to be indexed in Neo4j database.
server-config.json structure
| JSON Attribute | Description |
|---|---|
| correlationWindow | Denotes maxmimum time for a node to picked up by co-relation engine. 48 - the most optimal time. Eg: 48 |
| correlationFrequency | Denotes minimum time for a node to be picked up by co-relation engine. 3 - the most optimal time. Eg: 3 |
| batchSize | Denotes the batch size of the message. Eg: 2000 |
co-relation.json structure
| JSON Atttribute | Description |
|---|---|
| destination | Denotes the destination tool. |
| source | Denotes the source tool. |
| toolName | Denotes the tool name of source and destination. Eg. JIRA, GIT, JENKINS, SONAR |
| fields | Denotes common field between the source and destination tool Eg. jiraKeys, key, scmCommitId, commitId |
| relationName | Denotes relationship Name which will appear as relationship label in the Neo4j Eg. FROM_JIRA_TO_GIT, FROM_GIT_TO_JENKINS |
Steps to establish Co-Relaton:
Step 1: Stop all the running Agents
- All the agents running to be stopped.
Step 2: Stop the PlatformEngine
- Check if all the queues in the RabbitMq is empty and stop the Platform Engine.
Step 3: Changes in the Server-Config file
- Open server-config.json file and add the following at the end of the file.
"correlations" : {
"correlationWindow" : 48,
"correlationFrequency" : 3 ,
"batchSize" : 2000
}
Step 4: Create co-relation.json file
The co-relation file is key file to build relationship. The source value of the key “fields” should contain same value as of the destination. The location of the co-relation.json file should be same as the Server-config.json file namely INSIGHTS_HOME/.Insights/ co-relation.json.
JIRA -> GIT
Jira → Git relationship is only possible if the commit message has the Jirakey.
In the co-relation.json file DO NOT CHANGE, key :“fields” value: “jiraKeys” ,Key:”fields” value “key ”
NEO4J the git property name should be “message”.
[
{
"destination": {
"toolName": "**toolname**",
"fields": [
"**fieldname**"
]
},
"source": {
"toolName" : "**toolname**",
"fields": [
"**fieldname**"
]
},
"relationName" : "**relationshipname**"
}
]
Sample co-relation JSON file
[
{
"destination": {
"toolName": "JENKINS",
"fields": [
"scmCommitId"
]
},
"source": {
"toolName" : "GIT",
"fields": [
"commitId"
]
},
"relationName" : "FROM_GIT_TO_JENKINS"
},
{
"destination": {
"toolName": "GIT",
"fields": [
"jiraKeys"
]
},
"source": {
"toolName" : "JIRA",
"fields": [
"key"
]
},
"relationName" : "FROM_JIRA_TO_GIT"
},
{
"destination": {
"toolName": "HP",
"fields": [
"jiraKey"
]
},
"source": {
"toolName" : "JIRA",
"fields": [
"key"
]
},
"relationName" : "FROM_JIRA_TO_ALM"
}
]
Step 5: Indexing in Neo4j
All the fields required for the correlation needs to be indexed in Neo4j which includes all the fields mentioned in the co-relation.json file and correlation fields in label DATA - uuid, toolName, correlationTime, maxCorrelationTime, inSightsTime, inSightsTimeX. jiraKeys, jiraKeyProcessed.
Syntax to index the field in as follows:
create index on TOOLNAME(fieldname)
Mandatory fields
Label- DATA:uuid, toolName, correlationTime, maxCorrelationTime, inSightsTime, inSightsTimeX. jiraKeys, jiraKeyProcessed and all the fields in the co-relation.json file.
Example:
create index on :scm(jiraKeys)
create index on :JIRA(key)
create index on :DATA(uuid)
create index on : JENKINS(scmCommitId)
create index on : JENKINS(buildNumber)
Step 6: Start latest version of engine.
- Start the engine.
- The new relationship will be created in the Neo4j in few seconds.
Step 7: Start all the Agents
- Once all the expected relationships are created in neo4j, start all the agents.
Top Section