Installing Insights Inferences (Optional Feature)

Insights Inferences is an optional feature. Please find below step- by - step instructions on how to Install Insights Inferences

Standard Hardware configurations for Insights Inference - Spark

Linux server - 64 bit - 16 GB RAM and 50 GB HDD

Required Machine configuration
Number of Servers	1
OS Family	Linux
OS type	64 bit
RAM	16 GB
HDD	50 GB
Core	8

Capacity Requirements may vary based on the amount of data that needs to be indexed, and performance requirements.

Insights Inference - Spark Installation Instructions (For Insights 5.1 and below releases)

Prerequisite

Install Below Packages:

- For now InSights Inference is available only for Linux.
- Java 1.8+ with JAVA_HOME set in environment variables(http://www.oracle.com/technetwork/java/javase/downloads/)
- Hadoop 2.7.1/wintutils(https://github.com/steveloughran/winutils/tree/master/hadoop-2.7.1/bin)
- Scala 2.11.11(https://www.scala-lang.org/download/2.11.11.html)
- Spark-2.2.0-bin-hadoop2.7.tgz(https://spark.apache.org/downloads.html)

Set Path and Environment Variables (Change default locations as applicable)

Run below in a New Terminal:

sudo -H gedit /etc/environment
Type your password
Edit the text file just opened:

HADOOP_HOME = user/hadoop2.7.1/winutils/bin
SPARK_HOME = user/Spark/
SCALA_HOME = user/Scala-2.11.11/

Please change default file locations as applicable.
Save it
Once saved, logout and login again.

To setup KPI Jobs, please refer to the documentation by clicking here.

Insights Inference - Neo4j KPI Job Configuration (For Insights 5.2 and above releases)

Learn to Add/Edit Insights Inference - Neo4j KPI on this page.

Insights Inference - Neo4j KPI Job JSON Structure

KPI Attribute	Description
kpiID	It is like a primary key of a Insights - Inference KPI. It should be unique, and have numerical value. It shall also be specified same when adding custom KPI message for UI.
expectedTrend	It is expected trend of the Insights Inference - KPI. Eg. UPWARDS / DOWNWARDS
name	It denotes the name of the Insights Inference - KPI.
action	It is the Mathematical Function that is imposed on a Insights Inference - KPI. It shall also be specified same when adding custom KPI message for UI. Eg. AVERAGE / COUNT / MINMAX
schedule	It is the job schedule of a Insights Inference - KPI. Eg. DAILY / WEEKLY / MONTHLY / YEARLY
vector	It denotes the DevOps vector of a tool, which is specified in the Insights Inference - KPI. Eg. BUILD, CODEQUALITY, DEFECTS, DEPLOYMENT etc,.
toolName	It is the name of the DevOps tool. Eg. JIRA, Jenkins, Sonar, Rundeck etc,.
isGroupBy	It is used for Aggregation if more than one instance is specified. Its value is Boolean. Eg. true / false
groupByFieldName	It is required when "isGroupBy" is set to true. It is used to specify the description for "groupByField", on which the Aggregation would be imposed.
groupByField	It is required when "isGroupBy" is set to true. It is used for Aggregation if more than one instance is specified. It's generally the name of the field on which Aggregation will be imposed.
averageField	It is required only when the KPI "action" is AVERAGE. Input the name of the field on which AVERAGE Mathematical Calculations must be imposed.
startTimeField	It indicates the Start time property/field name of a DevOps tool. Its input is a String value. It is mandatory if your Insights Inference - KPI's "action" is "AVERAGE".
endTimeField	It indicates the End time property/field name of a DevOps tool. Its input is a String value. It is mandatory if your Insights Inference - KPI's "action" is "AVERAGE".
resultOutPutType	It indicates the result type of the Insights Inference - KPI. For instance, AVERAGE type KPI could have result as Average Time or Average of Numbers. Use time when the type of output is Time Duration, and number when the type of output is just a Normal Number. Eg. time / number
isComparisionKpi	It is a Boolean value. It generally indicates whether or not the KPI result should have a comparison with previous KPI result. Eg. true / false
esquery	It is an Elasticsearch query to fetch the data and collect the results.
timeFormat	It is the standard time format of a region. Eg. YYYY-MM-DD HH:MM
esresource	It is the node/index at which the tools data is present in Elasticsearch. Eg. neo4j-index-node/tool-data
es-index-read-missing-as-empty	It is a Boolean value used to denote undefined nodes/indexes in Elasticsearch as empty. Eg. true / false
nextRun	It is the next scheduled date on which Insights - KPI Job Execution will be executed. It is represented as Epoch Time in milliseconds.
lastRunTime	It is the previous scheduled date on which Insights - KPI Job Execution was run. It is represented as Epoch Time in milliseconds.
isActive	It denotes whether or not the Insights Inference - KPI is in use or not. It is generally used to Enable / Disable the KPI. It accepts a Boolean value. Eg. true / false

Create Insights KPI in Neo4j

1.Download kpi.csv file from https://github.com/CognizantOneDevOps/Insights/tree/master/PlatformInsights
2.Check KPI list , and add or Modify KPI as per requirement , File must be in csv format
Note: Please do not change header
3.Go to neo4j browser http://localhost:7474/browser/
4.Run following csv upload command

LOAD CSV WITH HEADERS FROM 'file:///kpi.csv' AS row

create (:INFERENCE:CONFIG {

kpiID:row.kpiID,

expectedTrend:row.expectedTrend,

name:row.name,

action:row.action,

schedule:row.schedule,

vector:row.vector,

toolName:row.toolName,

isGroupBy:row.isGroupBy,

groupByFieldName:COALESCE(row.groupByFieldName,""),

groupByField:COALESCE(row.groupByField,""),

averageField:row.averageField,

startTimeField:row.startTimeField,

endTimeField:row.endTimeField,

resultOutPutType:row.resultOutPutType,

isComparisionKpi:row.isComparisionKpi,

neo4jQuery:row.neo4jQuery,

timeFormat:row.timeFormat,

neo4jLabel:row.neo4jLabel,

nextRun:row.nextRun,

lastRunTime:row.lastRunTime,

isActive:row.isActive

})

RETURN row

Now Check all configuration in Neo4j with lable INFERENCE:CONFIG using following query

MATCH (n:INFERENCE:CONFIG) RETURN n LIMIT 25

Create KPI using: Neo4j Query

Use following query to create new KPI in neo4j ,Please replace dummy value before running it.

create (:INFERENCE:CONFIG {

kpiID:"222",

expectedTrend: "DOWNWARDS",

name:"Average Test Build Time",

action:"AVERAGE",

schedule:"DAILY",

vector:"BUILD",

toolName:"JENKINS",

isGroupBy:"FALSE",

groupByFieldName:"",

groupByField:"",

averageField:"",

startTimeField:"SPKstartTime",

endTimeField:"SPKendTime",

resultOutPutType:"number",

isComparisionKpi:"TRUE",

neo4jQuery:"MATCH (n:JENKINS:DATA) WHERE n.SPKvector = 'BUILD' and n.SPKstatus='Success' RETURN COALESCE(Avg(toInt(n.SPKduration)),0) as avgOutput",

timeFormat:"YYYY-MM-DD HH:MM",

neo4jLabel: "JENKINS:DATA",

nextRun:"1510770000000",

lastRunTime:"1561487460000",

isActive:"TRUE"

})

Inference Results

After calculation, all results are stored in Neo4j with label INFERENCE:RESULTS

MATCH (n:INFERENCE:RESULTS) RETURN n LIMIT 25

Top Section