Introduction
Insights collects data from various DevOps tools to create meaningful dashboard through KPIs. However, the raw data received from tool is not in format and/or sufficient to create advance dashboard. Insights uses something called offline data processing to enhance current data to assist advance KPI creations. Something like we do in datawarehousing.
Offline data processing used to be a manual processing wherein, implementation engineer used to write data enrichment Neo4j cypher queries into a json file. Insight data processing engine, then used to read these cypher queries and run them based on a schedule.
This approach has many drawbacks like manually modifying json on harddisk, no way to disable q particular query, change run schedule on the fly to name the few. New Offline Data Processing is a feature to automate disabling specific queries from offline processing through UI (User Interface). It provides functionality to control provide implementation engineer an intuitive UI to easy entire process. It provides implementation engineer full control on maintaining each offline processing Neo4j query to enhance productivity. Currently offline data update process happens through a Json file having Neo4j queries to update records. Insights engine process those Neo4j queries periodically based on fixed scheduleIt also provides engineer with post execution details like last run time, next run time, query processing time, error details if any.
Migration to Offline Data Processing from Database
Previously, Offline Data Processing was managed from File management by uploading the Data Enrichment File. Now to automate disabling specific queries from offline processing through UI, we must upload the Data Enrichment file with appropriate tool name & cron schedule.
If there are active offline data records in database, the engine will execute them otherwise based on Data Enrichment file in File Management.
Pre-Requisite
Insights >= v10.0
Upload Data Enrichment Json file in Offline Data Processing.
Only Admin can access this functionality.
Steps to configure Offline Data
Navigate to Offline Data Processing screen, existing offline data are listed here. To create a new Offline data, click on Add (+) icon.
...