Offline Data Processing Configuration

Introduction

Insights collects data from various DevOps tools to create meaningful dashboard through KPIs. However, the raw data received from tool is not in format and/or sufficient to create advance dashboard. Insights uses something called offline data processing to enhance current data to assist advance KPI creations. Something like we do in data warehousing.

Offline data processing used to be a manual processing wherein, implementation engineer used to write data enrichment Neo4j cypher queries into a json file. Insight data processing engine, then used to read these cypher queries and run them based on a schedule.

This approach has many drawbacks like manually modifying json on hard disk, no way to disable a particular query, change run schedule on the fly to name the few. New Offline Data Processing is a feature to provide implementation engineer an intuitive UI to easy entire process. It provides implementation engineer full control on maintaining each offline processing Neo4j query to enhance productivity. It also provides engineer with post execution details like last run time, next run time, query processing time, error details if any.

Migration From Older Version

Previously, Offline Data Processing was managed from File management by uploading the Data Enrichment File. Now to automate disabling specific queries from offline processing through UI, we must upload the Data Enrichment file with appropriate tool name & cron schedule.

If there are active offline data records in database, the engine will execute them otherwise based on Data Enrichment file in File Management.

Pre-Requisite

Insights >= v10.0
Upload Data Enrichment Json file in Offline Data Processing.
Only Admin can access this functionality.

Steps to configure Offline Data

Navigate to Offline Data Processing screen, existing offline data are listed here. To create a new Offline data, click on Add (+) icon.

Fill all mandatory details of Offline Data.
- Tool Name: Select a tool from dropdown.
- Query Name: Input a unique query name.
- Cron Schedule: Add a valid cron expression. (Ex: 0 */10 * ? * * for every 10 min & 0 0 0/1 ? * * * for every hour)
- Cypher Query: Input a correct query to fetch data.
- isActive: Active Offline data will be used for processing records.

Click on Save icon to save Offline data in database.
To store multiple Offline data at once, click on the Upload icon.

Then a pop-up will be displayed, click on Choose File button to choose a valid Json file with required Offline details and click on Upload icon to save all Offline records.

To edit a particular Offline data, select that Offline data and then click on the Edit button. In edit scenario fields which can be edited are Cron Schedule, Cypher Query and isActive flag.
To delete an Offline data, select that Offline data and then click on the Delete button.