Insights Architecture

Detailed information on various Insights components, and the Insights Architecture is covered under this section.

Insights Architecture - Components

Insights Architecture - Visual Representation

Insights contains Agents / Tools data collectors written in Python programming language for various DevOps Functional Blocks such as Application Lifecycle Management(ALM), Software Configuration Management(SCM), Continuous Integration(CI), Artifact Management, Code Quality, Continuous Testing, Deployment, and Cloud.
These Python Agents collect tool data from various tools such as Jira, Rally, GitHub, Bitbucket, Jenkins, Bamboo, Docker, Rundeck etc., using RESTful API calls from these given tools.
Once the agent collect data, it starts publishing the results into RabbitMQ.
Insights supports 32 different tools across DevOps functional blocks.

RabbitMQ is an open source message broker software that originally implemented Advance Message Queuing Protocol(AMPQ), and has since been extended with a plug-in architecture to support Streaming Text Oriented Messaging Protocol (STOMP), MQTT, and other protocols.
In Insights, RabbitMQ acts as a message broker between the Insights Agents and the Insights Engine.
The Insights Python Agents subscribe and publish the data into RabbitMQ, where it is queued, and then absorbed by Insights Engine.

Insights Engine runs between RabbitMQ and Neo4j DB. It consumes the data from RabbitMQ and creates the necessary nodes in Neo4j DB along with the co-relations between them.
Once it collects the data from the RabbitMQ queues, it applies Co-relation logic, Business Mapping (according to project hierarchy), and the corresponding node is created under following tool labels in Neo4j DB.

Data Storage section has three different databases such as Neo4j (Graph DB), Elasticsearch (NoSQL DB), and PostgreSQL (SQL DB).
The data that is present in the Neo4j DB is always in sync with Elasticsearch DB by using GraphAware-neo4j-to-elasticsearch plugin. This is a Neo4j plugin, and helps to process the data which is present in Neo4j labels to Elasticsearch indexes in JSON format.
Neo4j DB is used by Insights - Grafana custom dashboard panels. Elasticsearch DB caters Grafana's default dashboard panels, and Insights Inference Engine. PostgreSQL is used to store the user profiles.

Insights Inference Engine is responsible to derive inferences for DevOps metrics across various vectors at scheduled time interval such as Daily, Weekly, Monthly, Yearly.
It uses Apache Spark as its base component.
It fetches pre-configured Insights Inference KPIs that are present in Elasticsearch DB, imposes mathematical calculations, calculates results, and then pushes the results back into Elasticsearch DB. To know more about Insights Inference, click here.

Platform Service refers to the collection RESTful Services for Insights Application. It consists of a set of REST API that are used by cross platforms to fetch/update the results using REST calls.
It abstracts communication between Insights Data Access Layer (DAL), and the various Databases.
It also fetches the Insights Inferences result data from Elasticsearch DB, compares the like vector results, calculates trend, builds message observations, passes the data to Insights UI App, and then the observations are displayed on Insights Tab.

Grafana is an open source metric analytics & visualization suite. It is most commonly used for visualizing time series data for infrastructure and application analytics.
Grafana communicates with Neo4j, Elasticsearch, and PostgreSQL to fetch data for dashboards, store the user profiles, and provide role based access for Insights Application.
Insights Application uses Platform Services REST API calls to fetch results from Grafana for Authentication, User role management, and Dashboards.

Insights UI Application is hosted over a web server, and it is point from which the end user interacts with the system.

Apache http server provides a secure, efficient and extensible server that provides HTTP services in sync with the current HTTP standards
Insights uses Apache httpd server to access Insights User Interface, Grafana, Neo4j, RabbitMq, and Webhook.