Architecture of Insights

Architecture of Insights

Insights Architecture Components

Insights Architecture Pictorial Representation

Agents

• Insights contains Python Agents that are written across various DevOps Functional Blocks such as Application Lifecycle Management(ALM), Software Configuration Management(SCM), Continuous Integration(CI), Artifact Management, Code Quality, Continuous Testing, Deployment, and Cloud.
• These are Python Agents collect tool data from various tools such as Jira, Rally, GitHub, Bitbucket, Jenkins, Bamboo, Docker, Rundeck etc., using RESTful API calls from these given tools.
• Once the agent collect data, it starts publishing the results into RabbitMQ.

RabbitMQ

• Architecture of InsightsRabbitMQ is an open source message broker software that originally implemented Advance Message Queuing Protocol(AMPQ), and has since been extended with a plug-in architecture to support Streaming Text Oriented Messaging Protocol (STOMP), MQTT, and other protocols.
• In Insights, RabbitMQ acts as a message broker between the Insights Agents and the Insights Engine.
• The Insights Python Agents subscribe and publish the data into RabbitMQ, where it is queued, and then absorbed by Insights Engine.

Insights Engine

• Insights Engine runs between RabbitMQ and Neo4j DB. It creates queues into RabbitMQ, collects data from the queues, and the creates labels into Neo4j DB based on the configurations provided in Insights Python Agents.
• Once it collects the data from the RabbitMQ queues, it applies Co-relation logic, Business Mapping (according to project hierarchy), and the corresponding node is created under following tool labels in Neo4j DB.

Data Storage

• Data Storage section has three different databases such as Architecture of InsightsNeo4j (Graph DB), Architecture of InsightsElasticsearch (NoSQL DB), and Architecture of InsightsPostgreSQL (SQL DB).
• The data that is present in the Neo4j DB is always in sync with Elasticsearch DB by using GraphAware - Neo4j plugin.

Insights Inference Engine

• Insights Inference Engine is responsible to calculate the real time DevOps metrics observations across various vectors at scheduled time interval such as Daily, Weekly, Monthly, Yearly.
• It uses Apache SparkArchitecture of Insights as its base component.
• It fetches pre-configured Insights Inference KPIs that are present in Elasticsearch DB, imposes mathematical calculations, calculates results, and then pushes the results back into Elasticsearch DB.