Data Collection Configuration (Agent Configuration)

Agents are the tools data collectors of Insights. These are python snippets which connects to DevOps tools and collect data using REST protocol. Each agents needs to be registered in system (Postgres DB, agent_configuration table) to start collecting data. Agent Management section under 'Configuration' helps to Add / Change Status / Edit / Delete Agents.

When Insights is installed, one of the important steps is to install Agent Daemon. This is daemon process running on server where you want to run your tools agents. Daemon Agents helps Insights in Adding / Changing Status / Editing / Deleting Agents.

Daemon Agent needs connectivity to RabbitMQ. It is essential to ensure that agent Daemon is running (preferably on server 2, or as per the requirement) prior to registering an Agent from Insights UI (Under Configuration).

We recommend to take latest artifacts for PlatformService.war , PlatformEngine.jar and UI app.

NOTE - Current out-of-box agent daemon and agents support Windows, RHEL 7.x, CentOS 7.5.x and Ubuntu 16.04 OS. For other OS flavors, one need to follow specific instructions given below in Other OS section.

Agent Daemon is available in doc root at - https://infra.cogdevops.com:8443/repository/docroot/insights_install/release/latest/agentdaemon.zip

Follow below installation instructions for Daemon agent configuration.

Daemon Agent Configuration and Execution

On Agent server

Create a folder at desired location to hold AgentDaemon and Tools agent code. For Example - On Windows D:\\Agents
Set up environment variable INSIGHTS_AGENT_HOME pointing to above folder
Create the following folders under the environment variable path $INSIGHTS_AGENT_HOME
AgentDaemon and
PlatformAgents

For example - D:\\Agents\\AgentDaemon and D:\\Agents\\PlatformAgents

Provide executable and write permissions to these folders
Download agentdaemon.zip from https://infra.cogdevops.com:8443/repository/docroot/insights_install/release/latest/agentdaemon.zip and extract it inside AgentDaemon folder. Make sure folder structure starts with "com" and not "agentdaemon"
Update config.json for following highlighted fields

{

"baseExtractionPath": "%Path%(INSIGHTS_AGENT_HOME)/PlatformAgents",

"insightsTimeZone": "Asia/Kolkata",

"loggingSetting": {

"logLevel": "WARN"

},

"mqConfig": {

"host": "127.0.0.1",

"password": "guest",

"user": "guest",

"exchange": "iSight",

"agentExchange":"iAgent"

},

"publish": {

"health": "AGENT.DAEMON.HEALTH"

},

"isDebugAllowed": false,

"subscribe": {

"agentPkgQueue": "INSIGHTS.AGENTS.PACKAGE"

},

"timeStampFormat": "%Y-%m-%dT%H:%M:%SZ",

"toolsTimeZone": "GMT"

}

Create AgentDaemon as service -

RHEL 7.x/Ubuntu/CentOS OS

Use installdaemonagent.sh to create OS specific AgentDaemon service and provide OS input while running the script.
Service name is "InSightsDaemonAgent"

installdaemonagent.sh currently supports RHEL 7.x, Ubuntu 16.04 and CentOS 7.5.x, might need to update it based on other OS types and versions.

Windows OS

Install nssm utility on Agent server (which is part of Insights installation package )
Run installdaemonagent.bat provided inside ..\\Agents\\AgentDaemon folder
It will create Windows daemon agent service with name "DaemonAgent"
Validate whether service has been installed and started successfully

Once agent daemon start running successfully, it should have creates a queue in RabbitMQ with name 'INSIGHTS.AGENTS.PACKAGE'

On Application/Platform server (Tomcat)

Please note the nexus3 docroot raw repo URL - Browse URL: https://infra.cogdevops.com:8443/#browse/browse:docroot:insights_install%2Frelease or
Rest Service URL: https://infra.cogdevops.com:8443/nexusrepo/service/rest/repository/browse/docroot/insights_install/release

server-config.json should be updated as follows -

Add new section and change unzipPath parameter
If your web server doesn't have permission to connect to docroot URL, then make isOnlineRegistration as false and provide offlineAgentPath path. This is offline agent registration

"agentDetails": {
"isOnlineRegistration" : true,
"docrootUrl":"https://infra.cogdevops.com:8443/repository/docroot/insights_install",
"offlineAgentPath" : "D://download//agents",
"unzipPath":"D://download",
"agentExchange" : "iAgent",
"agentPkgQueue" : "INSIGHTS.AGENTS.PACKAGE"
}

Make sure that server-config.json has the following section

"messageQueue": {

"host": "localhost",

"user": "iSight",

"password": "iSight",

"prefetchCount" : 5

}

Offline agent registration

Inside offlineAgentPath, create folder and give it some version like v1.0 or same version which you are planning to manually download from docroot url.
Inside version folder create agent specific folders like "git", "jenkins", "jira" same as that of docroot
Download individual agent zip from docroot and unzip them in respective agent folders
Make sure folder structure is starting from com/cognizant/....

For Example typical structure assuming offlineAgentPath as "D://download//agents" --

"D:\\download\\agents\v1.0\git\com\cognizant..."

"D:\\download\\agents\v1.0\jenkins\com\cognizant..."

Deploy new PlatformService WAR

Run below SQL query to register Daemon Agent using PostgreSQL pgAdmin tool (Administration tool or SQL developer for PostgreSQL)

Update highlighted value '{"key":"value"}' in below query with daemon config.json.

Remove unique constraint on unique_key column and execute below query -

INSERT INTO public.agent_configuration(id, agent_id, agent_json, agent_key, agent_status, agent_version,data_update_supported, os_version, tool_category, tool_name,unique_key, update_date) VALUES (0, 0,'{"key":"value"}','daemon-1523257126' ,'' ,'' , FALSE,'' , 'DAEMONAGENT', 'AGENTDAEMON', '', current_date);

Register all required agents using Agent management tab in Admin screen. Follow Agent Management
Run latest new updated Insights Engine Jar
Validate all agents are running successfully and collecting data

Agent Cofiguration

Once configuration of Daemon Agent is done, please refer following section to configure an Agent:

Agent Management

Steps to upgrade older version of agents management

1. Stop all existing running agents

2. Make sure agents are in stop states.

3. Very Important - Take backup of existing agent code including tracking.json. This tracking.json is used during re-registration purpose.

4. Let Insights Engine consume all queues data. Verify all the queues have 0 pending data

5. Shut down Insights Engine.

6. Undeploy PlatformService WAR

7. Take back up of AGENT_CONFIGURATION table in PostgreSQL

8. DELETE all records from AGENT_CONFIGURATION table in PostgreSQL (using PostgreSQL pgAdmin tool (Administration tool or SQL developer for PostgreSQL)) after taking backup

9. DROP unique constraint on 'unique_key' column in PostgreSQL DB if you have that column.

9. Deploy new PlatformService WAR.

10. Register all required agents using Agent management tab under Configuration tab in Admin screen. Follow Agent Management

In this screen we will have the opportunity to edit the agent config json. Please make sure to get relevant fields from existing config json.
Upload agent tracking.json (step #3 above) during registration . This will make sure your data is collected from last run date.
Please enter agent Id.

11. Update all latest records from all health record.

MATCH (n:HEALTH:LATEST) REMOVE n:LATEST return n

12. Validate all agents are running successfully and collecting data as usual

Other OS

Running an application as a OS service differs based on Unix flavor. For example, Linux RHEL it runs from /etc/init.d folder, in Ubunut it runs from /etc/systemd/system. Hence to run agent daemon and agents on OS other than Insights supported OS version, follow below steps.

Once agent daemon is downloaded and extracted in $INSIGHTS_AGENT_HOME\AgentDaemon folder, modify installagentdaemon.sh for OS specific steps.
Follow offline agent registration steps given above
Once agent packages are downloaded in offlineAgentpath, modify individual agent's installagent.sh for OS specific steps.

If you have successfully run agents on OS other than currently supported OS version, we will encourage you to check-in those files in Github repository to help others.

Back to Top