DevOps Maturity Measurement

DevOps is a combination of 3 main areas - People, Process and Tools. While the areas of People and Process are subjective in their measurements, we should be able to measure the maturity of the tools integration using Insights. There are six main vectors which caters to the measurement of DevOps from a tools standpoint. Following are the vectors. 

1. Build - Areas covering how the source code is organized and how the application is packaged. 
2. Deploy - Areas covering how the application artifact is deployed across the various environments. 
3. Verify - Areas covering how the application is tested to ensure that it meets the necessary functional and non-functional requirements. 
4. Release - Areas covering how the application is released to the various controlled (& non-controlled) environments 
5. Measure - Areas covering how the application is performing with respect to set standards both from functional and non-functional perspective. 
6. Environment - Areas covering how the environment needed for application is provisioned and certified in the necessary terms. 

Each of the above vectors will have a combination of one or more tools - For example the measurement of Build Vector may entail fetching information from source control, build definition followed, Continuous Integration and Static Analyzer. 

Before proceeding to the measurement of DevOps Maturity - one should have a clearly defined maturity model and also what are the various stages of Maturity. For example the various stages of Maturity could be something like this. 

1. Initial - Indicates basic level of maturity and tilt towards automation
2. Managed - Basic tooling in place with adhoc automation. 
3. Defined - Integration in place with proper SDLC process defined in tools 
4. Quantitatively Managed - Pipeline established with End to End automation in place. 
5. Responsive - Continuous Improvement on the established pipeline based on the measurement parameters. 

So in a nutshell, a maturity model should be a 6x5 matrix which talks about the progressive maturity for each vector and the level for each stage, we can proceed to the next stage which is identification of KPI for the measurement. Please find below an example of the matrix for one vector across five stages. 

Once the matrix is defined - next step is to identify the KPI which will be able to govern the above defined maturity model and codify them as cypher queries in the necessary panels for measurement of maturity. 

Types of Dashboards 

Once the matrix and the KPI is defined - then the next step is to translate the KPI into cypher queries which will help to numerically measure the value for each of the KPI defined. Based on the numerical value, each organization can define the maturity level and they can choose to color code the value based on how below or above they are when compared to the threshold. The below dashboards can be built using extensive cyper queries inside Server status Panels inside Grafana. You can refer to the documentation on how to configure server status panels inside Insights here

There are 2 types of Dashboards which needs to be built at the very minimum in order to measure maturity progress. 

Vector Specific Dashboard - This dashboard will consists of individual metrics which define the build vector maturity. As indicated above, each vector can relate to more than one tool in terms of KPI measurement and these could be classified as sub-sections within each vector. Below is a sample dashboard shown for one vector - namely Build. 

The dashboard has 3 sections 

SCM - Talking about the various parameters related to the SCM maturity (like commits, orphan commits, No of SCM)

CI - Talking about the ratio between the pass : fail of the builds 

Code Quality - Talking about the various parameters related to the quality of the code which is being checked in. 

The above values could be measured for each portfolio or logical unit like the sprint. 

Many of the KPI shown below depend on the various parameters not limited to tool customization and pipeline definition inside the organization.

Maturity Score Dashboard - This Dashboard shows the consolidated score for each of the defined vectors and also shows overall progress of the maturity model. You should be able to provide a drill down to the individual vector dashboard shown above from this dashboard. The dashboard uses some of the advanced features like offline processing and caching in order to show the trends - i.e - Based upon the configured threshold values for each of the KPI - the queries are executed offline and the consolidated score are stored in Neo4j for faster processing. You can also check the maturity score based on individual portfolio and also logical work units like the sprint.