Contents

Overview:

FinOps is an evolving cloud financial management discipline and cultural practice that enables organizations to get maximum business value by helping engineering, finance, technology and business teams to collaborate on data-driven spending decisions.

FinOps is the practice of bringing a financial accountability cultural change to the variable spend model of cloud, enabling distributed engineering and business teams to make trade-offs between speed, cost, and quality in their cloud architecture and investment decisions.

“If it seems that FinOps is about saving money, then think again. FinOps is about making money.”

Cloud spend can drive more revenue, signal customer base growth, enable more product and feature release velocity, or even help shut down a data center. FinOps is all about removing blockers; empowering engineering teams to deliver better features, apps, and migrations faster; and enabling a cross-functional conversation about where to invest and when. Sometimes a business will decide to tighten the belt; sometimes it’ll decide to invest more. But now teams know why they’re making those decisions.

Principles

Principles that drive our FinOps practices

• Teams need to collaborate

• Everyone takes ownership for their cloud usage

• A centralized team drives FinOps

• Reports should be accessible

• and timely

• Decisions are driven by business value of cloud

• Take advantage of the variable cost model of the cloud

Personas:

The Personas that FinOps must support as stakeholders

• FinOPS practitioner

• Executive

• Business/Product owner

• Finance/Procurement

• Engineering/operations

Maturity levels:

• Crawl

• Walk

• Run

Phases:

• Inform(Visibility & Allocation)

• Optimize(Rates & Usages)

• Operate(Continuous improvement & Operations)

Domains:

Domains of activity we will have to perform in our FinOps practice

• Understanding Cloud Usage and Cost

• Performance Tracking & Benchmarking

• Real-Time Decision Making

• Cloud Rate Optimization

• Cloud Usage Optimization

• Organizational Alignment

Insights Application have following agents to collect FinOps data from Cloud like AWS, Azure portal

• AWS FinOps Agent using CSV:

•  Azure FinOps Agent with API:

•  Azure FinOps Agent using CSV:

AWS

Security credentials:

If access and secret access key is not found need to create an access key for the AWS account root user

Minimum permissions

To perform the following steps, you must have at least the following IAM permissions:

You must sign in as the AWS account root user, which requires no additional AWS Identity and Access Management (IAM) permissions. You can't perform these steps as an IAM user or role.

Use your AWS account's email address and password to sign in to the AWS Management Console as your AWS account root user.

Choose your account name in the navigation bar, and then choose Security Credentials as shown below:

Open

If you see a warning about accessing the security credentials for your AWS account, choose Continue to Security Credentials.

Expand the Access keys (access key ID and secret access key) section as shown below

Open

Choose Create New Access Key. If this option is not available, then you already have the maximum number of access keys. You must delete one of the existing access keys before you can create a new key. For more information, see IAM Object Quotas in the IAM User Guide.

A warning explains that you have only this one opportunity to view or download the secret access key. You can't retrieve it later.

If you choose Show Access Key, you can copy the access key ID and secret key from your browser window and paste it somewhere else.

If you choose Download Key File, you receive a file named rootkey.csv that contains the access key ID and the secret key. Save the file somewhere safe.

After downloading the key file, sample access key and secret access key once its opened in notepad will look something like the below screenshot

When you no longer need the access key we recommend that you delete it, or at least mark it inactive by choosing Make Inactive so that no one can misuse it.

Creating csv file in S3 bucket:

Need to create a cost explorer report below are the steps to create one:

1. Navigate to Billing

2. Navigate to cost and usage report

3. Create a report based by hitting on the create report button

This will create a new report creating wizard with 3 steps to create a report that will capture all the cost and other data based on frequency of selection(Hourly, Daily and Monthly) the cost of capturing the report is free of cost.

In the first step, user may have to provide name of the report as given below in the screenshot for reference

In the second step need to configure either an existing bucket or create a new one to store the report. The report can be versioned or overwrite the existing one. Finally the report has to selected as a GZIP file type for our processing

In the final step review and complete all the details.

Accessing S3 bucket:

Once the file is configured for daily creation, below is the structure by which the file will be placed in the S3 bucket. Copy the folder view of the S3 URI as shown below

For instance the S3 folder path will look something like this: s3://arunducketfordev/FinopsCostAndUsageReportCombined/FinopsCostAndUsageReportCombined

Copy the S3 folder path and place it in the config file under the "s3FilePath": "s3://arunducketfordev/FinopsCostAndUsageReportCombined/FinopsCostAndUsageReportCombined"

Similarly need to provide the aws access key and awsSecretkey. To connect to the specific aws account in the config file as shown below update the value with the ones to be used for accessing your aws console:

"awsAccesskey": "",

"awsSecretkey": "",

"s3FilePath": "",

Forecast data collection:

To get the forecast details, this portion is an optional data collection portion so provide Boolean value either true or false based on the requirement, if needed to capture the data for forecast then provide the data collection data for

1. forcastDataCollectionInDays as always 30 or less than 30 days, since AWS cannot gather more than specific time range usually it will be 30 days of data or slightly more days.

2. isForecastNeeded to either true or false based on requirement

3. forecastMetrics will always be UNBLENDED_COST

4. forecastGranularity will always be DAILY.

5. forecastRegions can be given based on our required region or allregions

"forcastDataCollectionInDays": 30,

"isForecastNeeded": true,

"forecastMetrics": "UNBLENDED_COST",

"forecastGranularity": "DAILY",

"forecastRegions": [

"us-east-1",

"us-west-1"

],

provide any specific region or provide “allregions” if needed for all the regions. Similar to below one:

"forecastrequest": {

"Filter": {

"Dimensions": {

"Key": "REGION",

"Values": [

"allregions"

]

}

}

}

Additional metrics data collection:

For aws metrics data capturing, provide what ever the additional metrics that are required. This portions is also an optional data collection similar to forecast data, so need to provide Boolean value either true or false for the parameter isMetricsNeeded

Below are the details needed to be configured to gathering the additional metrics:

1. period: this is usually one day data (60*60*24=86400) since the agent runs daily based on when it was last updated

2. resourceMetrics: this portion should be given based on what are the metrics that we need to capture additionally for instance here AWS/EC2 instance is to be captured, so given the supporting attributes as:

   1. dimensionsName, this is to give the api to filter out only InstanceId data

   2. productName: this is to give the api to filter out only product name data

   3. availablemetrics: for this specific product what are the metrics data to be captured

      1. CPUUtilization: give the statics data collection over here as average, sum, etc.,

      2. DiskReadOps: give the statics data collection over here as average, sum, etc.,

"additionalResourceDetails": {

"period": 86400,

"isMetricsNeeded": true,

"resourceMetrics": {

"AWS/EC2": {

"dimensionsName": "InstanceId",

"productName": "Amazon Elastic Compute Cloud",

"availablemetrics": {

"CPUUtilization": "Average",

"DiskReadOps": "Average"

}

},

Other parameters:

Provide the start from date and time to capture from any past dated from when the csv file started capturing the data startFrom: "2022-10-01T00:00:00",

3. Agent creation and data collection:

In the insights product go for agent creation under the configuration section and select the AWS agent from the drop down of the agent creation once after giving the type of OS, version of product to used for agent creation etc,. once the agent is created the agent will move to start status and will run continuously on daily basis based on when the csv file is created/updated in the S3 bucket.

Queue

Once the agent starts running the rabbit mq will be published with data gathered from the subscriber which here is the finops aws agent. The queue that will be listening to the subscribed data will be channelized 2 different queues one for gathering the data and other for the health of the agent. By default if the queue names are not changed in the agent configuration while creating the agent from insights product, below are the queues which will listen to the agent data and health data details:

"data": "FINOPS.AWSFINOPS.DATA",

"health": "FINOPS.AWSFINOPS.HEALTH"

Database for capturing and saving the collected data

The agent will create below labels in Neo4j server:

1. COST_MANAGEMENT: having unique fields as usagetype, servicename, operation, usagedatestart, lineitemtype, resourceid

2. RESOURCE_TAGS: having unique fields as resourceid, tagkey, tagvalue

3. RESOURCE_DETAILS: having unique fields as resourceid, servicename

4. RESOURCE_METRICS: having unique fields as resourceid, name, timestamp

5. If forecast is enabled then FORECAST is also created having unique fields as usagedate

Below relationships:

1. RESOURCE_HAS_COST: having relationship between RESOURCE_DETAILS and COST_MANAGEMENT with constraints on resourceid, cloudtype, servicename between those 2 labels.

2. TAGS_HAS_RESOURCE: having relationship between RESOURCE_TAGS and RESOURCE_DETAILS and LATEST with constraints on resourceid only between tose 2 labels

3. RESOURCE_HAS_METRICS: having relationship between RESOURCE_DETAILS and RESOURCE_METRICS with constraints on resourceid only between those 2 labels.

Azure FinOps Agent with REST API

This agent collect data using rest API, all cost released API call free of cost in azure, but you need more permission to get this data using rest API.

Prerequisite:

Capture SubscriptionId

• Login to Azure portal using https://portal.azure.com/

• In search box type “Subscriptions”

• Use SubscriptionsId field

1.2 Capture TenantId

• Login to Azure portal using https://portal.azure.com/

• In Search box for type Azure Active Directory.

• Select Tenant Id from page

• If you have multiple tenants from the overview page click on Manage Tenants to get the Tenant Id of a specific tenant.If you need the Tenant Id of the default Tenant you can get it from the Overview page itself.

1.3 Capture ClientId

Create Azure Active Directory Application

• Login to Azure Portal if you are not already logged in.

• Search for Azure Active Directory

• From left Menu of Azure Directory Overview -> click App registrations -> Click + New Registration.

• Enter your application name. Optionally you can enter the Redirect URI to get Authentication response.

• Click on Register.

Azure Client Id is Active Directory Application Id. Client Id is the unique identifier of an application created in Active Directory. You can have many applications in an Active Directory. Each application will have a different access level.

• Login to Azure Portal if you are not already logged in.

• Search for Azure Active Directory

• From left Menu of Azure Directory Overview -> click App registration

• click Owned Application tab -> Click the name of the application created in previous step.

• Copy Application Id which is nothing but client id

1.4Capture Secretkey

• Login to Azure Portal if you are not already logged in.

• Search for Azure Active Directory

• From left Menu of Azure Directory -> click App Registration -> Click the name of the application created in the previous step.

• From the left menu of selected application -> click Certificates & secrets

• Click on New Client Secret from right side pane -> Add description and expiration in Add a Client Secret screen -> Click Add button.

• Copy Value field, do not copy a secret key

1.5 Assign Permission to Subscription to extract cost data

• Login to Azure Portal if you are not already logged in.

• Search for Subscription

• Go to Access Control

• Click ADD

• Go to Add Role Assignment

• Select role Contributor

• Go to Next tab Member

• Select User, Group, Service Principle

• Click on add member

• Type application name that created as part of extract ClientId (Azure AD App registrations)

• Click on review and assign button and save it

• Do this activity for all Subscriptions or its parent Hierarchy

Configure Azure FinOps Agent

Please refer following documentation to configure agent

https://onedevops.atlassian.net/wiki/spaces/OI/pages/283377677/Registering+an+agent

To configure this agent, you need following basic configuration in config. Json, in prerequisite these are already capture

"azureSubscriptionId",

"azureTenantId",

"azureClientId",

"azureSecretkey"

There are following section in dynamic template, user may modify these based on need, these sections are

costManagement: this section uses to capture cost release data based on filter and group by.

Currently application capture it Daily to get more accurate cost

for more detail refer:

https://learn.microsoft.com/en-us/rest/api/cost-management/query/usage?tabs=HTTP

resourceDetails: this section uses to capture all resource details including subscription and all metadata release to resource

for more detail refer:

https://learn.microsoft.com/en-us/rest/api/resources/resource-groups/get

resourceTags: this section uses to capture resource tag detail

for more detail refer:

https://learn.microsoft.com/en-us/rest/api/resources/resource-groups/get

forecast: this section uses to capture forecast released data

for more info refer:

https://learn.microsoft.com/en-us/rest/api/cost-management/forecast/usage?tabs=HTTP

additionalResourceDetails: this section uses to capture metrics information, only configured metrics data will be collected

for more detail refer:

https://learn.microsoft.com/en-us/rest/api/monitor/metrics/list?tabs=HTTP

Azure FinOps Agent using CSV

Prerequisite

To configure this agent, user need following token information

"azureSubscriptionId",

"azureTenantId",

"azureClientId",

"azureSecretkey",

"storageAccountUrl",

"storageAccountKey",

"containerName",

"storageConnectionString"

SubscriptionId

Please refer “Azure FinOps Agent with REST API” prerequisite section

TenantId

Please refer “Azure FinOps Agent with REST API” prerequisite section

ClientId

Please refer “Azure FinOps Agent with REST API” prerequisite section

Secretkey

Please refer “Azure FinOps Agent with REST API” prerequisite section

Configure Azure cost analysis report

• Login to Azure portal

• Search for Cost Management + Billing

• Select Cost Management

• Left had side section scroll to setting and select Exports

• Clink on Add button

• Configure report as shown in below image

Configure storage details in same section

• Click on create report

• Go to storage account you selected in previous section

• Under Security + networking, select Access keys. Your account access keys appear, as well as the complete connection string for each key.

• Select Show keys to show your access keys and connection strings and to enable buttons to copy the values.

• Under key1, find the Key value. Select the Copy button to copy the account key.

• Alternately, you can copy the entire connection string. Under key1, find the Connection string value. Select the Copy button to copy the connection string.

• Add this value in following section

storageAccountUrl

This is nothing but

"https://< storage_account_name >.blob.core.windows.net"

storageAccountKey

get it from above section Security + networking -- > Access keys Key1 or any -- > Kay

containerName

use above configured name

storageConnectionString

get it from above section Security + networking -- > Access keys Key1 or any -- > Connection string

Configure Azure FinOps Agent

Please refer “Azure FinOps Agent with REST API” Configure Azure FinOps Agent section

Grafana dashboards:

Configure the Grafana with FinOPS data collected, some sample dashboards are attached below

GCP

• Pre-requisite

1. Create a service account with access to project. Save the credentials json file for the service account we need to add this json file in agent directory later.

2. Enable access to below mentioned APIs in google cloud console.

3. Export Billing detailed usage report

4. After 24hrs validate if the billing data is getting populated in bigquery table. Please note down the name of GCP billing table we need to add it in GCP agent’s config.json file.

Open

https://stackoverflow.com/questions/66207627/api-for-getting-daily-costs-related-to-gcp

• Python libraries required for GCP agent to execute

pip install db-dtypes google-api-core google-auth google-cloud-asset google-cloud-bigquery google-cloud-bigquery-storage google-cloud-billing-budgets google-cloud-core google-cloud-monitoring google-cloud-recommender google-cloud-python-client

• Configure GCP Finops agent

Please update the following fields in agent’s config.json

1. "datasetdetails": {
   "projectname": {
   "datasetname": "",
   "datareporttablename": ""
   }
   },

Enter projectname, datasetname is the name of big query dataset where the billing report is exported,

datareporttablename is the name of big query table name where billing data is present

2. "recommendationzones": [
"us-west4-b",
"us-central1-a",
"us-central1-b",
"us-central1-c",
"us-central1-f",
"us-east4-a",
"us-east4-b",
"us-east4-c"
],

Add recommended zones for which you need to get recommendations.

3. "credentials": "<path where you are placing the service account config json>”

This is the path were the service account config json is placed.