Set Up Billing Ingestion

Why

Everything downstream depends on queryable billing data. Without it: no custom reports, no cross-cloud comparison, no programmatic analysis, no anomaly detection, no forecasting. Billing data is the foundation of the entire FinOps practice. A pipeline that ingests cloud bills into a central, queryable data store — refreshed at least daily with alerting on failures — is the single most critical infrastructure investment.

What

Build a billing data pipeline that ingests cost and usage data from each cloud provider into a central queryable store. The pipeline must be automated, refresh at least daily, and include failure alerting.

How

AWS: Enable CUR 2.0 + FOCUS Export

1a — Create the S3 bucket (in a dedicated FinOps account):

aws s3 mb s3://my-finops-billing-data --region us-east-1

1b — Attach the Data Exports bucket policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "EnableAWSDataExportsToS3",
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "billingreports.amazonaws.com",
          "bcm-data-exports.amazonaws.com"
        ]
      },
      "Action": ["s3:PutObject", "s3:GetBucketPolicy"],
      "Resource": [
        "arn:aws:s3:::my-finops-billing-data",
        "arn:aws:s3:::my-finops-billing-data/*"
      ]
    }
  ]
}

1c — Create the Data Export (Console: Billing → Data Exports → Create):

Configuration:
  Table:             Cost and Usage Report (CUR 2.0)
  Include resource IDs: ✓
  Split cost allocation: ✓ (for EKS/ECS)
  Time granularity:  Hourly
  Format:            Parquet (recommended)
  Compression:       Snappy
  S3 bucket:         my-finops-billing-data
  Path prefix:       cur2/

Repeat for FOCUS 1.0: same flow, select “FOCUS (latest version) with AWS Columns”. No additional config needed — the schema is fixed by the spec.

1d — Set up Glue + Athena (for SQL access):

Create a Glue Crawler pointing at s3://my-finops-billing-data/cur2/, a Glue Database (e.g., finops_billing), and an Athena Workgroup with a query result bucket.

AWS limitations:

Azure: Enable Cost Management FOCUS Export

2a — Create a Storage Account:

Azure Portal: Storage Accounts → Create
  Resource Group:  rg-finops
  Name:            stfinopsbilling
  Region:          (align with analytics region)
  Performance:     Standard
  Redundancy:      LRS

2b — Create the Export:

Azure Portal: Cost Management → Exports → Create
  Dataset:         Cost and usage details (FOCUS)
  Version:         1.0r2
  Frequency:       Daily export of month-to-date costs
  Storage account: stfinopsbilling
  Container:       billing-exports
  Directory:       focus/
  Format:          Parquet (Snappy compression)
  File partitioning: ✓ Enabled
  Overwrite:       ✓

2c — Backfill historical data (up to 13 months via Portal, 7 years via API):

Install-Module -Name CostExportV2IngestData
Invoke-CostExportV2IngestData -IgnoreRoleCheck

2d — Connect analytics layer:

The Microsoft FinOps Toolkit (open-source) provides FinOps Hubs (automated ingestion into ADX or Fabric) and pre-built Power BI reports.

Azure required IAM roles:

Azure limitations:

GCP: Enable BigQuery Billing Export

3a — Create a dedicated FinOps project and dataset:

gcloud projects create finops-billing-admin --name="FinOps Billing"
gcloud billing projects link finops-billing-admin \
  --billing-account=01ABCD-EFGH23-4567IJ
gcloud services enable bigquery.googleapis.com \
  --project=finops-billing-admin

bq mk --dataset \
  --location=US \
  --description="Cloud Billing Export Data" \
  finops-billing-admin:cloud_billing_data

3b — Enable all three exports (Console: Billing → Billing export → BigQuery export):

☑ Standard usage cost data        → cloud_billing_data
☑ Detailed usage cost data        → cloud_billing_data (required for FOCUS)
☑ Pricing data                    → cloud_billing_data (required for FOCUS)

3c — Create the FOCUS BigQuery view (GCP does not natively export FOCUS):

Use Google’s “Guide to FOCUS” SQL or the FinOps Foundation FOCUS Converter (open-source) to create a virtual table that transforms Detailed + Pricing data into FOCUS schema at query time (zero additional storage cost).

GCP required IAM roles:

GCP limitations:

Set Up Pipeline Monitoring

Configure alerting for pipeline failures. If billing data stops arriving, every downstream report and alert goes stale silently.

Validate Data Quality

Run the FOCUS Validator on each export source. Cross-check total billed amount against the provider’s invoice for the previous month. Discrepancies >1% require investigation (common causes: tax/credits exclusion, amortisation differences, currency conversion).

Deliverable Checklist

• AWS CUR 2.0 + FOCUS export enabled and delivering to S3
• Glue Crawler + Athena configured for SQL access
• Azure FOCUS export enabled and delivering to Blob Storage
• Historical backfill completed (Azure)
• GCP Detailed + Pricing exports enabled in BigQuery
• FOCUS BigQuery view created (GCP)
• Pipeline failure alerting configured per provider
• Data quality validated against provider invoices
• FOCUS Validator run on each source