Dd Audit Cost Spike Investigation

by datadog-labs5b40c73824ecNo license177 starsListed Oct 8, 2026Updated Oct 8, 2026Repository updated today

Investigate a Datadog product usage or cost spike by correlating Usage Metering data (when/what spiked) with Audit Trail config changes (who changed what in the preceding window).

Instructions onlyDevOps & Cloud Business & Finance

AI-generated overview

Correlates Datadog usage metering data with Audit Trail config changes to explain a cost or usage spike.

What it does: Guides an investigation that first queries Datadog hourly usage metering to find when and which product family spiked, then searches Audit Trail for configuration changes in the preceding window. It narrows results to audit categories relevant to the spiking product and produces a report naming a likely causal change with a confidence level and next steps. It identifies configuration changes, not the specific user or process that submitted data.
When to use it: Use when a Datadog product's usage or cost rises unexpectedly and you need to find the configuration change behind it. It fits FinOps or operations investigations where billing data must be tied to change history.
Requirements: Datadog access: OAuth2 login via the pup CLI for audit queries, plus DD_API_KEY, DD_APP_KEY and DD_SITE for usage metering queries. Network access to the Datadog API and the pup CLI with jq are needed. No scripts ship with the skill; it is instructions only.

Audit Trail: Cost / Usage Spike Investigation

Identify what caused a Datadog usage spike by correlating billing data with configuration change history.

The causal chain is: someone changed something → that change increased data volume → usage spiked → cost went up. Usage Metering tells you when and what; Audit Trail tells you who made the change.

Prerequisites

bash

pup auth login   # OAuth2 (recommended) — covers audit queries# Usage Metering queries also need DD_API_KEY + DD_APP_KEYexport DD_API_KEY=<your-api-key>export DD_APP_KEY=<your-app-key>export DD_SITE=datadoghq.com

pup auth login   # OAuth2 (recommended) — covers audit queries# Usage Metering queries also need DD_API_KEY + DD_APP_KEYexport DD_API_KEY=<your-api-key>export DD_APP_KEY=<your-app-key>export DD_SITE=datadoghq.com

Scope Boundary

This skill identifies configuration changes that may have caused a spike. It does not identify which specific user or process submitted the data (e.g., which service sent the LLM spans). For per-submission attribution, use LLM Observability traces or APM instrumentation.

Investigation Workflow

Step 1 — Identify the spike window and product family

bash

START=$(date -u -v-7d +"%Y-%m-%dT%H:%M:%SZ" 2>/dev/null || date -u -d "7 days ago" +"%Y-%m-%dT%H:%M:%SZ")END=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
curl -s -G "https://api.${DD_SITE}/api/v2/usage/hourly_usage" \  -H "DD-API-KEY: ${DD_API_KEY}" \  -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \  --data-urlencode "filter[timestamp][start]=${START}" \  --data-urlencode "filter[timestamp][end]=${END}" \  --data-urlencode "filter[product_families]=all" \  | jq '[.data[] | {      timestamp: .attributes.timestamp,      product: .attributes.product_family,      measurements: [.attributes.measurements[] | {type: .usage_type, value: .value}]    }]'

START=$(date -u -v-7d +"%Y-%m-%dT%H:%M:%SZ" 2>/dev/null || date -u -d "7 days ago" +"%Y-%m-%dT%H:%M:%SZ")END=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
curl -s -G "https://api.${DD_SITE}/api/v2/usage/hourly_usage" \  -H "DD-API-KEY: ${DD_API_KEY}" \  -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \  --data-urlencode "filter[timestamp][start]=${START}" \  --data-urlencode "filter[timestamp][end]=${END}" \  --data-urlencode "filter[product_families]=all" \  | jq '[.data[] | {      timestamp: .attributes.timestamp,      product: .attributes.product_family,      measurements: [.attributes.measurements[] | {type: .usage_type, value: .value}]    }]'

Product families with LLM/AI coverage: llm_observability, bits_ai, logs, apm

Step 2 — Pinpoint the spike

From Step 1, identify the hour/day where volume jumped. Note the timestamp as SPIKE_TIME.

Step 3 — Search Audit Trail for config changes in the 24h preceding the spike

bash

pup audit-logs search \  --query "@action:(created OR modified OR deleted)" \  --from "SPIKE_TIME_MINUS_24H" \  --to "SPIKE_TIME" \  --limit 200 \  -o json \  | jq '[.data[] | {      timestamp: .attributes.timestamp,      user: .attributes.attributes.usr.email,      actor_type: .attributes.attributes.evt.actor.type,      action: .attributes.attributes.action,      event_category: .attributes.attributes.evt.name,      resource_type: .attributes.attributes.asset.type,      resource_id: .attributes.attributes.asset.id    }]'

pup audit-logs search \  --query "@action:(created OR modified OR deleted)" \  --from "SPIKE_TIME_MINUS_24H" \  --to "SPIKE_TIME" \  --limit 200 \  -o json \  | jq '[.data[] | {      timestamp: .attributes.timestamp,      user: .attributes.attributes.usr.email,      actor_type: .attributes.attributes.evt.actor.type,      action: .attributes.attributes.action,      event_category: .attributes.attributes.evt.name,      resource_type: .attributes.attributes.asset.type,      resource_id: .attributes.attributes.asset.id    }]'

Note: --from and --to accept ISO timestamps (e.g., 2026-05-01T14:00:00Z) or relative values (1h, 24h, 7d).

Step 4 — Narrow to product-relevant config changes

Filter to the audit categories most likely to affect the spiking product:

If this product spiked	Add to query
`llm_observability`	`@evt.name:(Integration OR APM OR "Log Management")`
`logs` / `indexed_logs`	`@evt.name:"Log Management" @asset.type:(pipeline OR index OR exclusion_filter)`
`apm` / `indexed_spans`	`@evt.name:APM @asset.type:(retention_filter OR sampling_rate)`
`rum`	`@evt.name:RUM`
`metrics`	`@evt.name:Metrics`

Example for LLM Observability spike:

bash

pup audit-logs search \  --query "@evt.name:(Integration OR APM OR \"Log Management\") @action:(created OR modified)" \  --from "SPIKE_TIME_MINUS_24H" \  --to "SPIKE_TIME" \  --limit 100 \  -o json \  | jq '[.data[] | {      timestamp: .attributes.timestamp,      user: .attributes.attributes.usr.email,      action: .attributes.attributes.action,      category: .attributes.attributes.evt.name,      resource_type: .attributes.attributes.asset.type,      resource_id: .attributes.attributes.asset.id    }]'

pup audit-logs search \  --query "@evt.name:(Integration OR APM OR \"Log Management\") @action:(created OR modified)" \  --from "SPIKE_TIME_MINUS_24H" \  --to "SPIKE_TIME" \  --limit 100 \  -o json \  | jq '[.data[] | {      timestamp: .attributes.timestamp,      user: .attributes.attributes.usr.email,      action: .attributes.attributes.action,      category: .attributes.attributes.evt.name,      resource_type: .attributes.attributes.asset.type,      resource_id: .attributes.attributes.asset.id    }]'

Output Format

Usage spike detected:  Product: <product_family>  Spike time: <SPIKE_TIME>  Volume: <baseline> → <spike_value> (<magnitude>×)
Configuration changes in 24h preceding spike:  <timestamp> | <user_email> | <action> <resource_type> <resource_id> | <category>
Likely causal change: <most-proximate change matching the product family>
Confidence: HIGH (single clear change) / MEDIUM (multiple candidates) / LOW (no matching changes)
Next steps:  - Confirm with <user_email> whether the change was intentional  - If unintentional: revert <resource_id> and monitor volume  - If intentional: update cost forecasts and alert thresholds

Usage spike detected:  Product: <product_family>  Spike time: <SPIKE_TIME>  Volume: <baseline> → <spike_value> (<magnitude>×)
Configuration changes in 24h preceding spike:  <timestamp> | <user_email> | <action> <resource_type> <resource_id> | <category>
Likely causal change: <most-proximate change matching the product family>
Confidence: HIGH (single clear change) / MEDIUM (multiple candidates) / LOW (no matching changes)
Next steps:  - Confirm with <user_email> whether the change was intentional  - If unintentional: revert <resource_id> and monitor volume  - If intentional: update cost forecasts and alert thresholds

When No Causal Change Is Found

The change may predate the 24h window — expand to 72h
The increase may be from application-side instrumentation changes — check deploys
The increase may be organic traffic growth — correlate with product launch or traffic event

References

Source and attribution

Source:datadog-labs/agent-skillsindd-audit/cost-spike-investigationat commit5b40c73

License: No license

Content belongs to its original authors. SourceWeft indexes it from a public repository.

Report or request removal

Dd Audit Cost Spike Investigation

Audit Trail: Cost / Usage Spike Investigation

Prerequisites

Scope Boundary

Investigation Workflow

Step 1 — Identify the spike window and product family

Step 2 — Pinpoint the spike

Step 3 — Search Audit Trail for config changes in the 24h preceding the spike

Step 4 — Narrow to product-relevant config changes

Output Format

When No Causal Change Is Found

References

Source and attribution

More from datadog-labs/agent-skills

Upgrade Browser Sdk V7

Upgrade Browser Sdk V6

Dd Audit Security Investigation

Upgrade Browser Sdk V5

Dd Audit Key Compromise

Troubleshoot Ssi

More in DevOps & Cloud

M5 Onboard

Yeet

Runbook

Incident Response

Deploy Checklist

Spanner Basics