Dd Audit Cost Spike Investigation

作者 datadog-labs5b40c73824ec無授權條款177 個星標收錄於 2026年10月8日更新於 2026年10月8日儲存庫今天更新

Investigate a Datadog product usage or cost spike by correlating Usage Metering data (when/what spiked) with Audit Trail config changes (who changed what in the preceding window).

僅含說明DevOps & Cloud Business & Finance

AI 產生的概覽

將 Datadog 用量計量資料與 Audit Trail 設定變更相互關聯，以解釋成本或用量暴增。

功能: 引導一項調查：先查詢 Datadog 每小時用量計量資料，找出暴增發生的時間與所屬產品家族，再檢索此前時間範圍內的 Audit Trail 設定變更。它會把結果收斂到與暴增產品相關的稽核類別，並產出報告，指出可能的因果變更、信心水準與後續步驟。它辨識的是設定變更，而非提交資料的特定使用者或程序。
適用情境: 當某個 Datadog 產品的用量或成本意外上升，而你需要找出背後的設定變更時使用。適合需要把帳務資料與變更歷史連結起來的 FinOps 或維運調查。
執行需求: 需要 Datadog 存取權：透過 pup CLI 進行 OAuth2 登入以查詢稽核記錄，並提供 DD_API_KEY、DD_APP_KEY 與 DD_SITE 以查詢用量計量資料。需要連線至 Datadog API 的網路存取，以及 pup CLI 和 jq。此技能未附帶指令碼，僅為操作說明。

Audit Trail: Cost / Usage Spike Investigation

Identify what caused a Datadog usage spike by correlating billing data with configuration change history.

The causal chain is: someone changed something → that change increased data volume → usage spiked → cost went up. Usage Metering tells you when and what; Audit Trail tells you who made the change.

Prerequisites

bash

pup auth login   # OAuth2 (recommended) — covers audit queries# Usage Metering queries also need DD_API_KEY + DD_APP_KEYexport DD_API_KEY=<your-api-key>export DD_APP_KEY=<your-app-key>export DD_SITE=datadoghq.com

pup auth login   # OAuth2 (recommended) — covers audit queries# Usage Metering queries also need DD_API_KEY + DD_APP_KEYexport DD_API_KEY=<your-api-key>export DD_APP_KEY=<your-app-key>export DD_SITE=datadoghq.com

Scope Boundary

This skill identifies configuration changes that may have caused a spike. It does not identify which specific user or process submitted the data (e.g., which service sent the LLM spans). For per-submission attribution, use LLM Observability traces or APM instrumentation.

Investigation Workflow

Step 1 — Identify the spike window and product family

bash

START=$(date -u -v-7d +"%Y-%m-%dT%H:%M:%SZ" 2>/dev/null || date -u -d "7 days ago" +"%Y-%m-%dT%H:%M:%SZ")END=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
curl -s -G "https://api.${DD_SITE}/api/v2/usage/hourly_usage" \  -H "DD-API-KEY: ${DD_API_KEY}" \  -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \  --data-urlencode "filter[timestamp][start]=${START}" \  --data-urlencode "filter[timestamp][end]=${END}" \  --data-urlencode "filter[product_families]=all" \  | jq '[.data[] | {      timestamp: .attributes.timestamp,      product: .attributes.product_family,      measurements: [.attributes.measurements[] | {type: .usage_type, value: .value}]    }]'

START=$(date -u -v-7d +"%Y-%m-%dT%H:%M:%SZ" 2>/dev/null || date -u -d "7 days ago" +"%Y-%m-%dT%H:%M:%SZ")END=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
curl -s -G "https://api.${DD_SITE}/api/v2/usage/hourly_usage" \  -H "DD-API-KEY: ${DD_API_KEY}" \  -H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \  --data-urlencode "filter[timestamp][start]=${START}" \  --data-urlencode "filter[timestamp][end]=${END}" \  --data-urlencode "filter[product_families]=all" \  | jq '[.data[] | {      timestamp: .attributes.timestamp,      product: .attributes.product_family,      measurements: [.attributes.measurements[] | {type: .usage_type, value: .value}]    }]'

Product families with LLM/AI coverage: llm_observability, bits_ai, logs, apm

Step 2 — Pinpoint the spike

From Step 1, identify the hour/day where volume jumped. Note the timestamp as SPIKE_TIME.

Step 3 — Search Audit Trail for config changes in the 24h preceding the spike

bash

pup audit-logs search \  --query "@action:(created OR modified OR deleted)" \  --from "SPIKE_TIME_MINUS_24H" \  --to "SPIKE_TIME" \  --limit 200 \  -o json \  | jq '[.data[] | {      timestamp: .attributes.timestamp,      user: .attributes.attributes.usr.email,      actor_type: .attributes.attributes.evt.actor.type,      action: .attributes.attributes.action,      event_category: .attributes.attributes.evt.name,      resource_type: .attributes.attributes.asset.type,      resource_id: .attributes.attributes.asset.id    }]'

pup audit-logs search \  --query "@action:(created OR modified OR deleted)" \  --from "SPIKE_TIME_MINUS_24H" \  --to "SPIKE_TIME" \  --limit 200 \  -o json \  | jq '[.data[] | {      timestamp: .attributes.timestamp,      user: .attributes.attributes.usr.email,      actor_type: .attributes.attributes.evt.actor.type,      action: .attributes.attributes.action,      event_category: .attributes.attributes.evt.name,      resource_type: .attributes.attributes.asset.type,      resource_id: .attributes.attributes.asset.id    }]'

Note: --from and --to accept ISO timestamps (e.g., 2026-05-01T14:00:00Z) or relative values (1h, 24h, 7d).

Step 4 — Narrow to product-relevant config changes

Filter to the audit categories most likely to affect the spiking product:

If this product spiked	Add to query
`llm_observability`	`@evt.name:(Integration OR APM OR "Log Management")`
`logs` / `indexed_logs`	`@evt.name:"Log Management" @asset.type:(pipeline OR index OR exclusion_filter)`
`apm` / `indexed_spans`	`@evt.name:APM @asset.type:(retention_filter OR sampling_rate)`
`rum`	`@evt.name:RUM`
`metrics`	`@evt.name:Metrics`

Example for LLM Observability spike:

bash

pup audit-logs search \  --query "@evt.name:(Integration OR APM OR \"Log Management\") @action:(created OR modified)" \  --from "SPIKE_TIME_MINUS_24H" \  --to "SPIKE_TIME" \  --limit 100 \  -o json \  | jq '[.data[] | {      timestamp: .attributes.timestamp,      user: .attributes.attributes.usr.email,      action: .attributes.attributes.action,      category: .attributes.attributes.evt.name,      resource_type: .attributes.attributes.asset.type,      resource_id: .attributes.attributes.asset.id    }]'

pup audit-logs search \  --query "@evt.name:(Integration OR APM OR \"Log Management\") @action:(created OR modified)" \  --from "SPIKE_TIME_MINUS_24H" \  --to "SPIKE_TIME" \  --limit 100 \  -o json \  | jq '[.data[] | {      timestamp: .attributes.timestamp,      user: .attributes.attributes.usr.email,      action: .attributes.attributes.action,      category: .attributes.attributes.evt.name,      resource_type: .attributes.attributes.asset.type,      resource_id: .attributes.attributes.asset.id    }]'

Output Format

Usage spike detected:  Product: <product_family>  Spike time: <SPIKE_TIME>  Volume: <baseline> → <spike_value> (<magnitude>×)
Configuration changes in 24h preceding spike:  <timestamp> | <user_email> | <action> <resource_type> <resource_id> | <category>
Likely causal change: <most-proximate change matching the product family>
Confidence: HIGH (single clear change) / MEDIUM (multiple candidates) / LOW (no matching changes)
Next steps:  - Confirm with <user_email> whether the change was intentional  - If unintentional: revert <resource_id> and monitor volume  - If intentional: update cost forecasts and alert thresholds

Usage spike detected:  Product: <product_family>  Spike time: <SPIKE_TIME>  Volume: <baseline> → <spike_value> (<magnitude>×)
Configuration changes in 24h preceding spike:  <timestamp> | <user_email> | <action> <resource_type> <resource_id> | <category>
Likely causal change: <most-proximate change matching the product family>
Confidence: HIGH (single clear change) / MEDIUM (multiple candidates) / LOW (no matching changes)
Next steps:  - Confirm with <user_email> whether the change was intentional  - If unintentional: revert <resource_id> and monitor volume  - If intentional: update cost forecasts and alert thresholds

When No Causal Change Is Found

The change may predate the 24h window — expand to 72h
The increase may be from application-side instrumentation changes — check deploys
The increase may be organic traffic growth — correlate with product launch or traffic event

References

來源與署名

來源：datadog-labs/agent-skills位於dd-audit/cost-spike-investigation提交5b40c73

授權條款：無授權條款

內容歸原作者所有。SourceWeft 從公開儲存庫中收錄這些內容。

檢舉或申請下架

Dd Audit Cost Spike Investigation

Audit Trail: Cost / Usage Spike Investigation

Prerequisites

Scope Boundary

Investigation Workflow

Step 1 — Identify the spike window and product family

Step 2 — Pinpoint the spike

Step 3 — Search Audit Trail for config changes in the 24h preceding the spike

Step 4 — Narrow to product-relevant config changes

Output Format

When No Causal Change Is Found

References

來源與署名

更多來自 datadog-labs/agent-skills 的技能

Upgrade Browser Sdk V7

Upgrade Browser Sdk V6

Dd Audit Security Investigation

Upgrade Browser Sdk V5

Dd Audit Key Compromise

Troubleshoot Ssi

更多DevOps & Cloud技能

Playwright Devops

M5 Onboard

Yeet

Runbook

Incident Response

Deploy Checklist