Dd Monitors

by datadog-labs5b40c73824ecNo license177 starsListed Oct 8, 2026Updated Oct 8, 2026Repository updated today

Monitor management - list, search, file-based create, and alerting best practices.

AI-generated overview

Manage Datadog monitors with pup: list, search, create from files, silence via downtime, and follow alerting best practices.

What it does: This skill provides instructions for managing Datadog monitors using the pup command-line tool. It covers listing and getting monitors, creating monitors from JSON files, and silencing alerts through downtime payloads. It also documents alerting best practices such as stable evaluation windows, proper scoping, recovery thresholds, and runbook links, plus a safe workflow that marks monitors for deletion instead of deleting them.
When to use it: Use it when you need to create, inspect, or maintain Datadog monitors and alerting rules. It is also useful for auditing monitors for missing owners or noisy alerts, and for silencing notifications during planned windows.
Requirements: Requires the pup CLI available in PATH and Datadog authentication via pup auth login. Network access to Datadog is needed. No scripts are shipped; the skill is instructions only.

Datadog Monitors

Create, manage, and maintain monitors for alerting.

Prerequisites

This requires pup in your path. See Setup Pup.

Command Execution Order (Token-Efficient)

For scoped commands, use this order:

Check context first (prior outputs, conversation, saved values).
If a required value is missing, run a discovery command first.
If still ambiguous, ask the user to confirm.
Then run the target command.
Avoid speculative commands likely to fail.

Quick Start

Common Operations

List Monitors

Get Monitor

Create Monitor

Silence Alerts (Downtime)

bash

# No pup monitors mute/unmute commands.# Use downtime payloads to silence monitor notifications.pup downtime create --file downtime.jsonpup downtime cancel <downtime_id>

# No pup monitors mute/unmute commands.# Use downtime payloads to silence monitor notifications.pup downtime create --file downtime.jsonpup downtime cancel <downtime_id>

Monitor Creation Best Practices

1. Avoid Alert Fatigue

python

# WRONG - will flap constantlyquery = "avg(last_1m):avg:system.cpu.user{*} > 50"  # ❌ Too sensitive
# CORRECT - stable alertingquery = "avg(last_5m):avg:system.cpu.user{env:prod} by {host} > 80"  # ✅ Reasonable window

# WRONG - will flap constantlyquery = "avg(last_1m):avg:system.cpu.user{*} > 50"  # ❌ Too sensitive
# CORRECT - stable alertingquery = "avg(last_5m):avg:system.cpu.user{env:prod} by {host} > 80"  # ✅ Reasonable window

2. Use Proper Scoping

python

# WRONG - alerts on everythingquery = "avg(last_5m):avg:system.cpu.user{*} > 80"  # ❌ No scope
# CORRECT - scoped to what mattersquery = "avg(last_5m):avg:system.cpu.user{env:prod,service:api} by {host} > 80"  # ✅

# WRONG - alerts on everythingquery = "avg(last_5m):avg:system.cpu.user{*} > 80"  # ❌ No scope
# CORRECT - scoped to what mattersquery = "avg(last_5m):avg:system.cpu.user{env:prod,service:api} by {host} > 80"  # ✅

3. Set Recovery Thresholds

python

monitor = {    "query": "avg(last_5m):avg:system.cpu.user{env:prod} > 80",    "options": {        "thresholds": {            "critical": 80,            "critical_recovery": 70,  # ✅ Prevents flapping            "warning": 60,            "warning_recovery": 50        }    }}

monitor = {    "query": "avg(last_5m):avg:system.cpu.user{env:prod} > 80",    "options": {        "thresholds": {            "critical": 80,            "critical_recovery": 70,  # ✅ Prevents flapping            "warning": 60,            "warning_recovery": 50        }    }}

4. Include Context in Messages

python

message = """## High CPU Alert
Host: {{host.name}}Current Value: {{value}}Threshold: {{threshold}}
### Runbook1. Check top processes: `ssh {{host.name}} 'top -bn1 | head -20'`2. Check recent deploys3. Scale if needed
@slack-ops @pagerduty-oncall"""

message = """## High CPU Alert
Host: {{host.name}}Current Value: {{value}}Threshold: {{threshold}}
### Runbook1. Check top processes: `ssh {{host.name}} 'top -bn1 | head -20'`2. Check recent deploys3. Scale if needed
@slack-ops @pagerduty-oncall"""

NEVER Delete Monitors Directly

Use safe deletion workflow (same as dashboards):

python

def safe_mark_monitor_for_deletion(monitor_id: str, client) -> bool:    """Mark monitor instead of deleting."""    monitor = client.get_monitor(monitor_id)    name = monitor.get("name", "")        if "[MARKED FOR DELETION]" in name:        print(f"Already marked: {name}")        return False        new_name = f"[MARKED FOR DELETION] {name}"    client.update_monitor(monitor_id, {"name": new_name})    print(f"✓ Marked: {new_name}")    return True

def safe_mark_monitor_for_deletion(monitor_id: str, client) -> bool:    """Mark monitor instead of deleting."""    monitor = client.get_monitor(monitor_id)    name = monitor.get("name", "")        if "[MARKED FOR DELETION]" in name:        print(f"Already marked: {name}")        return False        new_name = f"[MARKED FOR DELETION] {name}"    client.update_monitor(monitor_id, {"name": new_name})    print(f"✓ Marked: {new_name}")    return True

Monitor Types

Type	Use Case
`metric alert`	CPU, memory, custom metrics
`query alert`	Complex metric queries
`service check`	Agent check status
`event alert`	Event stream patterns
`log alert`	Log pattern matching
`composite`	Combine multiple monitors
`apm`	APM metrics

Audit Monitors

bash

# Find monitors without ownerspup monitors list | jq '.[] | select(.tags | contains(["team:"]) | not) | {id, name}'
# Find noisy monitors (high alert count)pup monitors list | jq 'sort_by(.overall_state_modified) | .[:10] | .[] | {id, name, status: .overall_state}'

# Find monitors without ownerspup monitors list | jq '.[] | select(.tags | contains(["team:"]) | not) | {id, name}'
# Find noisy monitors (high alert count)pup monitors list | jq 'sort_by(.overall_state_modified) | .[:10] | .[] | {id, name, status: .overall_state}'

Downtime vs Muting

Failure Handling

References

Source and attribution

Source:datadog-labs/agent-skillsindd-monitorsat commit5b40c73

License: No license

Content belongs to its original authors. SourceWeft indexes it from a public repository.

Report or request removal

Rule	Why
No flapping alerts	Use `last_Xm` not `last_1m`
Meaningful thresholds	Based on SLOs, not guesses
Actionable alerts	If no action needed, don't alert
Include runbook	`@runbook-url` in message

Problem	Fix
Alert not firing	Check query returns data, thresholds
Too many alerts	Increase window, add recovery threshold
No data alerts	Check agent connectivity, metric exists
Auth error	`pup auth refresh`

Dd Monitors

Datadog Monitors

Prerequisites

Command Execution Order (Token-Efficient)

Quick Start

Common Operations

List Monitors

Get Monitor

Create Monitor

Silence Alerts (Downtime)

Monitor Creation Best Practices

1. Avoid Alert Fatigue

2. Use Proper Scoping

3. Set Recovery Thresholds

4. Include Context in Messages

NEVER Delete Monitors Directly

Monitor Types

Audit Monitors

Downtime vs Muting

Failure Handling

References

Source and attribution

More from datadog-labs/agent-skills

Upgrade Browser Sdk V7

Upgrade Browser Sdk V6

Dd Audit Security Investigation

Upgrade Browser Sdk V5

Dd Audit Key Compromise

Dd Audit Cost Spike Investigation

More in DevOps & Cloud

Playwright Devops

M5 Onboard

Yeet

Runbook

Incident Response

Deploy Checklist

Use	When
Downtime	Any planned silence window
Monitor edit	Query/threshold behavior changes