EZO Watcher

The EZO Watcher wall in one answer: how many incidents are firing (by severity, and by who has them: unattended, acknowledged, silenced), how many are parked (a fix or a verdict that is not due yet: not live, never paged, counted in no firing or unattended number), every incident as one row (key, title, severity, attention: none, acked, silenced or parked, who acknowledged or silenced it or why, by whom and until when it is parked, age, comment count, link), and the sources that have gone stale, are erroring, or are configured but not started on this host (`skipped`, with its reason). Start here. The incident `key` it returns is what every other tool takes.

Takes no parameters.

Incidents, filtered. `state`: firing (default: every live incident, parked ones left out), unattended (firing, nobody has it, never a parked one), acknowledged, silenced, parked (a fix or a verdict that is not due yet: `parkedState`, `parkedUntil`, `parkedBy`, `parkedRef`, `parkedNote` say why, by whom and until when), resolved (episodes that ended in the last `hours`), or all (every incident, parked included). Filter by `severity`, `service` (substring) and `source` (the system that raised it: grafana, prod, sweep, netdata-cloud, ...). Resolved rows say whether a source resolved the incident or someone marked it resolved by hand, and who.

One incident, whole: its alerts (rule, host, labels, annotations, value, who reports it and when it last did, evidence links and commands), who acknowledged, silenced and parked it, how it ended if it ended (a source, or someone by hand, and any confirmation), its history of acknowledgements, silences, reopens and resolves with who did each, every comment, and the pager's last sends. Works for a firing incident and for one that resolved in the last 30 days. Raw payloads are left out.

One structured brief for fixing an incident without asking a person: the incident (summary, severity, first and last seen, state and who has it), the source and target that raised it (their config, with every credential left as the env-var NAME and never a value), the predicates the rule cell asserts and which one the last run failed, `discovered.foundAt`/`read` (where the endpoint was found in the project and what was read there), the last good observation beside the last bad one (bodies cut), a `reproduce` block with the exact curl/nc/command that re-runs the check locally, the other open incidents on the same service, the evidence links the source offers (a Sentry permalink and culprit, a Netdata chart, Grafana panel links), and `next`: fix, restart/deploy, then verify_fix. Reads only; start here when you are about to fix something rather than triage it.

What ended and what was done in a time range: every incident episode that resolved in it (when it started and ended, how long, who had it acknowledged, whether a source or someone by hand resolved it, any confirmation) and, newest first, the acknowledgements, silences, reopens and confirmations people and agents made, each with who. Give `from` and `to` as ISO times, or `hours` back from now (24 by default). Watch keeps 30 days.

Every source this Watcher reads or is pushed by (Grafana, vmalert, Netdata Cloud and the fleet load poll, Sentry, the HTTP and TCP probes, heartbeat, domain expiry, the feeds, the webhooks): its kind, state (ok, error, stale, idle), when it last answered, the error if any, `skipped` when this host did not start it (`missing <ENV>`; absent on a source that runs), how many alerts it reports firing, how many rules it lists, how many of its cells assert more than a nominal 200 (`asserted`) and, where it records one, `discovered`: where its endpoint was found. A source that has gone quiet or was not started means an all clear is not earned.

Takes no parameters.

Say you are working this incident: it shows as acknowledged by you on the tile, in the sheet and in History, its age stops counting as unattended, and the pager stops escalating it. Acknowledge only while you are actively working the issue; it does not fix, hide or resolve anything, and a person can take it off. Re-acknowledging replaces the earlier name. Put what you found or did in a comment, not here: the optional note is one line of context.

Take the acknowledgement (and any silence) off a firing incident, so it counts as unattended again and the pager resumes. Use it when you stop working an issue you acknowledged and no one else has it. `by` is recorded: History shows who handed it back.

Pause the pager for one firing incident, for at most 240 minutes (4 hours; more is refused, not shortened). It drops to the quiet tier of the wall and sends nothing until it runs out or someone takes it off. A silence hides a problem from the people who would act on it: use it only for something you know is expected right now (a deploy, a drill, a host you are rebooting), say why in `reason`, and never to make a real problem go quiet. If you are working the incident, acknowledge it instead.

Take a firing incident off the live wall into the Parked section, for at most 72 hours (more is refused, not shortened), once you have FIXED it or reached a VERDICT on it. `state`: fixed_undeployed (the fix is merged and not live yet), awaiting_source (the fix is live and the source's evaluation window has not caught up, so it still reports the alert), deferred (a person decided to handle it at a later date; the note says who and why). It works for any source, Grafana included. Unlike `resolve` it never reopens or pages while the park holds, and unlike `silence` it states a known cause: the tile reads "Parked until <time> by <you>" with your note, it counts as parked and not firing, and nothing is sent. It ends when the source resolves the alert, when `until` passes (the incident comes back unattended and its clock counts from `until`, so the pager treats it as a new problem), on `unpark`, or on a manual `resolve`; an acknowledgement or silence on the incident still wins over it, and a second park replaces the first. Park only after a fix or a verdict, never to hide a live problem you have not looked at or to quiet one you cannot explain. The park covers only the alerts firing when you park (get_incident lists their keys in `park.alertKeys`): an alert that joins the incident later (another host or instance, a new rule on it) ends the park at once, the incident comes back unattended and pages as a new problem, and History says which alert joined. An alert that resolves and fires again keeps the park.

End a park set by hand with `park` before it runs out: the incident is back in its severity tier, unattended and paging as a new problem would, with its clock starting now. Use it when the fix did not hold or the verdict was wrong. `by` is recorded: History shows who ended it. Refused for an incident that is not firing, one that was not parked by hand (a park the regression sweep states ends when the sweep stops stating it), and one whose park has already ended.

Add a comment to an incident, firing or resolved: what you found, a link, the commit that fixes it, what you ruled out. Markdown, at most 4000 characters (longer is refused, not cut). It is stored on the incident and shows in the incident sheet with your name and the time, and every one is written to the operator log. Comment as you learn things; a comment is the right place for findings, links and fix commits.

Ask the sources again instead of waiting for the next poll: with `incident` (an incident key), re-polls the sources that report its alerts and answers what each alert is now; with `alert` (an alert key such as grafana:1a2b3c4d), the same for one alert; with neither, re-polls every pull source and says how many alerts cleared. At most one poll per source every 10 seconds, so a repeat inside it is answered from the last poll. A pushing sender that takes requests (the regression sweep) is asked to run the check again: each firing alert it reports gets one request, an alert that already has an open one reuses it, and the answer's `pushed[].request` says where it stands (asked, picked up, waiting, ran, failed, cannot, or not picked up within 30 min); a repeat while it is open returns the same request, so it only reads where it stands. Any other pushing sender (webhooks) cannot be forced: the answer says when it last reported, when it reports next if it said, and when the alert clears if it stops sending it. It changes no alert except by the sources' own answers.

Close the fix loop: re-check the sources now, wait up to `timeoutSec` seconds for the source's fresh observation, and answer { resolved, observation, asserts }. When every alert of the incident is resolved by its source, it records the confirmation (the existing confirm_resolution path) with the note "verified by agent", so History reads "resolution confirmed by <you>" and the incident closes with the attribution. When the source still reports an alert firing it resolves nothing and says which assertion failed and why, so the fix can be repaired rather than guessed at. Never resolves an alert itself: only the source (or a person's resolve) ends an alert. Idempotent: after a green answer a repeat reads the resolved episode and records at most another confirmation.

Mark a firing incident resolved now, with a required reason, when you have fixed it and the source has not caught up (a push source repeats only every few hours). It leaves the wall, History records "marked resolved by <you>" with your reason, and the operator log records the write. It does not hide a live problem: if the source reports the alert as firing on its next poll or push, Watch reopens the incident as a NEW firing (unacknowledged, escalating, paging again) and records "reopened: source still firing after manual resolve by <you>". A manual resolve ends any acknowledgement or silence on the incident. You can never resolve, delete or edit an alert itself: only its source resolves alerts. After the source resolves, use confirm_resolution to record that you checked.

Record that an incident a source has already resolved is resolved: "resolution confirmed by <you>", with an optional note (the fix commit, the PR, an evidence link). It shows in the sheet and in History, and the operator log records the write. It changes no state, so it cannot hide anything: use it after you checked that the fix held. Refused for an incident that is firing, one that never resolved, and one that was marked resolved by hand while a source still reports it.