lastwords

[lastwords]

[Artifact Hub] [Release] [License]

lastwords

Memory snapshots before OOM kills, and logs and core dumps from crashes, for Kubernetes.

lastwords runs on every node. When a container gets close to its memory limit, it records what is using the memory. When a container dies, it saves the logs of that instance, the pod's events and any core dump. Everything stays on the node (or in S3), and you can read it with kubectl lastwords or from an AI agent over MCP.

helm install lastwords oci://ghcr.io/arvo-ai/charts/lastwords -n lastwords --create-namespace

The problem

When you look at a pod that died hours ago, most of the evidence is gone:

• OOMKilled leaves no heap dump. The kernel kills the process with SIGKILL. -XX:+HeapDumpOnOutOfMemoryError never fires, and Go, Node and Python get no warning either.
• kubectl logs --previous only keeps one restart. After a few crash-loop restarts, the logs from the first failure are gone.
• Events expire after one hour (--event-ttl, default 1h).
• Core dumps go nowhere. core_pattern is set per node, so crashing containers either dump nothing or dump to a path inside a container that is about to be deleted.

None of it can be recovered later. It has to be collected while the container is alive or right after it dies, which is what lastwords does.

What it captures

Every capture comes with a summary.md that says what happened in plain words:

# checkout-5498f94695-526hm/checkout was OOMKilled at 16:51:38 UTC (limit 384.0 MiB); 1 snapshot taken before the kill
## What lastwords found- Memory climbed steadily for 1m1s, the pattern of a leak or an unbounded cache. Raising the limit would only delay the next kill.- Before the kill (67% of limit): Largest application classes on the heap:  com.example.checkout.CheckoutService$CartSnapshot: 2860 instances ...- Before the kill (67% of limit): One class dominates the Java heap: byte[] holds 179.2 MiB (97%).

# order-consumer-76d5fcb94-8vm4p/consumer crashed at 03:09:30 UTC: Error, exit code 2
## What lastwords found- Crashed in main.registerDiscount at /src/main.go:19 (assignment to entry in nil map).

Use it

kubectl lastwords list                      # everything from the last 7 dayskubectl lastwords list -n shop --kind oomkillkubectl lastwords show 20261102T030712Z-af14b5kubectl lastwords get 20261102T030712Z-af14b5 jvm-class-histogram.txtkubectl lastwords top                       # containers closest to their memory limit right now

Install the CLI (it is also a kubectl plugin):

# macOS / Linuxcurl -sSL https://github.com/Arvo-AI/lastwords/releases/latest/download/lastwords_$(uname -s)_$(uname -m | sed s/aarch64/arm64/).tar.gz | tar xzsudo mv lastwords /usr/local/bin/ && sudo ln -sf /usr/local/bin/lastwords /usr/local/bin/kubectl-lastwords

The CLI talks to the agents through the Kubernetes API server, so it needs nothing but your kubeconfig.

Guides

Measured on EKS with the failures in this repo; every command and output is from a real run.

• Get a Java heap histogram before Kubernetes OOMKills the pod
• Why -XX:+HeapDumpOnOutOfMemoryError never writes a file in Kubernetes
• Java pod OOMKilled at 80% of its memory limit
• Get core dumps from crashing Kubernetes pods
• Find what is leaking in a Go pod before it is OOMKilled

Agent eval: setup, grader and every transcript from 24 runs with and without lastwords.

Let your AI agent read it

lastwords ships an MCP server. Claude Code, Codex, Cursor and any MCP client can list and read captures:

claude mcp add lastwords -- kubectl-lastwords mcpcodex mcp add lastwords -- kubectl-lastwords mcp

Then ask: "the checkout pods restarted overnight, why?" Tools: lastwords_list_captures, lastwords_get_capture, lastwords_read_file, lastwords_watchlist. All read-only.

Get told

helm upgrade lastwords oci://ghcr.io/arvo-ai/charts/lastwords -n lastwords --reuse-values \  --set notifications.slackWebhookUrl=https://hooks.slack.com/services/... \  --set notifications.webhookUrl=https://your-service/lastwords \  --set notifications.webhookSecret=$(openssl rand -hex 32)

With a secret set, every webhook delivery is signed: X-Lastwords-Signature: sha256=<HMAC-SHA256 of "<X-Lastwords-Timestamp>.<raw body>">. Verify it before acting on the event, and reject old timestamps:

import hmac, hashlib, time
def verify(secret: bytes, headers, body: bytes, tolerance=300) -> bool:    ts = headers["X-Lastwords-Timestamp"]    if abs(time.time() - int(ts)) > tolerance:        return False    want = "sha256=" + hmac.new(secret, ts.encode() + b"." + body, hashlib.sha256).hexdigest()    return hmac.compare_digest(want, headers["X-Lastwords-Signature"])

Go receivers can use github.com/Arvo-AI/lastwords/webhook.Verify.

Every capture also becomes a Kubernetes Event on the pod (LastWordsOOMKill, LastWordsMemoryPressure, LastWordsCrash, ...), so it shows up in kubectl describe pod and anything that watches events. Prometheus metrics are on :9095/metrics.

Configuration

Common settings (see values.yaml for all):

Pod annotations:

Uninstall

helm uninstall lastwords -n lastwords

On shutdown the agent puts the node's original core_pattern back (checked on EKS: systemd-coredump again after uninstall). Captures and the core helper binary stay in /var/lib/lastwords on each node, so a reinstall keeps your history, and a node whose agent was killed before it could restore core_pattern still has a working helper to hand cores to. Delete that directory on the nodes to remove everything.

How it works

A DaemonSet runs one small agent per node (a static Go binary; requests 20m CPU and 48 MiB). Measured on EKS (t3.large, Amazon Linux 2023): 8m CPU and 12 MiB on a quiet node, 20m CPU and 15 MiB with 35 containers, about 0.5m CPU per container watched.

1. It reads every container's memory accounting from the node's cgroup tree twice a second (memory.current, memory.stat, memory.events, PSI), computes the working set the way the kubelet does, and keeps a 10-minute timeline.
2. When a container crosses the threshold, or is climbing fast enough to hit its limit within 15 seconds, it collects evidence while the process is still alive. For the JVM it uses the HotSpot attach protocol (what jcmd uses) from outside the container; nothing is installed in your images. For Go it pulls /debug/pprof/heap.
3. It watches pods on its node. When a container terminates with an error, it copies the dead instance's log file before the next restart rotates it, and saves the pod's events.
4. With core dumps on, it points kernel.core_pattern at a small helper that writes the gzipped core next to the captures, and the agent matches it to the pod.
5. Captures stay on the node's disk under a size and age budget, optionally upload to S3, and are served to the CLI and MCP server through the API server's pod proxy.

More in docs/how-it-works.md.

Security

lastwords reads other processes' memory accounting, so it needs node-level access. Read docs/security.md before installing it in production. In short:

• Without core dumps the agent is not privileged. It runs with hostPID and the capabilities SYS_PTRACE, DAC_READ_SEARCH, DAC_OVERRIDE, CHOWN, FOWNER, KILL, plus a read-only root filesystem.
• Core dumps need privileged to set kernel.core_pattern. They are off by default.
• Full heap dumps are off by default because they contain whatever was in memory. The default class histogram contains class names and counts, not data.
• Command lines are redacted for arguments that look like secrets.
• Reading captures needs RBAC on pods/proxy in the lastwords namespace plus the API token secret. The chart creates a lastwords-reader Role.

FAQ

My pod was OOMKilled. How do I find out what used the memory? If lastwords was running, kubectl lastwords list --kind oomkill shows the kill, and the linked pressure capture shows what was using memory before it. If it wasn't running, nothing can recover that information. Install it so the next kill is captured.

Why didn't HeapDumpOnOutOfMemoryError produce a heap dump? That flag fires when the JVM itself runs out of heap. A container OOM kill is the kernel sending SIGKILL when the whole process (heap plus metaspace, threads, direct buffers, native memory) hits the cgroup limit. The JVM never gets a chance to run. lastwords takes the histogram before that point.

How do I get core dumps from Kubernetes pods? helm upgrade lastwords ... --set coreDumps.enabled=true. Cores are stored per capture as core.gz. Debug them with gdb and the same binary from the image.

kubectl logs --previous shows nothing useful. Where are the logs of the first crash? The kubelet only keeps the last terminated instance. lastwords copies each dead instance's log at the moment it dies (the first 3 crashes of a crash loop per 30 minutes, to bound disk use).

Does it replace Prometheus, Datadog or continuous profiling? No. Continuous profilers show where memory is allocated over time. lastwords takes a snapshot right before the kill without changes to your app, and keeps the logs and events that would otherwise expire.

Does it work on EKS, GKE, AKS, OpenShift? It works on any Kubernetes 1.25+ node with cgroup v2 (cgroup v1 is supported for memory too) and containerd or CRI-O. Core dumps need a node path the kernel can execute the helper from; see docs/platforms.md for per-platform notes.

Aurora

lastwords is built by the team behind Aurora, an open-source AI SRE that investigates incidents. Point notifications.webhookUrl at Aurora, or give Aurora the MCP server, and every OOM kill and crash arrives with its evidence attached.

License

Apache 2.0