Core Dumps

mohitmishra786/low-level-dev-skills/skills/debuggers/core-dumps

by mohitmishra786bdc58472fa9fNo license253 starsListed Oct 9, 2026Updated Oct 9, 2026Repository updated 3 months ago

Core dump analysis skill for production crash triage. Use when loading core files in GDB or LLDB, enabling core dump generation on Linux/macOS, mapping symbols with debuginfo or debuginfod, or extracting backtraces from crashes without re-running the program. Activates on queries about core files, ulimit, coredumpctl, debuginfod, crash triage, or analyzing segfaults from production binaries.

Instructions onlySoftware Development DevOps & Cloud

AI-generated overview

Guides agents through enabling, collecting and analysing core dumps for post-mortem crash triage with GDB, LLDB and debuginfod.

What it does: This skill provides step-by-step instructions for enabling core dump generation on Linux and macOS, including ulimit settings, kernel core patterns and systemd coredumpctl. It covers loading core files in GDB and LLDB, extracting backtraces, registers and locals, and resolving missing symbols through debuginfod or manual debug packages. It also describes separating debug info with objcopy and running non-interactive batch triage. A reference cheatsheet accompanies the workflow.
When to use it: Use it when a program has crashed in production and you need to analyse a core file without re-running the program. It fits questions about core files, ulimit, coredumpctl, debuginfod, crash triage or segfault analysis from production binaries.
Requirements: Requires a debugger such as GDB or LLDB, and on Linux optionally systemd coredumpctl, debuginfod client tools, binutils (objcopy, readelf) and elfutils. Network access may be needed to fetch symbols from debuginfod servers. It ships no scripts, only instructions and a reference cheatsheet.

Core Dumps

Purpose

Guide agents through enabling, collecting, and analysing core dumps for post-mortem crash investigation without rerunning the buggy program.

Triggers

"My program crashed in production — how do I analyse the core?"
"How do I enable core dumps on Linux?"
"I have a core file but no symbols / source"
"How do I use debuginfod to get symbols for a core?"
"coredumpctl show me the crash"

Workflow

1. Enable core dumps (Linux)

bash

# Per-session (lost on logout)ulimit -c unlimited
# Persistent (add to /etc/security/limits.conf)*   soft   core   unlimited*   hard   core   unlimited
# Check current limitulimit -c
# Set core pattern (where and how cores are named)# Default: 'core' in CWD — often not usefulsudo sysctl -w kernel.core_pattern=/tmp/core-%e-%p-%t# %e = executable, %p = PID, %t = timestamp
# Persistent (add to /etc/sysctl.d/99-core.conf)kernel.core_pattern=/tmp/core-%e-%p-%tkernel.core_uses_pid=1

# Per-session (lost on logout)ulimit -c unlimited
# Persistent (add to /etc/security/limits.conf)*   soft   core   unlimited*   hard   core   unlimited
# Check current limitulimit -c
# Set core pattern (where and how cores are named)# Default: 'core' in CWD — often not usefulsudo sysctl -w kernel.core_pattern=/tmp/core-%e-%p-%t# %e = executable, %p = PID, %t = timestamp
# Persistent (add to /etc/sysctl.d/99-core.conf)kernel.core_pattern=/tmp/core-%e-%p-%tkernel.core_uses_pid=1

2. systemd/coredumpctl (modern Linux)

If systemd manages core dumps (common on Ubuntu 20+, Fedora, Arch):

bash

# List recent crashescoredumpctl list
# Show details of the latest crashcoredumpctl info
# Load latest crash in GDBcoredumpctl gdb
# Load specific PID crashcoredumpctl gdb 12345
# Export core filecoredumpctl dump -o myapp.core PID

# List recent crashescoredumpctl list
# Show details of the latest crashcoredumpctl info
# Load latest crash in GDBcoredumpctl gdb
# Load specific PID crashcoredumpctl gdb 12345
# Export core filecoredumpctl dump -o myapp.core PID

Core storage location: /var/lib/systemd/coredump/.

3. Enable core dumps (macOS)

bash

# macOS uses /cores by default (must be root-writable)ulimit -c unlimited
# Checkls /cores/
# launchd-launched services: set in plist# <key>HardResourceLimits</key># <dict><key>Core</key><integer>9223372036854775807</integer></dict>

# macOS uses /cores by default (must be root-writable)ulimit -c unlimited
# Checkls /cores/
# launchd-launched services: set in plist# <key>HardResourceLimits</key># <dict><key>Core</key><integer>9223372036854775807</integer></dict>

4. Analyse a core with GDB

bash

# Load binary and coregdb ./prog core.12345
# If the binary was stripped, provide the unstripped copygdb ./prog-with-symbols core.12345
# Essential first commands(gdb) bt                    # call stack(gdb) bt full               # stack + locals(gdb) info registers        # CPU state at crash(gdb) frame 2               # jump to interesting frame(gdb) info locals           # local variables in frame(gdb) print ptr             # inspect a pointer
# All threads (multi-threaded crash)(gdb) thread apply all bt full

# Load binary and coregdb ./prog core.12345
# If the binary was stripped, provide the unstripped copygdb ./prog-with-symbols core.12345
# Essential first commands(gdb) bt                    # call stack(gdb) bt full               # stack + locals(gdb) info registers        # CPU state at crash(gdb) frame 2               # jump to interesting frame(gdb) info locals           # local variables in frame(gdb) print ptr             # inspect a pointer
# All threads (multi-threaded crash)(gdb) thread apply all bt full

5. Analyse a core with LLDB

bash

lldb ./prog -c core.12345
# Orlldb(lldb) target create ./prog --core core.12345
# Commands(lldb) bt(lldb) thread backtrace all(lldb) frame select 2(lldb) frame variable

lldb ./prog -c core.12345
# Orlldb(lldb) target create ./prog --core core.12345
# Commands(lldb) bt(lldb) thread backtrace all(lldb) frame select 2(lldb) frame variable

6. Missing symbols: debuginfod

debuginfod serves debug symbols from a central server, mapping build IDs to DWARF data.

bash

# Install client (Debian/Ubuntu)sudo apt install debuginfod
# Enable (add to ~/.bashrc or /etc/environment)export DEBUGINFOD_URLS="https://debuginfod.ubuntu.com https://debuginfod.elfutils.org"
# GDB auto-fetches symbols when DEBUGINFOD_URLS is setgdb ./prog core
# Manually querydebuginfod-find debuginfo <build-id>debuginfod-find source <build-id> /path/to/file.c

# Install client (Debian/Ubuntu)sudo apt install debuginfod
# Enable (add to ~/.bashrc or /etc/environment)export DEBUGINFOD_URLS="https://debuginfod.ubuntu.com https://debuginfod.elfutils.org"
# GDB auto-fetches symbols when DEBUGINFOD_URLS is setgdb ./prog core
# Manually querydebuginfod-find debuginfo <build-id>debuginfod-find source <build-id> /path/to/file.c

7. Missing symbols: manual approach

bash

# Check if binary has a build IDreadelf -n ./prog | grep Build
# Find the correct debug package# Debian: apt install prog-dbg or prog-dbgsym# RPM: dnf install prog-debuginfo
# Point GDB to debug symbols directory(gdb) set debug-file-directory /usr/lib/debug
# Or use eu-readelf to dump build ID, then find .debug fileeu-readelf -n ./progfind /usr/lib/debug -name "*.debug" | xargs eu-readelf -n 2>/dev/null | grep <build-id>

# Check if binary has a build IDreadelf -n ./prog | grep Build
# Find the correct debug package# Debian: apt install prog-dbg or prog-dbgsym# RPM: dnf install prog-debuginfo
# Point GDB to debug symbols directory(gdb) set debug-file-directory /usr/lib/debug
# Or use eu-readelf to dump build ID, then find .debug fileeu-readelf -n ./progfind /usr/lib/debug -name "*.debug" | xargs eu-readelf -n 2>/dev/null | grep <build-id>

8. Strip binaries and keep symbols

Best practice: build with symbols, strip for distribution, keep an unstripped copy.

bash

# Buildgcc -g -O2 -o prog main.c
# Separate debug infoobjcopy --only-keep-debug prog prog.debugobjcopy --strip-debug prog prog.stripped
# Add a debuglink so GDB finds the debug file automaticallyobjcopy --add-gnu-debuglink=prog.debug prog.stripped
# Deploy prog.stripped; keep prog.debug in a symbols store indexed by build-id

# Buildgcc -g -O2 -o prog main.c
# Separate debug infoobjcopy --only-keep-debug prog prog.debugobjcopy --strip-debug prog prog.stripped
# Add a debuglink so GDB finds the debug file automaticallyobjcopy --add-gnu-debuglink=prog.debug prog.stripped
# Deploy prog.stripped; keep prog.debug in a symbols store indexed by build-id

9. Quick triage from core without full debug session

bash

# Print backtrace non-interactivelygdb -batch -ex 'bt full' -ex 'thread apply all bt full' ./prog core 2>&1 | tee crash.txt
# Print registersgdb -batch -ex 'info registers' ./prog core
# Check signal that caused crashgdb -batch -ex 'info signal' ./prog core

# Print backtrace non-interactivelygdb -batch -ex 'bt full' -ex 'thread apply all bt full' ./prog core 2>&1 | tee crash.txt
# Print registersgdb -batch -ex 'info registers' ./prog core
# Check signal that caused crashgdb -batch -ex 'info signal' ./prog core

For a full cheatsheet covering core pattern tokens, coredumpctl, GDB/LLDB commands, debuginfod servers, and strip/symbol workflows, see references/cheatsheet.md [blocked].

Related skills

Use skills/debuggers/gdb for full GDB session details
Use skills/debuggers/lldb for LLDB-based analysis
Use skills/runtimes/sanitizers to catch the bug before it reaches production
Use skills/binaries/elf-inspection for readelf, build IDs, and binary inspection

Source and attribution

Source:mohitmishra786/low-level-dev-skillsinskills/debuggers/core-dumpsat commitbdc5847

License: No license

Content belongs to its original authors. SourceWeft indexes it from a public repository.

Report or request removal

Core Dumps

Core Dumps

Purpose

Triggers

Workflow

1. Enable core dumps (Linux)

2. systemd/coredumpctl (modern Linux)

3. Enable core dumps (macOS)

4. Analyse a core with GDB

5. Analyse a core with LLDB

6. Missing symbols: debuginfod

7. Missing symbols: manual approach

8. Strip binaries and keep symbols

9. Quick triage from core without full debug session

Related skills

Source and attribution

More from mohitmishra786/low-level-dev-skills

Zig Debugging

Simd Intrinsics

Zig Cross

Rustc Basics

Zig Cinterop

Zig Build System

More in Software Development

Playground

M5 Onboard

Cardputer Buddy

Web Design Guidelines

Vercel React Native Skills

Vercel React Best Practices