Ebpf

mohitmishra786/low-level-dev-skills/skills/observability/ebpf

by mohitmishra786bdc58472fa9fNo license253 starsListed Oct 9, 2026Updated Oct 9, 2026Repository updated 3 months ago

eBPF skill for Linux observability and networking. Use when writing eBPF programs with libbpf or bpftrace, attaching kprobes/tracepoints/XDP hooks, debugging verifier errors, working with eBPF maps, or achieving CO-RE portability across kernel versions. Activates on queries about eBPF, bpftool, bpftrace, XDP programs, libbpf, verifier errors, eBPF maps, or kernel tracing with BPF.

Instructions onlySoftware Development DevOps & Cloud

AI-generated overview

Guides writing, loading and debugging eBPF programs with libbpf, bpftrace and bpftool, covering maps, verifier errors, XDP and CO-RE.

What it does: This skill provides reference guidance for authoring, loading and debugging eBPF programs on Linux. It covers tool selection between bpftrace, libbpf and bpftool, eBPF map types, verifier error triage, XDP packet filtering, CO-RE portability, ring buffers, iterators and atomics. It produces code snippets, build and attach commands, and troubleshooting tables rather than executable artifacts.
When to use it: Use it when writing eBPF programs with libbpf or bpftrace, attaching kprobes, tracepoints or XDP hooks, debugging verifier errors, working with eBPF maps, or making programs portable across kernel versions with CO-RE.
Requirements: No scripts are shipped; it is instructions only. Following the guidance assumes a Linux system with eBPF support, plus tools such as clang, bpftool, bpftrace, libbpf, gcc and kernel BTF at /sys/kernel/btf/vmlinux, and typically root or elevated privileges to load programs.

eBPF

Purpose

Guide agents through writing, loading, and debugging eBPF programs using libbpf, bpftrace, and bpftool. Covers map types, program types, verifier errors, XDP networking, and CO-RE portability.

Triggers

"How do I write an eBPF program to trace system calls?"
"My eBPF program fails with a verifier error"
"How do I use bpftrace to trace kernel events?"
"How do I share data between kernel eBPF and userspace?"
"How do I write an XDP program for packet filtering?"
"How do I make my eBPF program portable across kernel versions (CO-RE)?"

Workflow

1. Choose the right tool

Goal?├── One-liner kernel tracing / scripting → bpftrace├── Production eBPF program with userspace → libbpf (C) or aya (Rust)├── Inspect loaded programs and maps → bpftool└── High-performance packet processing → XDP + libbpf

Goal?├── One-liner kernel tracing / scripting → bpftrace├── Production eBPF program with userspace → libbpf (C) or aya (Rust)├── Inspect loaded programs and maps → bpftool└── High-performance packet processing → XDP + libbpf

2. bpftrace — quick kernel tracing

bash

# Trace all execve calls with comm and argsbpftrace -e 'tracepoint:syscalls:sys_enter_execve { printf("%s %s\n", comm, str(args->filename)); }'
# Count syscalls by processbpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'
# Latency histogram for read() syscallbpftrace -e '  tracepoint:syscalls:sys_enter_read { @start[tid] = nsecs; }  tracepoint:syscalls:sys_exit_read  { @us = hist((nsecs - @start[tid]) / 1000); delete(@start[tid]); }'
# List available tracepointsbpftrace -l 'tracepoint:syscalls:*'bpftrace -l 'kprobe:tcp_*'

# Trace all execve calls with comm and argsbpftrace -e 'tracepoint:syscalls:sys_enter_execve { printf("%s %s\n", comm, str(args->filename)); }'
# Count syscalls by processbpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'
# Latency histogram for read() syscallbpftrace -e '  tracepoint:syscalls:sys_enter_read { @start[tid] = nsecs; }  tracepoint:syscalls:sys_exit_read  { @us = hist((nsecs - @start[tid]) / 1000); delete(@start[tid]); }'
# List available tracepointsbpftrace -l 'tracepoint:syscalls:*'bpftrace -l 'kprobe:tcp_*'

3. libbpf skeleton — minimal C program

// counter.bpf.c — kernel-side#include <vmlinux.h>#include <bpf/bpf_helpers.h>
struct {    __uint(type, BPF_MAP_TYPE_HASH);    __type(key, u32);    __type(value, u64);    __uint(max_entries, 1024);} call_count SEC(".maps");
SEC("tracepoint/syscalls/sys_enter_read")int trace_read(struct trace_event_raw_sys_enter *ctx){    u32 pid = bpf_get_current_pid_tgid() >> 32;    u64 *cnt = bpf_map_lookup_elem(&call_count, &pid);    if (cnt)        (*cnt)++;    else {        u64 one = 1;        bpf_map_update_elem(&call_count, &pid, &one, BPF_ANY);    }    return 0;}
char LICENSE[] SEC("license") = "GPL";

// counter.bpf.c — kernel-side#include <vmlinux.h>#include <bpf/bpf_helpers.h>
struct {    __uint(type, BPF_MAP_TYPE_HASH);    __type(key, u32);    __type(value, u64);    __uint(max_entries, 1024);} call_count SEC(".maps");
SEC("tracepoint/syscalls/sys_enter_read")int trace_read(struct trace_event_raw_sys_enter *ctx){    u32 pid = bpf_get_current_pid_tgid() >> 32;    u64 *cnt = bpf_map_lookup_elem(&call_count, &pid);    if (cnt)        (*cnt)++;    else {        u64 one = 1;        bpf_map_update_elem(&call_count, &pid, &one, BPF_ANY);    }    return 0;}
char LICENSE[] SEC("license") = "GPL";

// counter.c — userspace loader#include "counter.skel.h"
int main(void) {    struct counter_bpf *skel = counter_bpf__open();    if (!skel || counter_bpf__load(skel))        return 1;    counter_bpf__attach(skel);    // read map, print results    counter_bpf__destroy(skel);}

// counter.c — userspace loader#include "counter.skel.h"
int main(void) {    struct counter_bpf *skel = counter_bpf__open();    if (!skel || counter_bpf__load(skel))        return 1;    counter_bpf__attach(skel);    // read map, print results    counter_bpf__destroy(skel);}

bash

# Build with libbpf 1.xclang -g -O2 -target bpf -D__TARGET_ARCH_x86 -I/usr/include/bpf \      -c counter.bpf.c -o counter.bpf.obpftool gen skeleton counter.bpf.o > counter.skel.hgcc -o counter counter.c -lbpf -lelf -lz

# Build with libbpf 1.xclang -g -O2 -target bpf -D__TARGET_ARCH_x86 -I/usr/include/bpf \      -c counter.bpf.c -o counter.bpf.obpftool gen skeleton counter.bpf.o > counter.skel.hgcc -o counter counter.c -lbpf -lelf -lz

libbpf 1.x API changes:

// Open and load (replaces older bpf_object__open/load split patterns)struct counter_bpf *skel = counter_bpf__open();counter_bpf__load(skel);counter_bpf__attach(skel);
// Or explicit file openstruct bpf_object *obj = bpf_object__open_file("counter.bpf.o", NULL);bpf_object__load(obj);
// Skeleton generation (always via bpftool)// bpftool gen skeleton counter.bpf.o name counter > counter.skel.h

// Open and load (replaces older bpf_object__open/load split patterns)struct counter_bpf *skel = counter_bpf__open();counter_bpf__load(skel);counter_bpf__attach(skel);
// Or explicit file openstruct bpf_object *obj = bpf_object__open_file("counter.bpf.o", NULL);bpf_object__load(obj);
// Skeleton generation (always via bpftool)// bpftool gen skeleton counter.bpf.o name counter > counter.skel.h

4. eBPF map types

Map type	Key→Value	Use case
`BPF_MAP_TYPE_HASH`	arbitrary→arbitrary	Per-PID counters, state
`BPF_MAP_TYPE_ARRAY`	u32→fixed	Config, metrics indexed by CPU
`BPF_MAP_TYPE_PERCPU_HASH`	key→per-CPU val	High-frequency counters without locks
`BPF_MAP_TYPE_RINGBUF`	—	Efficient kernel→userspace events
`BPF_MAP_TYPE_PERF_EVENT_ARRAY`	—	Legacy perf event output
`BPF_MAP_TYPE_LRU_HASH`	key→val	Connection tracking, limited size
`BPF_MAP_TYPE_PROG_ARRAY`	u32→prog	Tail calls, program chaining
`BPF_MAP_TYPE_XSKMAP`	—	AF_XDP socket redirection

Use BPF_MAP_TYPE_RINGBUF over PERF_EVENT_ARRAY for new code — lower overhead, variable-size records.

5. Verifier error triage

Error message	Root cause	Fix
`invalid mem access 'scalar'`	Dereferencing unbounded pointer	Check pointer with null test before use
`R0 !read_ok`	Return without setting R0	Ensure all paths set a return value
`jump out of range`	Branch target beyond program end	Restructure conditionals
`back-edge detected`	Backward jump (loop)	Use `bpf_loop()` helper (kernel ≥5.17) or bounded loop
`unreachable insn`	Dead code after return	Remove dead branches
`invalid indirect read`	Stack read of uninitialised bytes	Zero-init structs: `struct foo x = {}`
`misaligned stack access`	Pointer arithmetic off alignment	Align reads to `__u64` boundaries

bash

# Get detailed verifier logbpftool prog load prog.bpf.o /sys/fs/bpf/prog type kprobe \    2>&1 | head -100
# Check loaded programsbpftool prog listbpftool prog dump xlated id 42

# Get detailed verifier logbpftool prog load prog.bpf.o /sys/fs/bpf/prog type kprobe \    2>&1 | head -100
# Check loaded programsbpftool prog listbpftool prog dump xlated id 42

6. XDP programs

// xdp_drop_icmp.bpf.c#include <vmlinux.h>#include <bpf/bpf_helpers.h>#include <bpf/bpf_endian.h>
SEC("xdp")int xdp_filter(struct xdp_md *ctx){    void *data_end = (void *)(long)ctx->data_end;    void *data     = (void *)(long)ctx->data;    struct ethhdr *eth = data;
    if ((void *)(eth + 1) > data_end)        return XDP_PASS;
    if (bpf_ntohs(eth->h_proto) != ETH_P_IP)        return XDP_PASS;
    struct iphdr *ip = (void *)(eth + 1);    if ((void *)(ip + 1) > data_end)        return XDP_PASS;
    if (ip->protocol == IPPROTO_ICMP)        return XDP_DROP;
    return XDP_PASS;}char LICENSE[] SEC("license") = "GPL";

// xdp_drop_icmp.bpf.c#include <vmlinux.h>#include <bpf/bpf_helpers.h>#include <bpf/bpf_endian.h>
SEC("xdp")int xdp_filter(struct xdp_md *ctx){    void *data_end = (void *)(long)ctx->data_end;    void *data     = (void *)(long)ctx->data;    struct ethhdr *eth = data;
    if ((void *)(eth + 1) > data_end)        return XDP_PASS;
    if (bpf_ntohs(eth->h_proto) != ETH_P_IP)        return XDP_PASS;
    struct iphdr *ip = (void *)(eth + 1);    if ((void *)(ip + 1) > data_end)        return XDP_PASS;
    if (ip->protocol == IPPROTO_ICMP)        return XDP_DROP;
    return XDP_PASS;}char LICENSE[] SEC("license") = "GPL";

bash

# Attach XDP program to interfaceip link set dev eth0 xdp obj xdp_drop_icmp.bpf.o sec xdp# Removeip link set dev eth0 xdp off# Use native (driver) mode for best performanceip link set dev eth0 xdp obj prog.bpf.o sec xdp mode native

# Attach XDP program to interfaceip link set dev eth0 xdp obj xdp_drop_icmp.bpf.o sec xdp# Removeip link set dev eth0 xdp off# Use native (driver) mode for best performanceip link set dev eth0 xdp obj prog.bpf.o sec xdp mode native

XDP return codes: XDP_PASS, XDP_DROP, XDP_TX (hairpin), XDP_REDIRECT.

7. CO-RE — compile once, run everywhere

CO-RE (Compile Once - Run Everywhere) uses BTF type info to relocate field accesses at load time.

// Use BTF-based field access (CO-RE aware)#include <vmlinux.h>        // generated from running kernel's BTF#include <bpf/bpf_core_read.h>
SEC("kprobe/tcp_connect")int trace_connect(struct pt_regs *ctx){    struct sock *sk = (struct sock *)PT_REGS_PARM1(ctx);    u16 dport = BPF_CORE_READ(sk, __sk_common.skc_dport);    // BPF_CORE_READ relocates the field offset at load time    bpf_printk("connect to port %d\n", bpf_ntohs(dport));    return 0;}

// Use BTF-based field access (CO-RE aware)#include <vmlinux.h>        // generated from running kernel's BTF#include <bpf/bpf_core_read.h>
SEC("kprobe/tcp_connect")int trace_connect(struct pt_regs *ctx){    struct sock *sk = (struct sock *)PT_REGS_PARM1(ctx);    u16 dport = BPF_CORE_READ(sk, __sk_common.skc_dport);    // BPF_CORE_READ relocates the field offset at load time    bpf_printk("connect to port %d\n", bpf_ntohs(dport));    return 0;}

bash

# Generate vmlinux.h from running kernelbpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h
# Verify BTF is enabledls /sys/kernel/btf/vmlinux

# Generate vmlinux.h from running kernelbpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h
# Verify BTF is enabledls /sys/kernel/btf/vmlinux

8. BPF ring buffer vs perf buffer

// Ring buffer (preferred for new programs — better perf, no per-CPU loss)struct {    __uint(type, BPF_MAP_TYPE_RINGBUF);    __uint(max_entries, 256 * 1024);} rb SEC(".maps");
SEC("kprobe/sys_open")int handle_open(struct pt_regs *ctx){    struct event *e = bpf_ringbuf_reserve(&rb, sizeof(*e), 0);    if (!e)        return 0;    e->pid = bpf_get_current_pid_tgid() >> 32;    bpf_ringbuf_submit(e, 0);    // bpf_ringbuf_discard(e, 0) on error paths    return 0;}

// Ring buffer (preferred for new programs — better perf, no per-CPU loss)struct {    __uint(type, BPF_MAP_TYPE_RINGBUF);    __uint(max_entries, 256 * 1024);} rb SEC(".maps");
SEC("kprobe/sys_open")int handle_open(struct pt_regs *ctx){    struct event *e = bpf_ringbuf_reserve(&rb, sizeof(*e), 0);    if (!e)        return 0;    e->pid = bpf_get_current_pid_tgid() >> 32;    bpf_ringbuf_submit(e, 0);    // bpf_ringbuf_discard(e, 0) on error paths    return 0;}

Feature	Ring buffer	Perf buffer
API	`bpf_ringbuf_reserve/submit`	`bpf_perf_event_output`
Backpressure	Reserve fails if full	May drop events
Userspace	`ring_buffer__poll` (libbpf)	`perf_buffer__poll`
Multi-producer	Yes	Per-CPU buffers

9. BPF iterators

struct {    __uint(type, BPF_MAP_TYPE_PROG_ARRAY);    __uint(max_entries, 1);} iter_prog SEC(".maps");
SEC("iter/task")int dump_tasks(struct bpf_iter__task *ctx){    struct task_struct *task = ctx->task;    if (task)        bpf_seq_printf(ctx->meta->seq, "%d %s\n", task->tgid, task->comm);    return 0;}

struct {    __uint(type, BPF_MAP_TYPE_PROG_ARRAY);    __uint(max_entries, 1);} iter_prog SEC(".maps");
SEC("iter/task")int dump_tasks(struct bpf_iter__task *ctx){    struct task_struct *task = ctx->task;    if (task)        bpf_seq_printf(ctx->meta->seq, "%d %s\n", task->tgid, task->comm);    return 0;}

bash

# Read iterator output from userspacebpftool prog tracelog   # or attach iter to seq_file readercat /sys/kernel/debug/tracing/trace_pipe

# Read iterator output from userspacebpftool prog tracelog   # or attach iter to seq_file readercat /sys/kernel/debug/tracing/trace_pipe

Iterators walk kernel data structures (tasks, maps, TCP sockets) without kprobe overhead per element.

10. BPF atomics

// GCC/Clang atomic builtins in BPF programs (kernel 5.12+)static __always_inline void inc_counter(__u32 *counter){    __sync_fetch_and_add(counter, 1);}
// Use for per-CPU or map-backed counters under concurrent probes

// GCC/Clang atomic builtins in BPF programs (kernel 5.12+)static __always_inline void inc_counter(__u32 *counter){    __sync_fetch_and_add(counter, 1);}
// Use for per-CPU or map-backed counters under concurrent probes

Prefer per-CPU array maps for high-frequency counters; use atomics when aggregating into a single map value.

For the full map types reference, see references/ebpf-map-types.md [blocked].

Related skills

Use skills/observability/ebpf-rust for Aya framework Rust eBPF programs
Use skills/profilers/linux-perf for perf-based tracing without eBPF
Use skills/runtimes/binary-hardening for seccomp-bpf syscall filtering
Use skills/low-level-programming/linux-kernel-modules for kernel module development
Use skills/async-io/af-xdp for XDP_REDIRECT to AF_XDP sockets

Source and attribution

Source:mohitmishra786/low-level-dev-skillsinskills/observability/ebpfat commitbdc5847

License: No license

Content belongs to its original authors. SourceWeft indexes it from a public repository.

Report or request removal

Ebpf

eBPF

Purpose

Triggers

Workflow

1. Choose the right tool

2. bpftrace — quick kernel tracing

3. libbpf skeleton — minimal C program

4. eBPF map types

5. Verifier error triage

6. XDP programs

7. CO-RE — compile once, run everywhere

8. BPF ring buffer vs perf buffer

9. BPF iterators

10. BPF atomics

Related skills

Source and attribution

More from mohitmishra786/low-level-dev-skills

Zig Debugging

Simd Intrinsics

Zig Cross

Rustc Basics

Zig Cinterop

Zig Build System

More in Software Development

Playground

M5 Onboard

Cardputer Buddy

Web Design Guidelines

Vercel React Native Skills

Vercel React Best Practices