Concurrency Debugging

Concurrency Debugging

Purpose

Guide agents through diagnosing and fixing concurrency bugs: reading ThreadSanitizer race reports, using Helgrind for lock-order analysis, detecting deadlocks with GDB thread inspection, identifying common std::atomic misuse patterns, and applying happens-before reasoning in C++ and Rust.

Triggers

• "ThreadSanitizer reported a data race — how do I read the report?"
• "My program deadlocks — how do I debug it?"
• "How do I use Helgrind to find threading bugs?"
• "Am I using std::atomic correctly?"
• "How does happens-before work in C++ memory ordering?"
• "How do I find which threads are deadlocked in GDB?"

Workflow

1. ThreadSanitizer (TSan) — race detection

# Build with TSanclang -fsanitize=thread -g -O1 -o prog main.c# or GCCgcc -fsanitize=thread -g -O1 -o prog main.c
# Run (TSan intercepts memory accesses at runtime)./prog
# TSan-specific optionsTSAN_OPTIONS="halt_on_error=1:second_deadlock_stack=1" ./prog

Reading a TSan report:

WARNING: ThreadSanitizer: data race (pid=12345)  Write of size 4 at 0x7f1234 by thread T2:    #0 increment /src/counter.c:8:5              ← access site in T2    #1 worker_thread /src/counter.c:22:3
  Previous read of size 4 at 0x7f1234 by thread T1:    #0 read_counter /src/counter.c:3:14          ← conflicting access in T1    #1 main /src/counter.c:30:5
  Thread T2 created at:    #0 pthread_create .../tsan_interceptors.cpp    #1 main /src/counter.c:28:3
SUMMARY: ThreadSanitizer: data race /src/counter.c:8:5 in increment

How to read:

1. Line 1: type of access (write/read) and address
2. Stack under "Write of size": the thread that performed the write
3. Stack under "Previous read/write": the conflicting thread
4. "Thread T2 created at": where the thread was spawned
5. Fix: the increment and read_counter functions access the same address without synchronization

Common races and fixes:

2. Helgrind — lock-order and race detection

Helgrind uses Valgrind infrastructure to detect lock ordering violations (potential deadlocks) and data races:

# Run with Helgrindvalgrind --tool=helgrind --log-file=helgrind.log ./prog
# Lock order violation report==1234== Thread #3: lock order "0x... M2" after "0x... M1"==1234== observed (incorrect) order==1234==    at pthread_mutex_lock (helgrind/...)==1234==    by worker2 /src/worker.c:45           ← T3 takes M2 then M1==1234====1234== required order established by acquisition of lock at address 0x... M1==1234==    at pthread_mutex_lock==1234==    by worker1 /src/worker.c:31            ← T1 takes M1 then M2

Lock-order violation = potential deadlock:

• Thread T1 acquires M1, then tries M2
• Thread T2 acquires M2, then tries M1
• Both can deadlock if they race

Fix: enforce a consistent global lock ordering. Always take M1 before M2 everywhere.

3. Deadlock detection with GDB

# Attach GDB to a deadlocked processgdb -p $(pgrep prog)
# Or run under GDB then trigger deadlock
(gdb) info threads          # list all threads and current state# * 1  Thread 0x... (LWP 1234) "prog" ... in __lll_lock_wait ()#   2  Thread 0x... (LWP 1235) "prog" ... in __lll_lock_wait ()# Threads blocked in __lll_lock_wait = waiting for mutex
(gdb) thread 1(gdb) bt                    # show which mutex thread 1 is waiting for
(gdb) thread 2(gdb) bt                    # show which mutex thread 2 holds/waits
# Find the mutex owner(gdb) p ((pthread_mutex_t*)0x601090)->__data.__owner   # Linux glibc mutex# prints TID of owning thread
# Python script to dump all mutex owners (GDB 7+)pythonimport gdbfor t in gdb.selected_inferior().threads():    t.switch()    print(f"Thread {t.num}: {gdb.execute('bt 3', to_string=True)}")end

4. std::atomic misuse patterns

// WRONG: atomic variable, but non-atomic compound operationstd::atomic<int> counter{0};if (counter == 0) counter = 1;   // not atomic together! TOCTOU race
// CORRECT: use compare_exchangeint expected = 0;counter.compare_exchange_strong(expected, 1);
// WRONG: relaxed ordering for sync flagstd::atomic<bool> ready{false};// Producer:data = 42;ready.store(true, std::memory_order_relaxed);  // WRONG: no happens-before
// CORRECT: release-acquire for publishing data// Producer:data = 42;ready.store(true, std::memory_order_release);   // syncs with acquire
// Consumer:if (ready.load(std::memory_order_acquire)) {    // syncs with release    use(data);  // safe to read data here}
// WRONG: using data across threads without atomic/mutex// int shared_data;  // non-atomic — UB on concurrent access
// CORRECT: protect with mutex or make atomicstd::mutex mtx;std::unique_lock lock(mtx);shared_data = 42;

5. Happens-before reasoning

In C++, happens-before is established by:

Sequenced-before (within a thread):  Statement A comes before B in code → A happens-before B
Synchronizes-with (across threads):  store(release) → load(acquire) on SAME atomic variable    → store happens-before load    → everything before store happens-before everything after load
Thread creation/join:  spawn(T) → any action in T         (create synchronizes-with)  any action in T → join(T)          (join synchronizes-before)
Mutex:  unlock(M) → lock(M) (next acquirer)

// Establishing happens-before across threadsstd::atomic<int> flag{0};int data = 0;
// Thread 1:data = 42;                        // Aflag.store(1, memory_order_release); // B: A sequenced-before B
// Thread 2:while (flag.load(memory_order_acquire) != 1) {}  // C: synchronizes-with Bint x = data;                     // D: C sequenced-before D// D reads 42: A happens-before B synchronizes-with C sequenced-before D//             → A happens-before D

6. Rust concurrency — compile-time guarantees

Rust prevents data races at compile time via ownership:

use std::sync::{Arc, Mutex};use std::thread;
// Shared mutable state: Arc<Mutex<T>>let counter = Arc::new(Mutex::new(0u32));
let c = Arc::clone(&counter);let t = thread::spawn(move || {    let mut val = c.lock().unwrap();    *val += 1;});
t.join().unwrap();println!("{}", *counter.lock().unwrap());
// Rust prevents:// - Sharing &mut T across threads (Sync not impl for &mut T)// - Moving non-Send types to threads (compiler error)// Use TSAN_OPTIONS with cargo test if TSan checks are needed:// RUSTFLAGS="-Z sanitizer=thread" cargo +nightly test

Related skills

• Use skills/runtimes/sanitizers for TSan build flags and other sanitizers
• Use skills/profilers/valgrind for Helgrind and Memcheck integration
• Use skills/debuggers/gdb for advanced GDB thread inspection
• Use skills/low-level-programming/memory-model for C++/Rust memory ordering theory