One common performance issue in Linux systems is a program consuming unexpectedly high CPU. At first glance, high CPU usage alone doesn’t necessarily indicate a problem, but if the reason for the high usage is unclear, it warrants further investigation. One possible explanation (for the sake of this post) is that the program is spawning a large number of short-lived processes, which can overwhelm system resources.
To confirm this suspicion, we typically start with standard tools such as top and ps. However, these tools may fail to capture short-lived processes due to their transient nature.
This is where eBPF (extended Berkeley Packet Filter) and BCC (BPF Compiler Collection) come into play. eBPF allows to run safe, sandboxed programs inside the Linux kernel, giving access to powerful observability primitives with minimal overhead. BCC builds on top of eBPF to make writing these tools simpler — combining kernel instrumentation with Python interfaces for user-space interaction.
One of the tools built with BCC is execsnoop, which provides a way to monitor process execution in real-time. It traces every exec() call as it happens, allowing us to confirm if excessive process spawning is indeed the issue — even for processes that live only milliseconds.
To better understand and illustrate this behavior, I will create a small bash script that simulates the rapid spawning of short-lived processes. This script will repeatedly launch a series of lightweight commands (such as /bin/true), which execute and exit almost instantly. The goal is to mimic a real-world scenario where a program creates many subprocesses that are too short-lived to be captured reliably by standard monitoring tools.
Here is the sample Bash script:
#!/bin/bash
# Filename: myloop.sh
while true; do
for i in {1..100}; do
/bin/true
done
done
BCC leverages eBPF (extended Berkeley Packet Filter), a modern in-kernel virtual machine that allows for safe, low-overhead instrumentation of the Linux kernel. Unlike ps or top, execsnoop captures every executed command as it happens, including those that exit almost immediately—making it ideal for catching short-lived processes.
To begin monitoring with execsnoop, run sudo execsnoop. This will display a real-time stream of every command that gets executed on the system, along with details such as process name, PID, parent PID, and arguments.
If you want to filter by a specific process name (e.g., the script I created earlier), you can use: sudo execsnoop -n myloop. This ensures that only executions triggered by the myloop process are shown, making it easier to focus on just the relevant events.
In one terminal, run the previously created myloop.sh script which repeatedly spawns short-lived /bin/true processes. Meanwhile, in another terminal, launch execsnoop to monitor process executions in real-time:
[fadai@host ~]$ sudo /usr/share/bcc/tools/execsnoop -n myloopThis command filters the output to only show executions initiated by the command named myloop. As the script runs, execsnoop will reveal each individual /bin/true execution, confirming the presence of rapid, short-lived processes that would otherwise go unnoticed using tools like ps or top. This helps validate the suspicion that the system is burdened by excessive process creation. Below is the output:
[fadai@host ~]$ sudo /usr/share/bcc/tools/execsnoop -n myloop
PCOMM PID PPID RET ARGS
myloop 2897100 2892484 0 ./myloop
true 2897101 2897100 0 /usr/bin/true
true 2897102 2897100 0 /usr/bin/true
true 2897103 2897100 0 /usr/bin/true
true 2897104 2897100 0 /usr/bin/true
true 2897105 2897100 0 /usr/bin/true
true 2897106 2897100 0 /usr/bin/true
true 2897107 2897100 0 /usr/bin/true
true 2897108 2897100 0 /usr/bin/true
true 2897109 2897100 0 /usr/bin/true
true 2897110 2897100 0 /usr/bin/true
true 2897111 2897100 0 /usr/bin/true
true 2897112 2897100 0 /usr/bin/true
true 2897113 2897100 0 /usr/bin/true
true 2897114 2897100 0 /usr/bin/true
true 2897115 2897100 0 /usr/bin/true
true 2897116 2897100 0 /usr/bin/true
true 2897117 2897100 0 /usr/bin/true
true 2897118 2897100 0 /usr/bin/true
true 2897119 2897100 0 /usr/bin/true
true 2897120 2897100 0 /usr/bin/true
Beyond its practical utility, execsnoop is also a great educational tool for learning how BPF (Berkeley Packet Filter) programs interact with the kernel. The execsnoop script, part of the BCC (BPF Compiler Collection), is written in Python. Let’s take a brief look at the parts of its source code that matter most when it comes to tracing process executions.
The script attaches BPF programs to kernel functions related to process execution—specifically, it hooks into the syscall execve, using kprobes. This allows the BPF code to run every time a new process is executed.
At the core is the makeprobe shell function, which builds a formatted probe string to capture syscall arguments:
function makeprobe {
func=$1
kname=execsnoop_$func
kprobe="p:$kname $func"
i=0
while (( i < argc + 1 )); do
kprobe="$kprobe +0(+$(( i * offset ))(%si)):string"
(( i++ ))
done
}This does several things:
- func=$1: Takes the kernel function name to probe, e.g., sys_execve, stub_execve, or do_execve.
- kname=execsnoop_$func: Sets a unique name for the kprobe event.
- kprobe=”p:$kname $func”: Begins the probe definition for the function.
- Then builds a sequence of argument fetches:
+0(%si):string
+0(+8(%si)):string
+0(+16(%si)):string
...Each of these offset expressions dereferences a pointer in %si, which holds a pointer to the argv[] array. So for example, +0(+8(%si)) effectively gets argv[1].
The argc and offset variables determine how many arguments to capture and how far apart they are in memory (usually 8 bytes for pointers).
Not all kernel versions expose the same symbol names. So execsnoop tries three possibilities in sequence:
makeprobe sys_execve
if ! echo $kprobe >> $tracing/kprobe_events; then
makeprobe stub_execve
if ! echo $kprobe >> $tracing/kprobe_events; then
makeprobe do_execve
if ! echo $kprobe >> $tracing/kprobe_events; then
edie "ERROR: adding a kprobe for execve. Exiting."
fi
fi
fiThis ensures compatibility with a wide range of kernel versions. The first that succeeds is used for tracing.
Once added, the probe is activated by enabling it:
echo 1 > /sys/kernel/debug/tracing/events/kprobes/$kname/enableAdditionally, execsnoop enables the sched_process_fork tracepoint, so it can later track the parent PID of each newly spawned process.
Once the kprobe and tracepoint are active, the kernel begins logging all matched events to the buffer.
( if (( opt_duration )); then
# wait then dump buffer
sleep $duration
cat -v trace
else
# print buffer live
cat -v trace_pipe
fi ) | $awk ...The script reads from this buffer in either of two modes:
- Buffered mode (-d):
Waits for the specified duration, then reads from /sys/kernel/debug/tracing/trace. - Live mode (default):
Streams output in real-time from /sys/kernel/debug/tracing/trace_pipe.
The awk script strips and reconstructs the command line from multiple argN= fields:
if ($0 ~ /arg1=""/) {
args = comm " [?]"
} else {
args = $0
sub(/.*arg1="/, "", args)
gsub(/" arg[0-9]*="/, " ", args)
sub(/"$/, "", args)
args = args " [...]"
}If the first argument is empty (e.g., arg1=””), it marks the output as ?. Otherwise, it cleans and combines the rest of the argN fields to reconstruct the original command.
Finally, the formatted output is printed to the console using this block from the awk script:
if (opt_time) {
time = $(3+o); sub(":", "", time)
printf "%-16s ", time
}
printf "%6s %6d %s\n", pid, getppid[pid], args
This does the following:
- If the -t flag was given, it adds a timestamp to each row.
- Uses getppid[pid] from the sched_process_fork tracepoint to correlate child → parent PID.
- Prints in a familiar PID PPID ARGS format:
TIME PID PPID COMMAND LINE
16:52:10.123456 1234 1010 /bin/ls -l [...]This gives a real-time stream (or snapshot, in -d mode) of every new process execution happening on the system, with parent-child relationship and command-line arguments.
BCC (BPF Compiler Collection) provides a powerful and flexible way to trace kernel-level activity using eBPF. It allows to:
- Attach to syscalls, tracepoints, and kprobes with ease
- Read complex kernel/user structures safely with helper functions
- Filter and process data in-kernel to reduce overhead
- Send structured event data to user space for rich analysis and reporting
The execsnoop tool from BCC is a concise yet powerful example of how eBPF can be leveraged for real-time observability in Linux. By attaching kprobes to the execve syscall, capturing arguments inside a BPF program, and streaming the data to user space, it offers visibility into process executions that are often too fast to catch with traditional tools.
