eBPF is not free: what our sensor actually costs you

Every runtime security vendor says their eBPF sensor is lightweight. Nobody publishes the p99. This is our attempt at the post we wanted to read before we built ours.

All numbers below are from our public benchmark suite, on c6i.4xlarge nodes running Kubernetes 1.32, kernel 6.8, sensor version 4.11. The suite and the raw results are in our GitHub org; the point of publishing them is that you can disagree with our methodology.

What the sensor attaches to

Twenty-two programs. The expensive ones, in rough order of cost:

• sys_enter_execve / execveat — process execution, every one
• security_socket_connect / accept — connection establishment, both directions
• security_file_open — filtered in-kernel to a configurable path set
• security_bprm_check — binary provenance
• tcp_sendmsg / tcp_cleanup_rbuf — byte accounting for egress baselines
• cgroup/sock_create — workload attribution

Everything else is cheap. The byte-accounting probes are the ones that will show up in your flame graph, because they run on a hot path.

The numbers

Overhead is expressed as additional node CPU as a percentage of one core-equivalent, plus added p99 latency on the workload’s own request path.

Workload shape	CPU p50	CPU p99	Sensor RSS	Added p99 latency
HTTP API, 4k req/s, short-lived conns	0.7%	1.4%	46 MB	0.31 ms
gRPC service, 900 req/s, long-lived conns	0.4%	0.8%	41 MB	0.09 ms
Batch / Spark executor, heavy file I/O	2.9%	6.1%	88 MB	n/a
CI runner, 40+ execve/s, churny pods	4.2%	9.8%	71 MB	n/a

The first two are the shapes most people deploy to, and 0.7% median is genuinely small. The last two are where the honesty starts.

node CPU attributable to sensor · CI runner fleet, 12hc6i.4xlarge

The two places we tell customers not to deploy

CI runner fleets. A pod that forks forty processes a second is pathological for execve tracing, and CI runners are the most disposable, least interesting workloads in most estates. Our advice is to exclude the CI namespace entirely and cover it at the control plane instead — you care about what the CI identity does in your cloud account, not what npm forks inside an ephemeral pod. Several customers ignore this and accept the 4%. That is a legitimate choice; it is just not the default we would pick.

Latency-critical trading and real-time media paths. If your p99 budget is measured in hundreds of microseconds, our 310µs on the HTTP shape is not noise, it is a third of your budget. Run the sensor in connection-metadata-only mode (drop the byte-accounting probes) and accept weaker egress baselining, or do not run it on those nodes.

Where the cost actually went

The naive version of this sensor was three times more expensive. Three changes did most of the work:

1. Filter in kernel, not user space. The first version sent every security_file_open to user space and filtered there. Moving the path-set filter into the eBPF program as a LPM_TRIE map cut file-probe cost by about 80%. Obvious in hindsight; we shipped the naive version for five months.

2. Ring buffer, not perf buffer. BPF_MAP_TYPE_RINGBUF instead of per-CPU perf buffers removed a per-event copy and a good deal of user-space reassembly. Worth roughly 20% of total sensor CPU, and it simplified ordering guarantees.

3. Aggregate bytes in kernel. We used to emit an event per tcp_sendmsg. We now accumulate into a per-socket counter map and emit on a one-second tick or socket close. For a chatty service this is a 400× reduction in event volume, and it is why the gRPC row above is cheaper than the HTTP row despite moving more bytes.

Measure it yourself

kubectl apply -f https://github.com/aegis-console/bench/releases/latest/bench.yaml
kubectl -n aegis-bench logs -l job=baseline --tail=40

The suite runs a matched pair — sensor on, sensor off — against your own workload profile for an hour and prints the delta. If your numbers are worse than the table above, open an issue with the output and we will look at it; that has happened eleven times and produced four real fixes.