We deleted 1,900 detections and caught more
Detection coverage is a vanity metric. Here is what happened when we replaced a rule library with forty correlation patterns, and what it cost us.
In 2023 we shipped with 2,340 detections. It was the number on the comparison page, and it was the number every buyer asked about, and it was — I want to be precise here — almost entirely worthless.
Today we ship 412. Our customers’ median time-to-page went from 6 minutes to 38 seconds, and the proportion of situations that a human marks as “worth waking me” went from 4% to 31%. This post is about why, and about the three things the change made worse.
What a rule library actually is
A detection library grows the way a legal code grows: by accretion, in response to incidents, and nobody is ever rewarded for deletion. Ours had 2,340 entries because 2,340 things had, at some point, seemed worth writing a rule about.
Roughly, they broke down like this:
| Category | Count | Fired in 2023 |
|---|---|---|
| Known-bad indicators (IPs, hashes, domains) | 1,180 | 41 times, 39 false |
| Single-event “suspicious API call” rules | 640 | 210,000 times |
| Config-state assertions (better done by CSPM) | 290 | continuously |
| Behavioural / baseline rules | 180 | 3,100 times |
| Correlation patterns | 50 | 2,900 times |
The last row is fourteen detections short of our entire current library, and it produced the overwhelming majority of everything anybody acted on.
Single-event rules are a category error
Here is a real detection we deleted: alert when iam:CreateAccessKey is called for a user
that is not the caller.
This is a genuinely suspicious action. It is also a completely normal action, performed by platform teams, roughly forty times a week in a mid-size estate. The rule fired 210,000 times across our customer base in 2023. It was true-positive eleven times.
The problem is not the threshold. There is no threshold — the action either happened or it did not. The problem is that a single API call carries almost no information about intent. Intent lives in the sequence: who called it, from where, after what, having last done what, and what happened in the ninety seconds afterwards.
What replaced it
The same underlying signal now participates in a correlation pattern. It does not page anyone by itself. It pages when it co-occurs with any two of:
- the calling session authenticating from an ASN that identity has not used in 90 days
- the target user having no console login in 30+ days (a dormant account)
- the new key being used within 10 minutes from a different source than the creator
- a subsequent
iam:AttachUserPolicyor trust-policy edit in the same session
In 2025 this pattern fired 94 times across all customers. Seventy-one were marked worth-waking. That is a 75% precision rate on a signal that used to run at 0.005%.
Three things that got worse
I said I would be specific about the costs.
1. Hunting got harder before it got easier. When every signal had a rule, customers could search “show me every time rule X fired”. After the change, the individual signals still exist but are not first-class objects with names. We had to build a signal-level query interface that we had not planned, and it took two quarters, and for those two quarters several customers were genuinely worse off.
2. Explaining a non-detection is harder. “Why did you not alert on this?” used to have an easy answer: no rule matched, here is the gap, here is the rule we will write. Now the answer is “two of the four correlation conditions were met and the threshold is three”, which is correct and much less satisfying. We now show the partially-matched patterns in the console for exactly this reason.
3. We lost some genuinely good indicator coverage. Deleting 1,180 indicator rules deleted about 40 that were pulling their weight — specific, well-sourced, low-volume indicators for active campaigns. We reintroduced 28 of them as correlation inputs rather than standalone rules. The other twelve we simply lost, and two customers noticed.
What we would tell you to ask a vendor
Not “how many detections do you have”. Ask:
- What percentage of your alerts does the median customer mark as actionable?
- How many alerts does the median customer receive per week, per hundred workloads?
- When a detection does not fire on something it should have, can I see how close it came?
We answer these on the product page, with current numbers, and we update them quarterly whether they improved or not.