Detection-as-Code: Test Detections Like Software

Q: What is detection-as-code?

Detection-as-code is the discipline of treating every security detection — a SIEM correlation rule, an EDR analytic, an alerting query — as a software artefact rather than as configuration typed into a console. In practice that means the detection lives as a file in version control, changes to it arrive as pull requests that are peer-reviewed, automated tests in CI prove the rule fires on known-bad input and stays silent on known-good input before it is trusted, coverage is computed against a framework like MITRE ATT&CK, and deployment to the SIEM happens through an automated pipeline with a git revert as the rollback path. The point is not a particular tool; it is that a detection acquires every property software engineering already knows how to guarantee — correctness, reviewability, testability, traceability, portability — because those properties come from making the thing a versioned artefact behind a pull request, not from where you store it. Detection-as-code is what closes the gap where detections were, for two decades, the only code in the building that shipped straight to production with none of the discipline the rest of engineering takes for granted.

Q: Why not just write detections directly in the SIEM console?

Because a detection written directly in a console is unreviewed, untested, untracked code running against production, and every failure mode of untested code applies — except the bug here is a missed breach. The console-first workflow optimises for speed of authoring over correctness of the deployed corpus, and it produces three chronic failures. Silent decay: a rule that fired last year stops matching because a field was renamed or a log source changed format, and nothing tells you, because the alert simply never comes. Duplication and drift: two analysts write overlapping rules that disagree, with no diff, no owner and no single source of truth. And unfalsifiable coverage: nobody can state which adversary techniques the corpus actually detects, so the answer to an auditor or a board is a confident guess. Detection-as-code moves the discovery of a broken detection left, into a pipeline run that happens before the rule is trusted, instead of into the incident where the technique it was meant to catch is used and no alert fires. The console is fine for exploration; it is a dangerous system of record.

Q: What is Sigma and why does it matter for detection-as-code?

Sigma is, in SigmaHQ's own words, "a generic and open signature format that allows you to describe relevant log events in a straightforward manner." A Sigma rule is a small YAML document naming a log source and a detection condition, and the current official toolchain — pySigma as the library and sigma-cli as the command-line converter — compiles it into a target backend's query language, whether that is Splunk SPL, Elastic KQL, Microsoft Sentinel KQL or a data-lake query. That indirection is what makes detection-as-code work. Because the rule is YAML it diffs cleanly in a pull request and a reviewer can read the logic; because it is backend-agnostic the same reviewed artefact deploys to every SIEM you run and a future migration is a recompile rather than a hand rewrite of the whole corpus; and because the format is shared and open you can pull from and contribute to a public corpus of community rules instead of every team re-deriving the same credential-dumping detection privately. The discipline is to keep the abstraction honest: not every detection expresses cleanly in Sigma, and the genuinely platform-specific ones should stay native but still live in Git and still pass review and test.

Q: How do you measure detection coverage against MITRE ATT&CK?

You tag every detection with the ATT&CK technique IDs it addresses, then compute coverage across the technique matrix and visualise it, typically as an ATT&CK Navigator heatmap where each technique is coloured by how many detections cover it. MITRE ATT&CK is the shared vocabulary — a globally accessible knowledge base of adversary tactics and techniques based on real-world observations — and tagging turns your corpus into something queryable, so an abstract question like "can we see lateral movement?" becomes a concrete one you can point at on a matrix. The essential caveat is that coverage breadth and detection quality are different axes. A green cell means a rule exists and is tagged to that technique; it does not mean the rule is good, that it fires on the variants a real adversary uses, or that the log source feeding it is even being ingested. A matrix that is all green because every technique has one weak, untuned rule is more dangerous than an honest one with fewer cells filled, because it manufactures false confidence. The fix is to pair the breadth view with a log-source-and-data-quality score — the DeTT&CT approach — so you report both "is there a detection?" and "is the data feeding it present and good enough for that detection to work?"

Q: How do you stop a detection from silently breaking over time?

With two layers of testing. At author time, every rule ships with test fixtures: a known-bad log event that must match and a known-good event that must not, so CI can prove on every change that the rule fires when it should and stays quiet when it should not. That catches logic errors and false-positive regressions the moment they are introduced. But it cannot catch the most insidious failure — a rule that was correct and stops working because reality moved underneath it: a field renamed in an agent upgrade, a log source that changed format, an ingestion pipeline that dropped the field the rule keys on. The catch for that is continuous validation: tools such as Atomic Red Team execute small, ATT&CK-mapped tests that emulate real technique behaviour in a controlled environment, and you assert that each produces the alert it should. Run on a schedule against your actual live pipeline, this is a regression suite for the SOC — it exercises the whole chain from endpoint to log to detection to alert, and a test that used to pass and now fails tells you a detection has decayed before an adversary discovers the same gap for free. Author-time unit tests plus scheduled adversary emulation is the combination that makes a broken detection a red build instead of a headline.

Q: Is detection-as-code a product I can buy?

No — detection-as-code is a discipline made of review, test, version control and coverage measurement, not a SKU, and confusing the two is a common and expensive mistake. A tool that stores rules in Git but has no test fixtures, no coverage model and no regression cadence has automated the filing, not the engineering; you get the veneer of the practice without the properties that make it valuable. The correct sequence is to establish the workflow first — get detections into version control, require pull requests, add known-bad and known-good fixtures, wire CI to block the merge on a failing test, map coverage to ATT&CK — and only then buy tools that fit and accelerate that workflow. Plenty of the machinery is open source: Sigma and its pySigma/sigma-cli toolchain for portable rules, ATT&CK Navigator for coverage visualisation, DeTT&CT for data-source scoring, and Atomic Red Team for validation. Vendors add real value in orchestration, scale and integration, but they cannot sell you the discipline, and a team that adopts a platform without the underlying practice ends up with an expensive rule repository that still decays silently.

Q: What are the biggest anti-patterns in detection engineering?

The first is chasing a 100%-green ATT&CK matrix, because breadth without quality is negative value: one tuned, tested, validated detection per technique beats ten untuned ones, and a matrix engineered to look complete for a board slide is a liability the first time it is trusted during an incident. The second is writing rules with no test fixtures — a Sigma rule without a known-bad and known-good sample is an assertion, not a detection, and if CI cannot prove it fires on the bad and stays silent on the good, you have shipped an opinion. The third is ignoring the false-positive budget: a detection that pages ten times a shift for benign activity gets muted by exhausted analysts, and a muted detection is a non-detection with extra steps, so alert fatigue must be treated as a first-class failure. The fourth is skipping the log-source reality check — a rule tagged to a technique whose data you do not actually ingest is a green cell detecting nothing. The fifth, and the one that defeats the whole exercise, is never re-testing deployed rules; detections decay silently, and without scheduled adversary emulation against the live pipeline you will discover a broken detection during the breach it should have caught.

Q: How does detection-as-code relate to purple teaming and continuous validation?

They are complementary halves of the same loop, and detection-as-code is what makes purple-team findings durable instead of one-off. A purple-team exercise pairs offensive emulation with defensive observation to find where detections are missing or weak; detection-as-code is the mechanism that captures each finding as a new or improved rule with a test fixture, so the gap that was found once cannot silently reopen. Continuous validation — running ATT&CK-mapped emulation such as Atomic Red Team on a schedule and asserting that each produces its expected alert — turns the purple-team spirit into a regression suite that runs without a human every time, so the corpus is exercised end to end against the live pipeline rather than only during an annual engagement. In a mature program the flow is circular: validation reveals a gap, the gap becomes a detection-backlog item, the new detection ships through the pipeline with a fixture and a coverage tag, and the next validation run confirms it fires. This is also the substrate that makes higher-level automation safe, because an AI-native SIEM or an autonomous SOC agent is only as good as the detections feeding it, and a decayed rule poisons everything downstream of it.

Satyam Kumar

Back to Blog

ai-architecture

Your Detections Are Software. You Just Never Tested Them Like Software

By Satyam KumarAugust 17, 202615 min read

detection-as-code detection engineering sigma rules mitre att&ck siem security operations threat detection atomic red team purple teaming detection coverage pysigma adversary emulation soc automation dettect detection regression testing 2026

Your Detections Are Software. You Just Never Tested Them Like Software

Frequently Asked Questions

Share this article

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

AI & Cloud Architect. Helping teams build systems that scale to millions.

LinkedIn GitHub

Your Detections Are Software. You Just Never Tested Them Like Software

Frequently Asked Questions

Share this article

Comments

Leave a comment