Security Data Lake: Fix the SIEM Cost Trap

Q: What is a security data lake and how is it different from a SIEM?

A security data lake is an architecture that stores security telemetry — logs, events, findings, network and identity data — in cheap object storage (typically S3-class), normalised to an open schema and queried on demand, rather than indexing all of it up front the way a SIEM does. The core difference is schema-on-read versus schema-on-write. A SIEM parses, structures and indexes every event at ingest, which is what makes ingest expensive and is why SIEM pricing is per-gigabyte; a data lake stores data in a columnar format and applies structure only when a query runs, so keeping data costs almost nothing and you pay compute only for the questions you actually ask. In the mature pattern the two coexist: the SIEM becomes a focused, hot detection engine holding a high-signal slice of recent data for real-time alerting, while the lake becomes the durable system of record holding the full-fidelity firehose for months or years. You are not replacing detection; you are moving retention and ad-hoc investigation off the expensive engine and onto storage priced for the job.

Q: Why does my SIEM bill keep growing even when attacks are flat?

Because ingest-priced SIEMs bill for the volume of data you send them, and that volume grows with your infrastructure independently of your threat level. Every new endpoint agent, every additional cloud account, every verbose microservice or Kubernetes cluster produces more logs, and an ingest-based model turns each gigabyte into a recurring charge whether or not the data is ever queried. Industry cost analyses commonly place ingest-based SIEM pricing in the range of $2 to $5 per GB per day of sustained ingest, so a mid-size enterprise pushing several terabytes a day is looking at a seven-figure annual number before an analyst runs a single search. The misalignment is that the pricing model rewards volume while security value is concentrated in a small high-signal fraction — you pay the same premium for a Windows security event that fires a detection and for the fortieth identical load-balancer health check that will never appear in any investigation. The dangerous but common response is to drop low-signal logs at the source to control the bill, which cuts cost by manufacturing exactly the blind spots that make future incidents unsolvable.

Q: What is schema-on-read and why does it make retention cheap?

Schema-on-read means data is stored in its columnar files without being parsed, structured or indexed at write time, and the schema — the interpretation of which bytes mean which field — is applied only when a query runs. This is the opposite of a SIEM, which is schema-on-write: it does the expensive parsing and indexing at ingest, which is precisely why ingest is the cost driver. The economic consequence is direct. Writing to a schema-on-read lake is cheap because you are essentially just landing files in object storage, priced in cents or fractions of a cent per gigabyte, so keeping data for years costs almost nothing. You pay compute only at query time, and only for the data a query actually touches. Combined with partitioning by source and date and a columnar format like Parquet, a well-designed query for a specific user over a specific month scans a few partitions and a few columns rather than a year of everything, keeping even the query cost small. The trade is latency: schema-on-read queries are seconds to minutes, not the sub-second response of a hot SIEM index, which is why the pattern keeps a hot tier for real-time detection and uses the lake for retention and investigation.

Q: What is OCSF and why should I normalise to it?

OCSF, the Open Cybersecurity Schema Framework, is an open and vendor-neutral schema for security events, originated by AWS, Splunk and a coalition of security vendors and iterated through the version 1.x series to roughly v1.9 by 2026, with adoption across vendors including Datadog, SentinelOne, Palo Alto Networks and Rapid7, and native normalisation in Amazon Security Lake. You should normalise to it because the alternative — raw logs from forty products each with its own field name for the same concept, such as source IP — turns every threat hunt and every detection into an archaeology project that no query language can span. When events are mapped to OCSF field names at write time, a detection or hunt is written once and runs across every source, and swapping a tool such as an EDR or firewall does not force a rewrite of your detection corpus because the new source maps to the same schema on the way in. This is the schema that turns a data lake into something queryable rather than a write-only graveyard. The discipline is to normalise in the routing tier so the mapping cost is paid once per event, and to retain the raw original alongside the normalised copy, because a schema mapping is code that can have bugs and you want the un-mangled source when an investigation hinges on a field the mapping dropped.

Q: How do I decide what stays in the SIEM versus what goes to the lake?

The decision is made per-source and per-event in a routing tier that sits between your log sources and any priced destination. The rule of thumb is that the SIEM gets the detection-relevant subset in real time, while the lake gets everything at full fidelity. In practice the routing tier does four things: it drops the provably useless such as duplicate heartbeats and health-check spam so you never pay to store them anywhere; it writes a full-fidelity copy of everything else to the lake regardless of detection value; it evaluates whether an event is detection-relevant and, if so, trims it to the fields detection needs and enriches it with context like asset owner, identity and GeoIP; and it forwards only that dense, enriched subset onward to the expensive SIEM. High-signal, time-sensitive sources — endpoint detections, authentication, cloud control-plane events — belong in the hot tier because you need to alert on them in seconds. High-volume, low-immediate-signal sources — verbose flow logs, DNS, successful-request logs — go to the lake in full and are queried when an investigation needs them. The key discipline is that routing to the lake is not the same as dropping: nothing is discarded to save money, the untrimmed original always lands in cheap storage, and only the priced copy is reduced.

Q: Is a security data lake a product I buy or a pattern I build?

It is a pattern, though several products implement it and can accelerate the build. The pattern has three essential parts: a tiering decision that splits hot detection from cold retention, an open schema and table format that keep the cold tier queryable, and a query engine in front of the storage. Amazon Security Lake is a managed implementation that has been generally available since 2023 and automatically normalises supported sources to OCSF, storing them as Parquet on Apache Iceberg in an S3 bucket you own; Snowflake, Google Security Operations, Panther and others sell their own variations. You can also assemble the pattern from open components — object storage, a routing tier such as Cribl Stream or Vector, OCSF normalisation, an Iceberg table format, and a query engine like Trino, Athena or a warehouse — which avoids locking your data into one vendor’s idea of open. The important point is that buying a product does not exempt you from the design work: a tool that lands raw logs in storage without partitioning, without a schema and without cost guardrails on the query tier reproduces the swamp with a nicer logo. Establish the tiering, schema and retention model first, then choose tools that fit it.

Q: Will a data lake actually save money or just move the cost around?

It moves cost from ingest-time to query-time, and whether that is a saving depends entirely on the design. The saving is real because the dominant cost in the ingest-priced model is paying premium per-gigabyte rates to store data that is rarely or never queried; moving that retention to object storage priced in cents replaces a large, guaranteed, recurring charge with a small storage charge plus a variable query charge you only incur when you investigate. Because most stored security data is never queried, the expected query cost is a fraction of the retention cost it displaces. But the pattern can fail to save money if the cold tier is built badly. Unpartitioned raw logs force full scans, so a single broad query can cost more than expected; an open lake with no quotas invites a SELECT-star across a year of data that erases the savings in one afternoon. The fixes are architectural: partition by source and date, store columnar so queries touch only the needed columns, put result caps and partition-pruning enforcement in front of the query engine, and instrument query cost the way you instrument the SIEM bill. Done properly the outcome is not merely a percentage saved; it is that visibility and cost stop being a trade-off, because adding a log source becomes a routing decision rather than a budget fight.

Q: Can I run detections directly on the data lake instead of a SIEM?

For some things yes, but not for the real-time detection the SIEM exists to do, and conflating the two is a design mistake. Schema-on-read lake queries run in seconds to minutes because they scan object storage at query time, which is fine for threat hunting, incident response across a long dwell period, historical investigation and scheduled batch analytics, but is far too slow for detections that must alert within seconds of an event. Real-time correlation, alerting on an active attack, and any rule with a sub-second latency requirement belong on the hot tier. The productive relationship between the two is a loop: you hunt in the lake across full history, and when a hunt surfaces a durable, high-value pattern you promote it into a real-time detection on the hot tier. Some newer stacks blur the line with streaming detection that runs on data in flight before it lands, and with detection engines that sit on lake storage for near-real-time rather than instant alerting, but the general principle holds — match the latency of the detection to the tier, keep instant detection hot, and use the lake for the investigations and history the hot tier is too small and too expensive to hold. The lake makes the SIEM better by letting it be small; it does not replace what the SIEM is for.

Q: How long should I retain security logs, and in which tier?

Retention is a function of two independent axes — regulatory obligation and investigative usefulness — and each maps to a different storage tier with a different cost, so the answer is almost never one number for everything. A defensible three-tier design keeps a hot window of roughly 30 to 90 days in the SIEM, where most detection and active investigation happen, priced high and deliberately kept small. It keeps a warm tier of six to eighteen months in the queryable lake, full-fidelity and normalised on object storage, cheap enough to retain and fast enough for incident response and threat hunting across a breach that dwelled for months — which matters because attacker dwell time is frequently measured in weeks or months, and a 90-day-only retention means the origin of a long intrusion has already aged out. And it keeps a cold or archival tier of one to seven or more years in object-storage glacier classes to satisfy compliance mandates such as PCI DSS, HIPAA and regional data-retention law, at prices low enough that keeping the data is cheaper than the meeting to discuss deleting it. The governance discipline is to tier by obligation and value, document the rationale, and make the access path to each tier auditable, so that when a regulator or an incident asks why a given log was or was not available, the answer is a design decision rather than an accident of the budget.

Satyam Kumar

返回博客

ai-architecture

Your SIEM Bill Is a Data Architecture Problem, Not a Security One

By Satyam KumarAugust 18, 202614 min read

security data lake siem cost ocsf schema-on-read log management tiered retention amazon security lake apache iceberg cribl security operations soc architecture detection engineering data lake architecture log ingestion cost security telemetry 2026

Your SIEM Bill Is a Data Architecture Problem, Not a Security One

Frequently Asked Questions

分享这篇文章

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

人工智能和云架构师。帮助团队构建可扩展到数百万的系统。

LinkedIn GitHub

Your SIEM Bill Is a Data Architecture Problem, Not a Security One

Frequently Asked Questions

分享这篇文章

Comments

Leave a comment