Logging Without Leaking: Redact PII and Secrets at Emit

Q: Why are application logs such a serious data-leak risk?

Logs are dangerous precisely because they feel harmless. A single log line is not a single copy of data — within milliseconds it is replicated to a local file, a log shipper, a central collector, a hot search tier, cold archival storage, and very often a third-party observability vendor that sits outside your trust boundary. Retention then holds every one of those copies for months or years. On top of that, read access to logs is almost always far broader than read access to the production database the logs describe, because logs are treated as a debugging convenience rather than a data store. Put those three properties together — massive replication, long retention, and broad access — and any sensitive value that lands in a log becomes the most widely distributed, longest-lived, and least-controlled copy of that data anywhere in your estate. That is why a leaked email or token in a log is often a bigger exposure than the same value in the database: the database has one guarded copy, and the log system has dozens of poorly guarded ones.

Q: What does "redact at emit, not at read" actually mean?

It refers to where in the pipeline you remove sensitive data, and it is the single most important decision in secure logging. Redact at read means the raw sensitive value is written into the log and stored, and you rely on the query tool or SIEM to mask it when someone views it — for example a view that shows only the last four digits of a card number. The problem is that this protects exactly one consumer, the person using that view, while the unmasked raw value has already been written to disk, shipped to the collector, copied to cold storage and backups, and possibly exported to a third-party vendor, all before anyone opened the masked view. Redact at emit means the sensitive value is stripped, dropped, or tokenised inside the application process, before the log line is serialised and handed to any shipper. That is the only point in the pipeline that is genuinely before replication, so it is the only place redaction actually prevents the copy from existing. A collector-level redaction pass is a worthwhile second layer for defence in depth, but by the time data reaches the collector it has already crossed the process boundary and may have touched local disk, so emit-time redaction must be the primary control.

Q: Why is an allow-list better than a deny-list of regexes for log redaction?

A deny-list approach tries to find and mask known-sensitive patterns — email addresses, card numbers, common token formats — in otherwise free-text logs, and it is a structurally losing game. It misses any format it was not written for, it misses new fields added to objects six months after the regex was written, it misses values that are base64-encoded or otherwise transformed, and it cannot see inside data logged by third-party libraries. Every one of those gaps is a silent leak. An allow-list inverts the default: you explicitly declare which fields are safe to log, and the logging layer drops or tokenises everything else automatically. This fails closed, which is the property you want for a security control — a field nobody classified is treated as unsafe and withheld, rather than leaked because no rule happened to match it. Allow-listing is only practical with structured logging, where events are typed fields rather than interpolated strings, which is a second reason to move away from free-text log messages: you cannot reliably allow-list what you cannot address by field name.

Q: Should secrets be masked or dropped in logs?

Secrets — passwords, API keys, bearer tokens, private keys, session cookies — should be dropped entirely, never masked, and this is a genuine distinction from how you handle PII. Masking a secret, for example replacing it with asterisks or showing a prefix, still does two harmful things: it confirms to anyone reading the log exactly where a secret was present and in what context, which is useful reconnaissance for an attacker, and it leaves you one masking bug away from disclosure, because the raw value was still flowing through the logging code and any error in the masking path prints it in full. Dropping the secret at source removes both risks. More fundamentally, a secret appearing in a log at all is a signal of an upstream problem: secrets should be handled through a secrets manager and injected in a way that keeps them out of the objects and requests your code logs, so if one is reaching the logger you have a gap in secret handling to fix, not just a line to mask. PII, by contrast, often needs to be tokenised rather than dropped, because you need a stable reference to correlate events without storing the identity.

Q: How do I keep logs useful for debugging and correlation if I strip the identifying data?

The technique is tokenisation, and it lets you keep the investigative value of logs without keeping the sensitive value. Instead of logging a user’s email or account number so you can trace their activity across services, you log a stable pseudonym derived from that identifier — for example a keyed hash — so that the same user produces the same token everywhere, and you can follow a session end to end, join events across microservices, and count distinct users, all without any raw identity being written down. The mapping from token back to identity, if you need it at all, lives in a single access-controlled system rather than being smeared across every log store. This directly satisfies the pseudonymisation expectation in data-protection regimes such as GDPR Article 32, and it is the same reversible-tokenisation pattern used in dedicated PII redaction pipelines. The key discipline is to decide per field which of three things you need: the raw value (almost never in a log), a stable token for correlation (for identifiers you trace on), or nothing at all (drop the field), and to make the safe option the default so that engineers have to opt in to logging anything more revealing.

Q: Is masking sensitive data in the SIEM enough for compliance?

No, and relying on it is a common and dangerous misreading of what the control does. SIEM-side masking is a read-time access control: it changes what a particular analyst sees in a particular tool. It does nothing about the fact that the raw, unmasked data was ingested and is stored in the SIEM’s underlying indices, in its cold storage, in the collectors that fed it, in the application log files on each host, and in your backups — every one of which is a place the data can be exfiltrated or subpoenaed regardless of what the analyst’s view shows. From a data-protection standpoint, the sensitive data has been processed and retained the moment it was written, so the obligations attach whether or not a UI masks it later. Genuine compliance requires that the sensitive data either never be written (dropped at emit) or be pseudonymised before storage (tokenised at emit), plus access tiering and retention limits on whatever remains. SIEM masking is a reasonable additional layer for the analysts who do have access, but treating it as the primary or sole control leaves the raw data replicated across your estate and leaves you exposed.

Q: How should access and retention differ across a log estate?

Once you accept that logs carry classified data, the log estate needs the same access-tiering and retention discipline as any other data store, rather than being one big searchable pool that the whole engineering organisation can read. A practical split has at least three tiers. An operational tier — latency, status codes, trace and span IDs, error rates — carries little or no sensitive data and can have broad read access and longer retention, because it is what most engineers need most of the time. An audit and security-event tier needs restricted access and strong integrity guarantees, often write-once storage, because it is what you rely on during an investigation and must be able to trust. A residual-sensitive tier, holding anything that survived redaction and might still be identifying, needs tight, role-scoped access, field-level authorisation where possible, and the shortest retention you can justify, because every additional month it is kept is another month of exposure. Tiering this way also controls cost, since the same volume discipline that shrinks a multi-terabyte SIEM bill shrinks the blast radius of a log-store compromise, so the security and the economics point in the same direction.

Q: Does redacting logs mean I will lose the visibility I need to detect attacks?

It should not, and if it does you have designed it wrong, because blinding yourself is its own failure mode — it is exactly the weakness OWASP catalogues as A09, Security Logging and Monitoring Failures. The goal of secure logging is not to log less but to log the right things: you keep the security-relevant signal — authentication events, authorisation decisions, access to sensitive resources, administrative actions, anomalies — while removing the sensitive payloads that carry no detection value. In practice, knowing that a specific tokenised user failed authentication twenty times from a new device is fully sufficient for detection, and you do not need their raw email or password attempt to see the attack; the token, the event type, the timing and the source are the signal. The mistake to avoid is treating redaction and detection as opposed, when in fact good structured logging improves both at once: typed, classified events are easier to alert on and easier to redact than free-text messages. Design the event schema around the questions your detection and investigation actually ask, keep those fields, tokenise the identifiers, and drop the raw sensitive values that were never part of the answer.

Satyam Kumar

Zurück zum Blog

cyber-security-patterns

Logging Without Leaking: PII, Secrets and the Observability Boundary (2026)

By Satyam KumarSeptember 13, 202610 min read

security logging log redaction pii in logs secrets in logs emit-time redaction structured logging observability boundary data loss prevention cwe-532 owasp a09 opentelemetry collector pseudonymisation gdpr article 32 log retention tiering tokenisation secure logging 2026

Logging Without Leaking: PII, Secrets and the Observability Boundary (2026)

Frequently Asked Questions

Artikel teilen

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

KI & Cloud Architekt. Ich helfe Teams, Systeme zu bauen, die auf Millionen skalieren.

LinkedIn GitHub

Logging Without Leaking: PII, Secrets and the Observability Boundary (2026)

Frequently Asked Questions

Artikel teilen

Comments

Leave a comment