Secret Rotation at Scale: Zero-Outage Architecture

Q: Why does rotating all credentials after a breach take so long?

Because in most organisations rotation was never designed as a capability; it was a policy with a calendar attached. Rotating a single secret requires coordinating three parties: the issuer that creates the new value, the target system that must accept it, and every consumer that holds the old one. When an incident forces rotation of everything at once, three gaps surface simultaneously. The inventory is incomplete, so nobody knows which services use which secret, and copies in support tickets, CRM notes and CI logs are not in any store at all. Many consumers read their secret once at boot and never again, so rotating the value without a fleet restart breaks them, and a fleet restart during an incident turns a security event into an availability event. And credentials with no version identifier force a hard cutover, because the target accepts only one value at a time. The August 2025 Salesloft Drift campaign and the malicious nx releases both ended with the instruction to rotate everything that might have been exposed; teams that had rehearsed a rotate-everything action per credential class did it in hours, and teams that had a 90-day rotation policy but no architecture spent weeks and were left unsure whether they had finished. The time is spent on discovery and on avoiding outages, not on the rotation itself.

Q: How do you rotate a secret without causing an outage?

Use an expand-then-contract sequence with an overlap window in which both the old and the new credential are valid. First, issue the new credential at the target while the old one still works. Second, test the new credential against the target from the rotation controller, not from production traffic. Third, promote it in the secret store so that consumers fetching the current version receive the new value. Fourth, let consumers refresh, either on their cache time-to-live, on an authentication failure, or on a push event emitted when the secret was promoted. Fifth, and this is the step most teams skip, watch the target’s own authentication logs until the old credential is no longer being presented. Only then revoke it. AWS Secrets Manager documents this in its alternating-users rotation strategy for databases: two database users exist, the rotation function sets a new password on the inactive one and then switches which user the secret points at, so a client that fetched the secret moments before rotation keeps working until its next fetch. The single-user strategy, which changes the password on the one user, is a cutover and is only appropriate for consumers that can tolerate a brief failure. The generalisation is to give every credential type two slots with distinct identifiers, so that revoking one never revokes the service.

Q: What is the difference between static secret rotation and dynamic secrets?

Static rotation periodically replaces a long-lived shared value, such as a database password or an API key, that every consumer of that target uses. It reduces the useful lifetime of a stolen value to the rotation interval, but the value is still shared, still long-lived between rotations, and still has to be distributed and refreshed across all consumers. Dynamic secrets, as implemented in HashiCorp Vault and similar engines, invert the model: each consumer requests its own unique credential on demand, the engine creates it at the target with a lease and a time-to-live, and the consumer must renew the lease to keep using it. When the lease expires or is revoked, the engine deletes the credential at the target. There is no shared value to rotate, a leak is scoped to one consumer, and revoke-everything means revoking the leases and waiting for the TTL rather than coordinating a fleet-wide change. The honest trade-off is that the secrets engine becomes a tier-zero dependency: if it is unavailable and leases expire, every consumer loses access at once, so it needs the availability design of a database. Dynamic secrets are the right destination for internal credentials such as database users and cloud access; they are unavailable for the long tail of third-party SaaS that issues one static key per account.

Q: What did the Salesloft Drift and Nx incidents teach about credential rotation?

Both incidents in August 2025 turned credential rotation from a compliance task into an incident-response capability, and both exposed the same gaps. In the Drift campaign, Google Threat Intelligence Group reported that the actor UNC6395 used compromised OAuth tokens associated with the Salesloft Drift integration between 8 and 18 August to export large volumes of data from numerous corporate Salesforce instances, and then searched the stolen data for further credentials such as AWS access keys, passwords and Snowflake tokens. Salesloft and Salesforce revoked all active Drift access and refresh tokens on 20 August, but every affected organisation then had to find and rotate whatever secrets had been sitting in support cases and CRM records. The lesson is that rotation covers the secret store, while the copies that attackers actually harvest live outside it, which means secret scanning and redaction of tickets and logs are part of the rotation programme. In the Nx advisory published on 27 August, malicious versions of the nx package and several plugins scanned developer machines for credentials such as GitHub and npm tokens and SSH keys and published them to attacker-created repositories; the advisory told affected users to rotate. The lesson there is that developer workstations hold standing credentials that rarely appear in any inventory. In both cases the organisations that could act quickly were those that had already defined a rotate-everything action per credential class and rehearsed it.

Q: How should you build a credential inventory for rotation at scale?

Generate it from evidence rather than maintaining it by hand, because a hand-maintained inventory is wrong by the second week. The trustworthy sources are the ones that cannot lie: the audit log of which principals fetched which secret from the secret store, such as CloudTrail for AWS Secrets Manager; the target systems’ own authentication records, such as database login logs showing which users connected from which hosts; API gateway records of which keys were presented; identity-provider records of token issuance; and secret-scanning findings across repositories, CI logs and ticketing systems. Reconciling these against the secret store’s list of secrets yields the two lists that matter. Secrets that nobody has fetched in ninety days are candidates for deletion, which shrinks the rotation surface. Fetches or authentications using values that are not in the store are the shadow copies you would otherwise discover during an incident. NIST SP 800-57 frames the same discipline for cryptographic keys as the cryptoperiod: every key has a defined lifetime and known consumers, and a key with neither is a liability. Once the inventory is generated continuously, classify each secret by target type, because the rotation state machine differs for database users, cloud credentials, third-party API keys, signing keys and OAuth clients, and the rotation controller needs one implementation per class.

Q: What is the consumer contract for secret rotation?

It is the short set of behaviours every service must implement so that it can pick up a rotated secret without a restart or a deploy, and it is where most rotation-related outages originate. A compliant consumer caches a secret for no longer than a fixed time-to-live, typically a few minutes, so that a promoted value propagates within that window. It refreshes the secret and retries once on an authentication failure before failing the request, so that a consumer that fetched moments before rotation self-heals. It subscribes to a push event emitted when a secret is promoted, so that the fleet converges in seconds rather than at the next cache expiry. And it never persists a secret to disk or bakes it into an image, because those copies fall outside every rotation mechanism. For dynamic credentials the contract is the lease: renew before expiry, treat revocation as a signal to obtain a new lease, and handle the case where the issuer is temporarily unavailable without discarding a still-valid lease. Services that read a secret once at boot cannot be enrolled in a rotation programme regardless of how good the controller is, so fixing them is a prerequisite rather than a later phase. Measuring the fleet’s convergence time after a promotion, from the moment the new version is current until the last consumer stops presenting the old one, is the operational metric that tells you whether the contract is actually implemented.

Q: How often should secrets be rotated?

Less often than you think as a security control, and far faster than you think as an incident capability. Rotating a static secret on a 90-day schedule is a compliance artefact: if the value was stolen on day one, the attacker has 89 days, and the schedule does nothing about the copies the attacker already took. The security property that matters is time-to-worthless, meaning how quickly a leaked credential can be made useless end to end, and that is governed by architecture rather than cadence. For credentials that can be made dynamic, the effective rotation interval is the lease TTL, often minutes, and revocation is immediate. For credentials that must remain static, the schedule should be whatever the platform manages for you, with the real investment going into the overlap window, the consumer refresh path and the evidence-gated revoke that let you rotate on demand without an outage. NIST SP 800-57 provides the cryptographic framing through cryptoperiods, which are defined lifetimes based on the sensitivity of what the key protects and the exposure of its use, and the same reasoning applies to application secrets. The practical answer is to define a rotation service-level objective per credential class, meaning the maximum time to make every credential in that class worthless, and to rehearse meeting it quarterly, because the one number a Drift or Nx victim needed on day one was whether they had a two-hour problem or a three-week one.

Q: Does secret rotation protect against session token theft or over-privileged credentials?

No, and understanding what rotation does not cover is as important as implementing it. Rotation shortens the lifetime of a credential at rest and makes it possible to revoke a leaked value quickly, but it does not help when an attacker steals a token from a live session and uses it within its validity window; that threat is addressed by proof-of-possession mechanisms such as DPoP and token binding, and by session anomaly detection, not by rotating the underlying secret. Rotation also does not shrink what a leaked credential can do while it is valid: a rotated credential that still carries thousands of effective permissions is still a catastrophic leak for the hour it lives, which is the entitlement problem that cloud infrastructure entitlement management and least-privilege account design address. And rotation does not reach copies of secrets that live outside the store, in support tickets, CRM notes, CI logs and screenshots, which is precisely where the Salesloft Drift attacker went hunting for AWS keys and Snowflake tokens; secret scanning and redaction have to cover those. The complete picture is therefore three separate programmes that reinforce each other: rotation and dynamic issuance to bound lifetime, least privilege to bound blast radius, and proof-of-possession plus scanning to bound the value of a copy. Treating rotation as the whole answer leaves two of the three doors open.

Satyam Kumar

返回博客

security

Secret Rotation at Scale: The Architecture Behind "We Rotated Everything" (2026)

By Satyam KumarSeptember 6, 202613 min read

secret rotation secrets management credential rotation dynamic secrets hashicorp vault aws secrets manager workload identity rotation without downtime alternating users rotation salesloft drift breach nx supply chain attack oauth token theft credential leak response secret sprawl nist sp 800-57 incident response

Secret Rotation at Scale: The Architecture Behind "We Rotated Everything" (2026)

Frequently Asked Questions

分享这篇文章

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

人工智能和云架构师。帮助团队构建可扩展到数百万的系统。

LinkedIn GitHub

Secret Rotation at Scale: The Architecture Behind "We Rotated Everything" (2026)

Frequently Asked Questions

分享这篇文章

Comments

Leave a comment