Chaos Engineering Experiment Design: A 2026 Guide

Q: What is the difference between chaos engineering and fault injection?

Fault injection is the act of introducing a failure: killing a process, adding latency, dropping a dependency. Chaos engineering is the experimental discipline that uses fault injection to test a hypothesis. The Principles of Chaos Engineering define it as experimenting on a system to build confidence in its ability to withstand turbulent conditions, through four steps: define a measurable steady state, hypothesise it will hold, introduce real-world variables, and try to disprove the hypothesis. Without a stated steady state and a prediction that could fail, injecting a fault is a demonstration, not an experiment, and it rarely teaches you anything you did not already believe.

Q: What should I use as the steady-state metric?

A service level indicator that users would notice, such as orders completed per minute, login success rate, or p99 latency for a critical journey. Infrastructure metrics like CPU, memory and pod count make poor steady-state signals because they can look healthy while users fail, and can spike while users are entirely unaffected. The steady-state metric is also what your automatic abort condition should watch. If you cannot observe the user-facing metric in near real time, you are not ready to run experiments, because you will not know whether the system is holding until customers tell you.

Q: How much does chaos engineering tooling cost in 2026?

Very little relative to the engineering time. AWS Fault Injection Service pricing lists $0.10 per action-minute, with an additional $0.10 per action-minute for each additional target account, and AWS gives a worked example of 50 action-minutes costing $5. Open-source tools such as LitmusChaos are free to run, and Azure offers Chaos Studio. The meaningful costs are designing experiments with real hypotheses, building blast-radius controls and observability, the people time for game days, and fixing what the experiments find. Budget for those rather than for the tool licence.

Q: How do I limit blast radius so an experiment does not cause an outage?

Use the boundaries your architecture already has before adding procedural ones. Run inside one cell, one shard, one zone, one tenant tier or a small percentage of traffic routed through a faulted path. Start in staging with synthetic load, then internal users, then a single cell, and widen only after the hypothesis holds. Wire automatic abort conditions to the steady-state metric and test them by running a deliberately failing experiment. Plan recovery in advance for faults that stopping the tool cannot undo, such as deleted resources or filled queues. If your system has no internal boundaries, the blast radius is everything and the first finding is that you need boundaries.

Q: Should chaos experiments run in production?

Eventually, for mechanisms already proven elsewhere, because only production has real traffic, real data shapes and real dependencies. But production is the last step, not the first. Prove the hypothesis in staging, then on internal traffic, then on one cell or a small percentage with automatic abort on a user-facing metric. Never run production experiments while known findings from earlier runs remain unfixed, because that re-proves a known weakness on customers. Small teams with a single region and no redundancy usually gain more from a design review and tabletop exercise, since there is nothing yet to fail over to.

Q: Could chaos engineering have prevented the October 2025 AWS us-east-1 outage?

Not the root cause. AWS described a latent race condition inside DynamoDB automated DNS management that left the regional endpoint with an empty DNS record, with cascading effects on EC2 launches, Lambda, EKS, STS and other services. That bug lived inside AWS and was outside any customer experiment space. What customer experiments could have done is reveal in advance how their own systems behave when a regional dependency stops resolving: whether they fail over, degrade gracefully, retry into a storm or fail closed. Many teams learned that answer live during the incident instead of in a controlled experiment.

Q: Why do chaos engineering programmes stall?

Most often because findings carry blame. A good experiment shows that a team system is weaker than they believed. If that finding appears in a manager scorecard or triggers a post-incident style inquisition, teams learn to propose experiments that cannot fail, and the programme becomes a stream of green results that teaches nothing. Apply the blameless postmortem standard from Google SRE to experiment findings, let owning teams design experiments on their own systems, count disproved hypotheses as positive output, and measure the programme by mechanisms fixed rather than experiments run.

Q: Do chaos experiments satisfy resilience testing requirements such as DORA?

They can contribute evidence, but only if they are documented as experiments. A list of faults injected is not evidence of resilience. Auditors and regulators expect to see what was tested, why, the expected behaviour, the observed result, and what was done about failures. Hypothesis-driven experiments naturally produce that record: a stated steady state and hypothesis, the fault and scope, the result, the finding owner and the fix date, and the continuous check that now guards it. Pair this with scenario-based testing for third-party and severe-but-plausible events, which chaos experiments alone do not fully cover.

Satyam Kumar

Назад к блогу

multi-cloud-infrastructure

A Chaos Experiment Is a Hypothesis Test, Not a Fault Injection: Steady State, Blast Radius, Abort Conditions and the Blame Problem (2026)

By Satyam KumarOctober 1, 202611 min read

chaos engineering chaos experiment design steady state hypothesis blast radius aws fault injection service fis stop conditions game day resilience testing continuous verification sre reliability programme blameless postmortem fault injection testing site reliability engineering dora resilience testing cell-based architecture aws us-east-1 outage october 2025

A Chaos Experiment Is a Hypothesis Test, Not a Fault Injection: Steady State, Blast Radius, Abort Conditions and the Blame Problem (2026)

Frequently Asked Questions

Поделиться статьёй

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

AI & Cloud архитектор. Помогаю командам строить системы, масштабируемые до миллионов.

LinkedIn GitHub

A Chaos Experiment Is a Hypothesis Test, Not a Fault Injection: Steady State, Blast Radius, Abort Conditions and the Blame Problem (2026)

Frequently Asked Questions

Поделиться статьёй

Comments

Leave a comment