Silent Data Corruption in GPU Fleets: 2026 Guide

Q: What is silent data corruption on a GPU?

Silent data corruption, or SDC, is when a processor returns a wrong result without raising any error. Memory protected by ECC can detect and often correct bit flips, and the driver reports Xid errors for many faults, but datapath and logic defects in parts of the chip that have no check logic produce incorrect arithmetic that nothing flags. Meta's Silent Data Corruptions at Scale paper described exactly this for CPUs: the errors are not captured by the processor's reporting mechanisms, so they are not traceable at the hardware level, and they show up as application problems that can take months to debug. In GPU training the effect is amplified because data-parallel all-reduce spreads a wrong gradient from one device to every replica, and because training is noisy enough that a small perturbation looks like ordinary variance or a data issue.

Q: How common are hardware failures in large LLM training runs?

The best public number is Meta's Llama 3 report. During a 54-day snapshot of 405B pre-training on up to 16K H100 GPUs, the team recorded 466 job interruptions: 47 planned for maintenance and 419 unexpected. About 78 percent of the unexpected interruptions were attributed to confirmed or suspected hardware issues. Faulty GPUs accounted for 148 interruptions, GPU HBM3 memory for 72, software bugs for 54, network switches and cables for 35 and unplanned host maintenance for 32. Silent data corruption appears as 6 interruptions, 1.4 percent. Read that last number carefully: the table counts events that stopped the job, so it only includes corruptions that some check caught. Corruptions that did not interrupt anything are not in it. Meta still achieved more than 90 percent effective training time, largely through automation and fast checkpointing.

Q: Why is a crashing GPU cheaper than one that silently corrupts data?

Because the cost of a crash is bounded and the cost of a silent error is not. When a GPU crashes, the job stops, the scheduler replaces the node and training resumes from the last checkpoint, so you lose the steps since that checkpoint and the restart time. When a GPU computes wrong answers silently, it keeps contributing corrupted gradients that the all-reduce distributes to every replica, for as long as nobody notices. Ma and colleagues showed in 2025 that real SDCs from unhealthy cloud nodes can push a model to a different optimum with different weights and cause loss spikes, while most individual perturbations look small. The bill is every step since the device went bad, the compute to re-run them, the engineering time spent chasing a phantom regression, and potentially a slightly worse model shipped to customers.

Q: How do you detect silent data corruption during a training job?

With cheap invariants that run inside the job and a replay step that turns suspicion into evidence. In pure data parallelism every replica applies the same averaged gradient, so sampled checksums of a slice of parameters across replicas every few hundred steps should agree; a replica that disagrees names its node. Loss and gradient-norm anomaly detection catches the spike pattern that SDC produces. NaN and Inf counters catch gross numerical blow-ups, and a burst of Xid or correctable ECC events is treated as a trigger even below the vendor's alert threshold. None of these is conclusive alone. When one fires, mark the node Suspect, re-run the window of steps since the last clean checkpoint on known-good nodes with the same data order, and compare bitwise. A match is a false alarm to tune; a divergence implicates the node with evidence.

Q: How often should GPUs be screened with DCGM diagnostics?

More often than once at delivery, and on triggers as well as on a schedule. NVIDIA's documentation gives the run-level budgets on Hopper systems: level 1 under 2.5 seconds, level 2 under 2.5 minutes on 4 GPUs and under 10.5 minutes on 8, level 3 under 10 minutes on 4 and under 35 minutes on 8, and level 4 under 45 minutes on 4 and under 2.25 hours on 8, each including the tests beneath it. A reasonable starting cadence is level 3 or 4 at admission and after any RMA, firmware change or thermal event, a drained level-2 re-screen weekly or fortnightly, and a short known-answer check at every job start. Then tune the cadence against your own detection history rather than a default, because the right interval depends on how quickly faults appear in your fleet.

Q: What is a correctness lease for an accelerator?

It is a record attached to each GPU or node saying which correctness tests it last passed, when, and when that result expires. The scheduler only places training work on nodes with an unexpired lease. Screening issues the lease, a passing periodic re-screen renews it, and in-job triggers such as a cross-replica checksum mismatch, a loss anomaly or an Xid burst revoke it and move the node to a Suspect state for replay. The point of the idea is to turn a vague goal, keep the GPUs healthy, into a state machine the platform can enforce automatically. On Kubernetes the lease can be a node condition or label and quarantine a taint; on Slurm it lives in node features. The anti-pattern it replaces is a health list kept by humans that the scheduler never reads.

Q: Does FP8 training make silent data corruption worse?

It changes the risk profile, and the research is still young, so be wary of anyone quoting a precise number. LLM-PRISM ran 7,664 fault-injected training runs across FP16, BF16 and FP8 and found that the impact of hardware faults depends on numeric format as well as fault type and rate, with particular formats able to cause catastrophic divergence at moderate fault rates. TrainSDC found backward-pass vulnerability is governed by gradient exponent distributions, which is exactly what narrow formats and their scaling factors manipulate. The practical guidance is to re-baseline every in-job anomaly threshold when you move from BF16 to FP8 rather than carry thresholds over, and to treat loss-scaling and amax-history anomalies as possible SDC triggers as well as numerics bugs, until your own fleet data says otherwise.

Q: When is building SDC detection not worth it?

When jobs are small and short. If you fine-tune on eight GPUs for a few hours, run the vendor's level-2 or level-3 diagnostic when you get the machine, watch the loss curve, keep a few checkpoints and rerun if something looks odd; the chance of an SDC in that job is low and a rerun is cheap. The lease state machine, in-job checksums, deterministic replay and long checkpoint ladders earn their cost when jobs run for days or weeks across hundreds or thousands of accelerators and a wrong answer means a model you ship. Even then, be honest about limits: screening proves only that a device passed the tests you ran, checksums miss corruption that is averaged in before the all-reduce, and replay is too expensive to run without another signal. Inference fleets need a different defence, known-answer probes and per-node quality canaries.

Satyam Kumar

Back to Blog

architecture

Silent Data Corruption in GPU Fleets in 2026: The GPU That Crashes Is the Cheap One, So Treat Correctness as a Test Result With an Expiry Date

By Satyam KumarOctober 24, 202620 min read

silent data corruption sdc gpu gpu fleet reliability llm training failures llama 3 hardware failures dcgm diagnostics gpu health checks hbm3 errors trainsdc cross-replica checksum deterministic replay training checkpoint strategy llm training fp8 training numerics gpu quarantine ml infrastructure reliability accelerator hardware faults

Silent Data Corruption in GPU Fleets in 2026: The GPU That Crashes Is the Cheap One, So Treat Correctness as a Test Result With an Expiry Date

Frequently Asked Questions

Share this article

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

AI & Cloud Architect. Helping teams build systems that scale to millions.

LinkedIn GitHub

Silent Data Corruption in GPU Fleets in 2026: The GPU That Crashes Is the Cheap One, So Treat Correctness as a Test Result With an Expiry Date

Frequently Asked Questions

Share this article

Comments

Leave a comment