Pod Distribution Drift on EKS: Descheduler Guide (2026)

Q: Why does my Kubernetes workload become unbalanced across zones after a node disruption?

Because topology spread constraints are evaluated by the scheduler only when a pod is placed, and Kubernetes never revisits a running pod to check the distribution again. When a zone loses capacity, through an instance replacement, a managed node group upgrade, a Spot interruption or a consolidation pass, replacement pods are placed in the zones that can take them. When capacity returns, nothing moves those pods back. Workloads whose autoscaler keeps creating new pods recover gradually as new replicas land in the restored zone, but workloads at a steady replica count stay skewed indefinitely. The Kubernetes documentation lists this as a known limitation: there is no guarantee constraints remain satisfied when pods are removed, and it points to the descheduler as the rebalancing tool.

Q: What is the difference between ScheduleAnyway and DoNotSchedule for topology spread?

DoNotSchedule is a hard constraint that acts as a filter: a pod that cannot be placed within maxSkew stays Pending until a zone has room. ScheduleAnyway is a soft constraint that acts as a scoring preference: the scheduler favours placements that reduce skew but will still place the pod in any zone that can take it, so the workload stays available and the distribution absorbs the imbalance. DoNotSchedule is the API default when whenUnsatisfiable is omitted. The trade is which failure you want to see. Soft constraints fail silently by drifting, hard constraints fail loudly with pending pods, and during a zone gap a hard-constrained fleet refuses to scale into the surviving zones beyond maxSkew. Neither re-evaluates pods that are already running, which is why drift accrues under both when capacity changes.

Q: How does the Kubernetes descheduler fix pod distribution drift?

The descheduler runs on a periodic loop as a Job, CronJob or Deployment. It is not an admission controller and it never places pods. It evaluates running pods against a policy, evicts the pods that violate it, and relies on the normal kube-scheduler to place the replacements. For drift the relevant plugin is RemovePodsViolatingTopologySpreadConstraint, which evicts the minimum number of pods needed to bring each constraint within maxSkew. By default it acts only on hard constraints; you list ScheduleAnyway in its constraints argument to make it act on soft ones. Because it uses the eviction API, PodDisruptionBudgets apply. In AWS's measured run it restored a skew of 1 in 183 evictions against a theoretical minimum of about 180, with no unready pods at any 30-second sample.

Q: Is it safe to run the descheduler in production?

It is safe when you treat every correction as a pod restart and control the blast radius. Put a PodDisruptionBudget on every workload in scope before enabling anything, because the eviction API refuses evictions that would violate the budget. Keep topologyBalanceNodeFit on so the descheduler skips pods that have no better zone to move to; in AWS's test it skipped every eviction whose only destination was a still-cordoned zone. Cap maxNoOfPodsToEvictPerNamespace at roughly 10 to 20 percent of the namespace's pods, run at intervals of 15 minutes or longer, start with one namespace of restart-tolerant workloads for 48 to 72 hours, and pause it during planned drains and cluster upgrades. It does not evict pods with local storage, DaemonSet pods or standalone pods by default. Applications with slow warm-up need lower caps.

Q: When should I not use the descheduler?

Skip it for workloads where a restart costs more than the skew: Kafka consumers in the middle of partition assignment, batch or training jobs close to a checkpoint, and anything with a long warm-up. Skip it where uneven placement is deliberate, such as pods pinned to a zone by an EBS volume, publish and subscribe groups, or service pairs kept in the same zone for latency. Skip it for very small Deployments: with two replicas across three zones one zone is necessarily empty, so skew 1 is the floor and a pass can restart half the workload for a result that can never reach zero. If zone balance is a correctness requirement, use a hard constraint and alert on Pending pods instead. If skew is affordable, monitor it and choose to do nothing, deliberately.

Q: How does a PodDisruptionBudget interact with the descheduler, and what is the small-Deployment trap?

The descheduler evicts through the eviction subresource, so it will not evict a pod if doing so would violate the pod's disruption budget; the budget limits how many pods of a workload can be unavailable at once, which throttles the rate of correction rather than the total. The trap is percentage rounding. Kubernetes rounds up the number of pods that may be disrupted when maxUnavailable is a percentage, so a 10 percent budget on a two-replica Deployment permits one disruption, half the workload. Use absolute PDB values for Deployments below roughly ten replicas and consider excluding them from descheduling. Also remember that PDBs do not govern Deployment rolling updates, which are handled by the Deployment controller, so a budget protects against evictions and drains but not against a bad rollout.

Q: How do I detect pod distribution drift before it causes an incident?

Measure skew per workload and per zone, with explicit zeros. Three pitfalls from AWS's walkthrough apply to a Prometheus setup. kube-state-metrics does not export the zone label by default, so you must add topology.kubernetes.io/zone to its label allowlist or every per-zone query returns empty. A zone with no pods produces no series at all, so a plain max minus min over per-zone counts understates skew exactly when it is worst; insert a zero for every zone that has nodes but no matching pods. And in CronJob mode the descheduler pod lives seconds, shorter than a scrape interval, so its eviction counter is never collected. Alert on per-workload skew above the configured maxSkew for a sustained period, not on the fleet aggregate, which can move the wrong way while a workload is corrected.

Q: What does drift cost beyond the risk of a zone outage?

The primary cost is resilience: a workload with half its pods in each of two zones and none in the third is less resilient to a single-zone disruption than its spec implies, and losing either surviving zone removes half the capacity instead of a third, which headroom sized for a three-way split may not cover. Secondary costs accumulate quietly. Traffic that would have stayed zone-local crosses zone boundaries and is billed in both directions. Subnet IP addresses concentrate in the surviving zones. Node utilisation becomes uneven, which distorts autoscaling signals and right-sizing decisions made from that data. None of these trigger an alarm on their own, which is why drift persists. They are worth quantifying before deciding whether the restarts caused by a descheduler are cheaper than the skew.

Satyam Kumar

عودة إلى المدونة

multi-cloud-infrastructure

A Topology Spread Constraint Is an Admission-Time Promise, Not a Runtime Invariant: Fixing Pod Distribution Drift on EKS with the Descheduler (2026)

By Satyam KumarSeptember 30, 202615 min read

pod distribution drift kubernetes descheduler topology spread constraints maxskew scheduleanyway eks availability zone imbalance pod disruption budget eviction removepodsviolatingtopologyspreadconstraint kubernetes rebalancing pods karpenter consolidation zones eks multi-az resilience kubernetes scheduler limitations topologybalancenodefit cross-az data transfer kubernetes kube-state-metrics zone label kubernetes platform engineering sre kubernetes

A Topology Spread Constraint Is an Admission-Time Promise, Not a Runtime Invariant: Fixing Pod Distribution Drift on EKS with the Descheduler (2026)

Frequently Asked Questions

شارك هذه المقالة

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

مهندس الذكاء الاصطناعي والسحابة. مساعدة الفرق على بناء أنظمة تتسع للملايين.

LinkedIn GitHub

A Topology Spread Constraint Is an Admission-Time Promise, Not a Runtime Invariant: Fixing Pod Distribution Drift on EKS with the Descheduler (2026)

Frequently Asked Questions

شارك هذه المقالة

Comments

Leave a comment