You Ran Out of Power Before You Ran Out of Racks

Q: Why is power the constraint on AI capacity now instead of floor space?

Because the density changed by an order of magnitude and the supply side did not keep up. For thirty years a capacity plan started with "how many racks fit," and power was sized afterwards, because a traditional rack drew 5 to 10 kW and any reasonably-built hall could feed a lot of them. Modern AI racks draw 120 to 150 kW each — a single one is the power and heat of three to four fully-loaded traditional racks. That inverts the whole planning order. You no longer ask how many racks fit; you ask how many kilowatts the site can actually deliver and when, and everything else — rack count, accelerator choice, cooling method, even whether you build or rent — is derived from that envelope. The visible symptom is a hall that is half empty on the floor but full at the substation and the busway. Teams that still plan in rack units discover the mismatch at commissioning, which is the most expensive possible moment to discover it, because the silicon is already bought and the launch is already announced.

Q: How much power does a modern AI rack actually draw?

A rack-scale AI system like the NVIDIA GB200 NVL72 draws on the order of 120 to 150 kW, and next-generation designs are already pushing higher. For comparison, a conventional enterprise rack was provisioned at 5 to 10 kW, and even a dense general-compute rack sat around 15 to 20 kW. So one AI rack is roughly the power draw of an entire row of older equipment. Two consequences follow. First, the number that matters for planning is facility power, not GPU nameplate wattage — the building also pays for cooling, distribution losses and overhead, so the real draw is the PUE-inflated figure, meaningfully above the compute number. Second, the memory system is a large and often-overlooked share of that power: high-bandwidth memory moving hundreds of gigabytes per second per stack consumes real wattage, which is why accelerator selection is partly a power-efficiency decision and not only a question of peak FLOPs. Planning to nameplate GPU wattage systematically under-provisions the distribution and cooling that actually gate deployment.

Q: Why can I not just add more air conditioning to cool dense racks?

Because air cooling hits a hard physical ceiling around 30 to 35 kW per rack, and above it the approach stops working regardless of how much cooling you add. The problem is the volume of air you must move through the server chassis to carry the heat away. As density climbs past the ceiling, the airflow required drives fan power consumption and acoustic noise to levels that are untenable — you spend an escalating fraction of your power budget just moving air, and the servers cannot physically expel heat fast enough. That is why direct-to-chip liquid cooling, rear-door heat exchangers and immersion have moved from exotic to standard for anything modern. Liquid carries far more heat per unit volume than air, so it is the only way to clear a 120 kW rack. The architectural consequence is that cooling becomes a first-class facility system with its own plumbing, failure domains and maintenance model, rather than a set of CRAC units bolted to the room. Designing air-first for modern accelerators means a cooling re-fit before your first deployment, which is both expensive and slow.

Q: What is a grid interconnection queue and why does it take years?

A grid interconnection queue is the utility and grid-operator process that a new large electrical load — or a new generator — must go through before it can be energised, including studies of how the connection affects grid stability and what upgrades the wider network needs. For a large data centre this is the single longest lead-time item in the whole project. In constrained US markets, securing new power capacity through the queue now runs three to four years, which is frequently longer than it takes to design and build the facility itself. The reasons are structural: the volume of new load requesting connection has surged, the studies are sequential and interdependent, and the transmission upgrades they identify take years to construct. Critically, this is not an engineering problem you can spend your way out of on your own site — the timeline belongs to the utility and the grid operator. The practical implication is that your capacity date is a grid date. If you did not start the interconnection process two years before you need the megawatts, no amount of GPU allocation, capital or construction speed closes the gap, and the honest response is to look at a different region, a colocation provider who already energised, or cloud capacity.

Q: Can I put high-density AI racks in my existing data center?

Usually not without rebuilding the power and cooling spine first, even when there is plenty of empty floor. A hall built for 5 to 10 kW racks has busways, power distribution units, breakers and often a transformer all sized for that density. Dropping 120 kW racks into it is not a rack swap — it is a re-wire of the power distribution backbone and frequently a transformer upgrade, because the existing copper and switchgear physically cannot carry the current. On top of that, the room almost certainly has air cooling, which as covered above cannot clear the heat of dense racks at all, so you are also adding liquid cooling infrastructure. The trap is reading empty floor as spare capacity. Floor space is the one thing you are not short of; the constraints are the distribution layer inside the building and the feed coming into it. The correct first step is an honest assessment of what the existing power spine can deliver to a rack position and what the cooling can remove from it, because those two numbers, not the square footage, decide whether the room can host modern racks without a rebuild.

Q: Given power constraints, should I build, colocate, or use cloud?

Decide it against a calendar and a megawatt number rather than against unit price, which is the usual but wrong first axis. If you own or can acquire a site but the grid interconnection date is two to four years out and the business needs capacity in twelve months, building is not an option for that deadline no matter how attractive the long-run economics look — the grid queue makes the decision for you. Colocation is the fastest path to already-energised, high-density-ready capacity, because a good provider has already fought the interconnection and cooling battles; you pay a premium to skip the queue. Cloud is the right answer when demand is uncertain, spiky, or needed immediately, because you are renting someone else power envelope by the hour and can exit if the workload changes. The mature position treats this as a live, ongoing decision rather than a one-time procurement: build where you have secured long-lead grid capacity for predictable baseline load, colocate for high-density needs inside the build lead time, and use cloud for burst and uncertainty. The deciding variables are where megawatts are actually available and when, not the headline cost per rack.

Q: How much electricity will AI data centers use by 2030?

The IEA Energy and AI analysis projects global data-centre electricity consumption to more than double between 2024 and 2030, reaching about 945 terawatt-hours by the end of the decade — roughly equivalent to the entire current electricity demand of Japan, and just under 3% of global electricity. The growth is heavily concentrated in AI-accelerated servers, whose consumption grows far faster than conventional servers, and geographically in the United States and China, which together account for close to 80% of the global increase. In the US specifically, the Department of Energy 2024 report put data centres at 4.4% of national electricity in 2023 — about 176 TWh — and projected 6.7% to 12% by 2028, a range of roughly 325 to 580 TWh. The reason these figures matter for an individual architect is that demand at this slope is exactly what created the multi-year interconnection queues: the aggregate load is growing several times faster than the grid can add transmission and generation, so scarcity of grid capacity is not a temporary blip but the structural condition you are planning inside of for the rest of the decade.

Q: Does improving GPU utilisation actually help with power constraints?

Yes, and it is the fastest capacity you will find, because it does not depend on the grid at all. When power is the binding constraint and the interconnection queue is measured in years, a cluster running at 20% utilisation is wasting the single scarcest resource you have — every idle watt is a watt of grid capacity you waited years for and are now not using for useful work. Improving utilisation and cutting per-inference power draw directly extends how much work fits inside a fixed megawatt envelope, which is functionally identical to acquiring more capacity except that it takes weeks instead of years and requires no utility. This reframes efficiency work: right-sizing, better batching and scheduling, reducing idle fleet, and inference-level power reductions are not sustainability nice-to-haves in a power-constrained world, they are capacity-creation levers. The correct sequence is therefore to fix utilisation before buying more power, because migrating a poorly-utilised fleet into a larger envelope just buys a more expensive way to run idle capacity. The unglamorous work of reclaiming stranded capacity usually pays sooner and more reliably than waiting in the queue for more.

Satyam Kumar

Retour au blog

ai-architecture

You Ran Out of Power Before You Ran Out of Racks

By Satyam KumarAugust 5, 202610 min read

data center power density ai datacenter liquid cooling grid interconnection rack power density nvidia gb200 direct-to-chip cooling power usage effectiveness ai infrastructure capacity planning data center energy gpu cluster colocation grid capacity ai capacity planning 2026

You Ran Out of Power Before You Ran Out of Racks

Frequently Asked Questions

Partager l'article

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

Architecte AI & Cloud. J'aide les équipes à construire des systèmes qui scalent pour des millions.

LinkedIn GitHub

You Ran Out of Power Before You Ran Out of Racks

Frequently Asked Questions

Partager l'article

Comments

Leave a comment