Engineering Tech Lead, vCluster For decades, infrastructure provisioning followed a predictable script. A systems engineer would look at a steady incline in server utilisation, factor in a modest buffer for peak traffic, and plan accordingly. Sometimes it meant more drives and RAM for a modest capacity uplift. Other times, a few more blades had to be ordered for the rack. It was a world governed by predictable growth and the reliable rhythm of Moore’s Law.
That model no longer holds.
The catalyst is the rise of large language models and generative AI. As organisations race to train, fine-tune, and deploy these workloads, data center infrastructure is experiencing a different demand profile. AI workloads do not climb a gentle slope. A large GPU cluster may sit relatively idle during data preparation, then demand hundreds of kilowatts of power and substantial network throughput when a training run begins.
Traditional capacity planning, built on predictable steady growth, is not suited to that pattern. Infrastructure leaders need to reconsider how they architect, power, and fund GPU clusters.
Understanding AI workload profiles
Traditional applications tend to exhibit a baseline of activity with predictable daily or seasonal peaks. Capacity planners use statistical models to project this usage forward.
AI clusters, however, are characterised by a “burst and plateau” profile. During a training phase, hundreds or thousands of GPUs must act as one distributed computer. When execution begins, the cluster can move from low utilisation to near 100% almost immediately. This state may persist for days, weeks, or months. Once the run completes or checkpoints, utilisation drops again.
This creates a difficult choice. If you size infrastructure for peak training load, you face substantial capital expenditure on hardware that may sit idle between runs. If you size for average utilisation, data scientists face queues, and training runs may fail because the required resources are not available together.
Power and cooling are the constraints
In the past, capacity was often discussed in terms of floor space and server counts. For high-density AI, power delivery and thermal management are usually the harder constraints.
Many legacy enterprise environments were designed around single-digit or low-double-digit kilowatts per rack. By comparison, NVIDIA’s published design for a GB200 NVL72 rack calls for 120 kW of cooling capacity and direct liquid cooling. That density changes both facility design and operations.
When a training run starts, planners cannot look only at total monthly kilowatt-hours. They must understand transient step loads and how quickly the power-delivery infrastructure can respond to demand.
Forced-air cooling cannot handle every high-density configuration. Direct-to-chip liquid cooling and rear-door heat exchangers have to be considered as part of the capacity equation. If a facility cannot dissipate heat at full load, expensive silicon will thermally throttle and training timelines will slip.
The network fabric is part of the computer
In a conventional web application, networking is often not the primary constraint. In a GPU cluster, the network fabric is part of the computer.
During distributed training, models are split across hundreds or thousands of nodes using techniques such as data parallelism and tensor parallelism. At the end of forward and backward passes, these nodes synchronise weight gradients through collective communication operations such as All-Reduce.
This creates bursty east-west traffic. If one link experiences packet loss or jitter, the training run can slow while GPUs wait for the straggler to catch up.
Capacity planning for AI therefore requires dedicated, non-blocking back-end fabrics, typically InfiniBand or RDMA over Converged Ethernet (RoCE). Planners cannot assume that an oversubscribed network will be sufficient. Current AI networking platforms operate at 400 Gb/s and 800 Gb/s, which makes topology, congestion control, cabling, and operational discipline part of cluster capacity.
A dense network rack. Cable bend radius, airflow, service access, and tray capacity become operating constraints at this scale. Photo provided by the author.
The volume of physical cabling required for distributed networking can overwhelm a traditional data center. I have seen InfiniBand racks with so many active optical cables spilling out of the back that the doors had to be removed. Some cables were kinked or damaged on the way to the GPU racks. The bundles restricted airflow and consumed overhead tray capacity needed for Ethernet copper and fibre.
A practical capacity-planning playbook
The response is to move from static provisioning to a more dynamic strategy.
1. Use hybrid architecture where it fits
Building an on-premises data center for peak AI training load is financially unrealistic for many enterprises. A hybrid model can keep a steady-state cluster on premises for data curation, smaller fine-tuning tasks, and low-latency inference, then use hyperscalers or specialist GPU clouds when a larger training run is required.
This only works if the data path, security model, software environment, and commercial capacity are prepared in advance. “Cloud bursting” is not an emergency button if transferring data or acquiring the right cluster takes weeks.
2. Treat scheduling as capacity management
Spiky demand has to be managed in software. Kubernetes and workload managers such as Slurm support priorities and preemption. Used carefully, priority queues, quotas, and checkpointing can pause or requeue lower-priority work so that a high-priority training run receives a complete allocation.
The scheduler cannot create power, cooling, or network capacity. It can, however, decide which work uses those constrained resources and reduce avoidable idle time.
3. Use fractional GPUs for smaller workloads
Not every AI task requires a dedicated cluster of H100 GPUs. For inference and smaller fine-tuning pipelines, NVIDIA Multi-Instance GPU (MIG) and other GPU virtualisation methods can partition a physical accelerator into smaller isolated instances.
Packing smaller workloads onto a card can improve baseline utilisation between large training runs. It is also useful for GPU labs where developers need test environments and sales engineers need smaller demonstration sandboxes.
4. Plan around shorter silicon cycles
AI hardware moves faster than the traditional server estate. Product generations may turn over in roughly 18–24 months, while enterprise server depreciation commonly assumes 3–5 years.
Capacity planning must therefore include the financial lifecycle of the hardware. Shorter leases, subscription access, and planned replacement can reduce the risk of being left with silicon that consumes more power for less useful work than the next generation.
Plan for volatility
Bursty AI clusters have forced software architecture, systems engineering, and physical facilities management into the same capacity conversation.
Planning for the next GPU cluster means looking beyond CPU utilisation and storage growth. Capacity planners must balance high-density power, cooling limits, non-blocking network fabrics, dynamic scheduling, external capacity, and the financial life of the hardware.
The linear spreadsheet is still useful, but its inputs have changed. The practical goal is not to eliminate volatility. It is to design infrastructure and operating processes that can absorb it.