Building AI infrastructure that can keep up with business growth
Most AI projects do not fail at the model. They fail the first time real usage shows up.
A pilot that served twenty people politely can fall over at two thousand. A system that handled a few hundred queries a day can start costing more each month than the team that built it.
The fix is rarely a better model. It is capacity planning, and in most companies it happens about a year later than it should.
Growth breaks AI systems in a predictable order
Scaling failures arrive in sequence rather than all at once, which is useful if you know the running order.
The data pipeline goes first. A model in production needs fresher, cleaner and more frequent inputs than a demo ever did, and the pipeline that was fine for a proof of concept is usually the first thing to buckle.
Throughput goes second, when concurrent users turn a comfortable batch job into a queue.
Cost goes third, and it is normally discovered in a finance review rather than on a monitoring dashboard.
The physical envelope goes last: power, cooling, rack space, and the contract terms attached to all three. It fails least often and takes longest to fix, which is exactly the wrong combination.
Planning in that order costs very little. Discovering it in that order costs a quarter.
Training and inference are two different buying problems
The single most expensive habit in AI infrastructure is treating training and inference as one line item.
Training is bursty. It runs hard for days or weeks, tolerates latency, and cares about memory capacity, memory bandwidth and the interconnect between accelerators.
Inference is the opposite. It runs constantly, it is judged on response time, and its demand curve tracks your user growth rather than your research calendar.
Sizing one purchase for both leaves you with hardware that is idle most of the month and still somehow short during a training run. Separating them early gives you two smaller, clearer questions: how much burst capacity does research need this year, and what does steady-state serving cost per user.
The second question is the one that decides whether the product has a margin.
Buy capacity in tiers, not in one purchase
Very few workloads need the newest silicon available. Frontier training does. Fine-tuning, evaluation runs, smaller-model training, embedding generation and overnight batch jobs generally do not.
A tiered fleet reflects that. Newest parts for the work that genuinely needs them, previous-generation parts for the large middle of the workload, and rented cloud capacity for peaks you cannot predict.
This is also how the economics stay sane. Buying the top tier for everything means paying frontier prices to run jobs that would have finished overnight on hardware two generations back.
That is why a real secondary market has grown up around previous-generation data centre parts, and why A100 GPUs for AI training still turn up in working fleets years after the architectures that replaced them shipped.
The A100 is a good illustration of why older accelerators stay useful. NVIDIA put it into full production in May 2020, it ships in 40GB and 80GB HBM2e configurations, and its Multi-Instance GPU feature can partition a single card into as many as seven isolated instances, each with dedicated compute and memory (NVIDIA MIG User Guide).
That last property matters more than raw benchmark position when several teams share a fleet. One card serving seven contained experiments beats one card monopolised by whoever booked it first.
Whatever tier you buy into, insist on documentation: the exact configuration, the test record, and the warranty term in writing. Hardware with no paperwork is a maintenance problem wearing a discount.
The costs that never appear on the quote
Accelerators are quoted as a unit price and consumed as a power bill.
An A100 80GB in PCIe form has a 300W thermal design power rating, and the SXM version is rated at 400W. Multiply that across a populated chassis and the constraint stops being budget and starts being amps, airflow and floor loading.
Colocation contracts reflect this. Space is increasingly sold by power draw rather than by rack, so a dense AI deployment can outgrow a contract that still has plenty of physical room left in it.
Networking is the other quiet line item. Distributed training moves enormous volumes of traffic between nodes, and an interconnect sized for ordinary application servers will happily throttle a cluster you paid a great deal for.
None of this is a reason to avoid owning hardware. It is a reason to have the facilities conversation before the purchase order, not after the pallet arrives.
The pipeline usually fails before the hardware does
A surprising share of “we need more GPUs” conclusions are really data problems in costume.
Accelerators sitting at low utilisation are often starved rather than overloaded: storage throughput cannot feed them, preprocessing is single-threaded, or the feature store refreshes on a schedule that no longer matches the product.
Buying more compute to solve that makes the utilisation number worse and the invoice bigger.
Before any hardware decision, measure how long your accelerators spend waiting for data. If the answer is uncomfortable, the next purchase is storage and pipeline engineering, not silicon.
Measure utilisation, not capacity
Capacity is what you own. Utilisation is what you actually got for it, and only one of those two numbers predicts your next problem.
Four metrics carry most of the signal: accelerator utilisation, memory occupancy, job queue wait time, and cost per unit of useful output such as a thousand inferences or one completed training run.
Queue wait time is the one to watch for growth planning. Rising wait times are the earliest honest signal that the fleet is undersized, and they show up weeks before anyone complains in a business review.
Cost per unit of output is the one to take into budget meetings. It converts an infrastructure argument into a margin argument, which is the only version of the conversation a finance team can act on.
Where to start this quarter
The demand side of this is not speculative. The International Energy Agency’s Energy and AI report, published in April 2025, projected that global data centre electricity consumption would roughly double by 2030 to around 945 terawatt hours, with AI named as the largest driver of that growth (IEA).
Capacity planned late will be planned into a tighter market than the one you are looking at now.
Three moves make the next twelve months easier. Write down your expected workloads, split into training and inference.
Instrument what you already run for thirty days, so the utilisation and queue numbers are real rather than remembered.
Then buy in tiers, with documentation, and with a plan for what each tier does when it is no longer your fastest.
Infrastructure that keeps up with growth is rarely the biggest deployment in the room. It is the one whose owners knew which part would break next.