Advisory · AI infrastructure strategy
The AI factory blueprint: match the silicon to the strategy
Anyone can read a spec sheet. The harder question is which NVIDIA system your strategy actually needs, and at which numeric precision.
Jensen Huang stood on the CES 2026 stage, held up a Vera Rubin board, and told the room it was already in full production. The promise attached to it was the kind that makes finance directors reach for the chequebook: up to five times the inference performance of Blackwell and roughly a tenth of the cost per token. Somewhere in every audience for that pitch sits a person whose real job is to ask a quieter question. Not "how fast is it," but "fast at what, for whom, and paid back over how many years."
This piece is written for that person. It is a blueprint, not a hype reel. The tools have never been more powerful, and the temptation to buy the biggest box in the catalogue has never been stronger, which is exactly why the discipline matters. The right platform is not the fastest one. It is the one your operating model actually needs. So we start where a good advisor always starts, with the work, and only then do we reach for the silicon.
01/ Strategy before silicon
The most expensive mistake is buying capability you cannot feed
There is a pattern that plays out in procurement meetings all over the world right now. A vendor deck lands, the headline multiplier is enormous, and the conversation jumps straight to which rack to order. The operating model, the actual shape of the work the organisation does, never enters the room. That is the moment a good advisor slows everything down, because the fastest way to waste a seven figure budget is to buy a machine your workload can never keep busy.
The discipline is old and it is boring and it works. Look honestly at your current operating model: what you run today, how often, at what latency, for how many users, against models of what size. Then define the target operating model: where you intend to be in eighteen to thirty six months, not where a keynote says the industry is going. Only when those two pictures are clear do you choose technology, because now you are matching a system to a purpose instead of buying a purpose to justify a system. Strategy first, silicon second, in that order every time.
None of this is an argument against ambition or against NVIDIA. The point is the opposite. When you buy against a real target operating model, you tend to buy more confidently and defend the spend more easily, because every line in the bill of materials traces back to a use case and a payback. The waste is not in spending money on GPUs. The waste is in spending it on the wrong axis, on raw compute when your bottleneck was latency, or on a rack of coherent memory when your models fit comfortably on a single card.
There is a second reason to start from the work rather than the chip, and it is the quiet thing NVIDIA sells that never shows up as a number on the board. When Jensen Huang bet the company on CUDA in the late 2000s, long before there was an obvious market for it, he was not selling silicon. He was building a software moat that today means your engineers, your frameworks, and your operators already speak the platform's language. That lock-in is real and it cuts both ways, so factor it into the strategy deliberately rather than discovering it after the purchase order clears.
02/ The three questions that actually decide it
Latency, raw compute, or memory bandwidth. Pick your bottleneck
Almost every infrastructure decision collapses into one honest question: what is the thing you are actually short of? Not what sounds impressive, but what limits the work. In practice there are three answers, and a fourth that quietly gates all of them. Naming yours correctly is most of the job.
Ultra-low latency is the need when a human or an agent is waiting on every token, and the time to first token and the gap between tokens decide whether the product feels alive. Raw compute, the pure throughput of matrix math, is the need when you are training, fine tuning, or pushing enormous batches through inference and you are paid by total work done per hour. Memory bandwidth is the need that hides in plain sight: for many inference and scientific workloads the GPU is starved not for arithmetic but for the speed at which it can move weights and activations in and out of memory, and that is where the generational leaps quietly land.
The fourth dimension is working-set reach, and it decides the others. A model and its context that fit inside one GPU's memory are a different engineering problem from one that must be sharded across eight GPUs, which is different again from one that wants a coherent domain of seventy two. Reach is what turns a chip decision into a rack decision. Read the diagram below as a set of dials rather than a scoreboard: your job is to know which dial your work is turning hardest.
03/ Precision is a strategy decision, not a footnote
The same GPU is six different machines, depending on the number format
Here is the part that most buying conversations skip, and it is the part that can flip the whole decision. A modern NVIDIA GPU does not have one performance figure. It has a ladder of them, one per numeric precision, and the rungs are separated by more than a thousand times. Choosing the precision your workload runs at is not a tuning detail you hand to the engineers afterwards. It is part of defining the workload itself, and it can change which platform you should buy.
The logic is simple once you see it. Fewer bits per number means more numbers moved and multiplied per second, and a smaller memory footprint, at the cost of numerical range and accuracy. So very low precision such as NVFP4 is the language of high-throughput inference, where a well-quantised model gives you the most tokens per second per dollar. FP8 and FP6 are the modern mainstream for training and heavy inference. FP16 and BF16 remain the numerically safe choice for training that has to converge cleanly. And FP32 and FP64 are where scientific computing, simulation, and anything that cannot tolerate rounding error still live. A platform that is breathtaking at FP4 inference can be the wrong tool for FP64 double-precision HPC, and no amount of headline PFLOPS changes that.
The Rubin GPU makes the ladder unusually stark, because NVIDIA publishes the whole thing. Read the rungs below on a logarithmic scale, because a linear one would flatten everything below FP8 into an invisible sliver. That visual crush is the point: the machine you are buying is enormous for low-precision AI and comparatively modest for double precision, so your precision has to match your purpose before the numbers mean anything.
| Precision | Rubin peak (per GPU) | What it is really for | Watch-out |
|---|---|---|---|
| NVFP4 (FP4) | 50 PF inference 35 PF training | Highest-throughput inference; frontier low-precision training | Inference figure is sparse; needs careful quantisation |
| FP8 / FP6 | 17.5 PF training | Mainstream training and high-throughput inference | The current sweet spot; validate accuracy per model |
| FP16 / BF16 | 4 PF | Numerically safe training and fine-tuning | Four times below FP8; use where convergence needs it |
| TF32 | 2 PF | Drop-in tensor math for FP32-style training | Convenience precision, not full FP32 range |
| FP32 | 130 TF | General compute, graphics, non-tensor paths | Orders of magnitude below the tensor formats |
| FP64 | 33 TF | Double-precision scientific and HPC work | If this is your need, raw AI PFLOPS mislead you |
04/ The three factories you can buy today
One is a generalist, one is a scale-up island, one is a rack that thinks as a whole
NVIDIA's current enterprise reference architectures give you three genuinely different machines, and the difference that matters most is not on the spec sheet's first line. It is how the GPUs talk to each other. Get that one idea straight and most of the decision resolves itself, because the interconnect is what decides which models you can run without fighting your own hardware.
The RTX PRO 6000 AI factory is the flexible generalist. Its GPUs each hold 96 GB of GDDR7 and connect through PCIe, with no NVSwitch fabric binding them together, so eight cards in a chassis are eight cooperating islands rather than one pooled brain. That sounds like a limitation and sometimes it is, but it is also why this platform fits a normal enterprise rack, sips power by comparison, runs graphics and rendering and virtual workstations, and covers the widest spread of workloads of the three. The HGX B300 AI factory raises the ceiling by wiring eight Blackwell Ultra GPUs into a single NVLink domain through NVSwitch, so those eight cards, with 288 GB of fast HBM3e each, behave like one tightly coupled accelerator for training and large-model inference. The GB300 NVL72 takes the same idea and stretches it across an entire rack, so all seventy two GPUs sit inside one NVLink domain with coherent Grace CPU memory, at the cost of mandatory liquid cooling and up to 142 kilowatts that your building has to supply.
| Decision point | RTX PRO 6000 | HGX B300 | GB300 NVL72 |
|---|---|---|---|
| GPU class | RTX PRO 6000 Blackwell, PCIe | B300 Blackwell Ultra, SXM | B300 in Grace Blackwell Ultra |
| Memory per GPU | 96 GB GDDR7 | 288 GB HBM3e | 288 GB HBM3e |
| Bandwidth per GPU | 1.6 TB/s | 8 TB/s | 8 TB/s |
| Scale-up fabric | PCIe Gen5, no NVSwitch | NVLink 5, 8-GPU domain | NVLink 5, 72-GPU domain |
| Coherent CPU-GPU memory | No | No (x86 host) | Yes, Grace NVLink-C2C |
| Max GPU power (TGP) | up to 600 W | up to 1,400 W | up to 1,400 W |
| Cooling | Air or liquid | Air or liquid, dense | Direct liquid, mandatory |
| Power envelope | OEM-defined, modest | ~14.5 kW / system | up to 142 kW / rack |
| Server customisation | High (2/4/8 GPU, CPU, RAM) | Medium-high | Low inside the rack |
| MIG partitions per GPU | up to 4 | up to 7 | up to 7 |
| Best when | Mixed enterprise, graphics, small and medium training | Serious 8-GPU training and inference | Trillion-parameter, MoE, reasoning at rack scale |
What "unified memory" actually means, and why it can cost you
The most expensive misunderstanding in this whole category is the word "unified." A datasheet that says a rack has 20 terabytes of GPU memory is telling the truth about capacity and quietly misleading you about architecture. That 20 terabytes is seventy two separate pools of 288 GB, each one physically welded to its own GPU. NVLink lets a GPU reach into a neighbour's memory quickly, and on NVL72 the Grace CPU and its GPUs share a coherent address space, but coherent addressability is not the same thing as one flat pool where every byte is equally fast.
This matters because it changes how a model has to be written and placed. On the RTX PRO platform, an eight-GPU node holds 768 GB of VRAM in total, yet a single un-sharded process cannot treat it as one 768 GB device: the model must be partitioned across the cards deliberately. NVL72 gives you a far larger and genuinely coherent domain, which is exactly why it exists, but even there local HBM is fastest, the Grace LPDDR5X is a separate slower tier, and remote GPU memory travels across NVLink. Buy the coherent rack when your working set truly needs it. Do not pay for it to solve a problem that careful sharding on a cheaper platform would have solved for a fraction of the cost.
05/ The fit finder
Stop reading the table. Drag your requirements instead
A table tells you what each platform is. It does not tell you what you need. So here is the same logic turned into a tool you can steer. Set your objective, pick the precision your models really run at, then drag the working set, the latency target, the throughput you are chasing, and, crucially, how ready your building actually is. The recommendation updates as you move, and it shows its reasoning, because a recommendation you cannot interrogate is worth nothing.
Watch how the answer moves. Push latency to real-time and the whole picture tilts toward the deterministic decode engines. Switch precision from FP4 to FP32 and platforms reshuffle, because a machine tuned for low-precision inference is not the machine you want for double-precision science. Turn facility readiness down to a standard enterprise room and the biggest racks quietly disqualify themselves, no matter how good their numbers look. That last move is the one most decks never show you.
Fit finder
Drag your requirements. The recommendation updates live.
06/ Reading the roadmap without buying the hype
Vera Rubin is not one faster chip. It is a system that stopped pretending one chip should do everything
When NVIDIA unveiled Vera Rubin, the easy headline was the raw Rubin GPU number. The more interesting story, and the one that maps directly onto everything above, is that the platform is deliberately seven specialised chips instead of one heroic accelerator. That is the operating-model-first idea rendered in silicon: stop forcing a single processor to do request handling, training, decode, networking, storage, and security, and instead give each job the chip that removes its specific bottleneck.
The cast divides cleanly. Three chips execute work: the Vera CPU, an 88-core Arm processor with up to 1.5 TB of memory for orchestration, agents, and KV-cache management; the Rubin GPU, with 288 GB of HBM4 and the broad training and inference role; and the Groq 3 LPU, an SRAM-heavy accelerator built for one thing, deterministic low-latency decode. The other four move, secure, and coordinate data: NVLink 6 for GPU scale-up, ConnectX-9 and Spectrum-6 for scale-out, and BlueField-4 for infrastructure, storage, and in-silicon security. Nobody is asking you to buy all seven in a box. You compose the ones your target operating model needs.
The three engines that actually compute, and why they coexist
Look closely at the three execution chips and you see the precision and bandwidth arguments from earlier turned into hardware. The numbers that matter are not just capacity and speed, but the ratio between them. A Groq LPU carries only 500 MB of memory, yet moves it at 150 TB/s, which is a velocity tier, not a capacity tier. A Rubin GPU balances 288 GB against 22 TB/s. A Vera CPU trades speed for reach, holding 1.5 TB at a comparatively gentle 1.2 TB/s. Divide bandwidth by capacity and the three chips reveal their true jobs.
| Engine | Local memory | Bandwidth | Bandwidth per GB | Best stage of the job |
|---|---|---|---|---|
| Vera CPU | up to 1.5 TB LPDDR5X | 1.2 TB/s | ~0.8 GB/s | Orchestration, agents, tools, ETL, KV-cache |
| Rubin GPU | 288 GB HBM4 | 22 TB/s | ~76 GB/s | Prefill, attention, training, flexible inference |
| Groq 3 LPU | 500 MB SRAM | 150 TB/s | ~300,000 GB/s | Latency-sensitive decode, FFN and MoE experts |
This is why the Rubin GPU and the Groq LPU ship as partners rather than rivals. In a modern reasoning workload, the prefill and attention phases want the GPU's huge, fast HBM, while the token-by-token decode phase wants the LPU's tiny, blindingly fast SRAM and its deterministic schedule. NVIDIA's Dynamo software routes each stage to the engine that suits it, so a single request can be served by three different kinds of chip without the user ever knowing. For a buyer, the lesson is not "order a Groq rack." It is that the future of inference is heterogeneous, and your target operating model should be written in terms of stages and their needs, not a single box that does it all.
None of this changes the discipline. If anything it sharpens it, because a disaggregated platform rewards buyers who know their workload in detail and punishes those who do not. The organisation that mapped its stages can compose exactly the right mix of Vera, Rubin, and Groq. The organisation that just wanted "the new NVIDIA thing" will overbuy on one axis and starve another. For the deeper architecture, our companion piece on the Vera Rubin NVL72 rack walks the physical build, and our essay on why NVIDIA is not really a GPU company anymore explains the software and systems moat that makes this whole platform sticky.
07/ The blueprint, step by step
A repeatable way to buy NVIDIA infrastructure without buying regret
Everything so far becomes a sequence you can run for any workload, any brand, any budget. It is deliberately unglamorous, because the glamour is what gets organisations into trouble. Follow it in order and each line of your eventual bill of materials will trace back to a use case and a payback, which is the only defence that survives a budget review.
- Map the current operating model. Write down what you actually run today: the workloads, their model sizes, their latency commitments, their utilisation, and the precision they use right now. You cannot improve a picture you have not drawn, and most organisations have never drawn this one honestly.
- Define the target operating model. Decide where you need to be in eighteen to thirty six months, expressed as use cases that create value, not as hardware you admire. This is the document the whole purchase serves, and it is worth more than any vendor deck.
- Name the binding constraint, and the precision. For each workload, say plainly whether it is limited by latency, raw compute, memory bandwidth, or working-set reach, and at which numeric format it runs. This single sentence per workload does more to size the estate than any benchmark.
- Shortlist platforms by fit, not by prestige. Match each workload to the smallest system that satisfies its constraint, using the logic in the fit finder. Right-size the scale-up domain: do not buy a coherent rack to run a model that shards cleanly across eight cards.
- Clear the facility and data-path gate early. Confirm the power, the cooling, the liquid loop, the rack density, the storage throughput, and the fabric before you sign anything. The accelerators are only as fast as the data pipeline that feeds them, and a rack you cannot power is a very expensive radiator.
- Deploy at a scalable unit, then grow on evidence. Start at the documented scalable unit, instrument utilisation and payback, and let real numbers, not roadmap envy, pull you to the next size. Keep the Vera Rubin roadmap as an option you exercise when the operating model asks for it, never as an obligation the calendar imposes.
08/ The verdict
Buy the operating model, not the GPU
The silicon in this piece is genuinely extraordinary. A single Rubin GPU quotes fifty petaflops of low-precision inference, a GB300 NVL72 turns seventy two accelerators into one coherent domain, and a Groq LPU moves half a gigabyte of memory at a hundred and fifty terabytes a second. It would be easy, and expensive, to fall in love with any of it.
The advisor's job is to love the outcome instead. The organisations that win the next few years will not be the ones that bought the biggest number. They will be the ones that knew their work well enough to buy precisely, that named their constraint before they named a product, that treated FP4 and FP64 as different strategies rather than different rows, and that walked their building's power plan before they walked a sales floor. NVIDIA has built a catalogue deep enough to serve almost any operating model. Your advantage is not access to it. Your advantage is the discipline to take only the part your strategy actually needs, and to make every watt and every dollar earn its place.
09/ Questions worth asking
The questions a good architect will ask you back
It turns on latency and working-set size. For flexible enterprise inference and mixed workloads, an RTX PRO 6000 AI factory is often enough. For large-model, low-latency inference at scale, a GB300 NVL72 gives you a single 72-GPU NVLink domain. For ultra-low-latency decode in the Vera Rubin era, a Groq 3 LPX rack pairs with Rubin GPUs so each stage of the request runs on the engine that suits it.
HGX B300 is an eight-GPU server that creates an eight-GPU NVLink scale-up domain, with wide OEM choice of CPU, memory, and cooling. GB300 NVL72 is a standardised, liquid-cooled rack that links all seventy two GPUs into one NVLink domain with coherent Grace CPU memory, at up to 142 kilowatts per rack. HGX fits a conventional server model, NVL72 is a rack-scale appliance that your facility has to be built for.
Precision sets both throughput and accuracy. Low precision such as NVFP4 gives the highest inference throughput and the smallest memory footprint, FP16 and BF16 are common for training that must converge cleanly, and FP32 and FP64 matter for scientific and high-precision work. A platform that is excellent at FP4 inference can be the wrong choice for FP64 HPC, so the number format belongs in the workload definition, not in a tuning session afterwards.
No. NVL72 provides coherent addressability between each Grace CPU and its paired Blackwell GPUs, plus a rack-wide NVLink fabric for fast peer access across seventy two GPUs. But HBM stays physically attached to each GPU and LPDDR5X stays attached to each CPU. Unified addressability is not one uniform physical pool, so locality and topology still shape performance, and your software still has to place data with intent.