Training and inference are pushing AI infrastructure design in different directions. Large-scale training typically concentrates resources on a coordinated computational task. Inference must sustain a continuous flow of requests with different context lengths, processing requirements, and response-time expectations.
AI agents deepen that divergence. A single task can involve repeated model calls, retrieval, tool use, and code execution, with each step producing information that informs the next. Infrastructure must support the entire sequence, including the CPU work, context retention, and data movement between inference calls.
As applications become more interactive and incorporate voice, images, and video, these demands will become more pronounced. Generalist GPU clusters are giving way to specialized architectures and inference is leading the way.
Radiant has been working with some of the biggest inference providers in the world. These engagements reinforce a practical lesson: inference performance depends on how the entire cluster is configured. Bare-metal servers, CPU capacity, GPU allocation, storage, and networking must work together to deliver responsive applications, accommodate changing demand, and keep serving costs sustainable.
Why Inference Demands a Different Cluster Design
Inference makes infrastructure performance visible to users. A coding assistant needs to respond while a developer is working. A voice application must sustain a natural exchange. An AI agent may need to retrieve information, execute code, inspect the result, and return to the model several times before completing a task.
These agentic workflows change what responsiveness means. Fast token generation is one contributor to task completion time. Delays in tool execution, data access, or subsequent model calls can accumulate across the sequence. Multiple agents working concurrently can also create bursts of demand across both GPU and CPU resources.
Each activity places different demands on the underlying hardware. Fast initial responses require sufficient prompt-processing capacity. Smooth generation depends on sustained access to compute and memory. Multimodal applications introduce additional input processing, while agents require CPU capacity to run tools and execution environments alongside model serving.
Context length compounds these demands. Long documents, code repositories, conversation histories, and accumulated tool results can increase both processing requirements and the memory occupied by the key-value, or KV cache. Supporting a model’s maximum context window is therefore different from sustaining many concurrent, multi-step tasks at an acceptable speed.
The main performance measures translate into concrete hardware requirements:
These requirements shape both infrastructure architecture and economics. They influence how clusters are configured, how capacity scales, and what it costs to deliver a consistent user experience. As inference demand grows, matching infrastructure to the workload becomes increasingly important to both performance and profitability.
Six infrastructure characteristics are particularly important.
1. Balanced CPU–GPU Capacity for Agentic and Multimodal Inference
AI agents make CPU performance part of the inference experience.
Between model calls, an agent may execute generated code, query a database, process retrieved information, or validate a result. These operations often run on CPUs, sometimes within isolated execution environments. Slow execution delays the next inference call and extends the time required to complete the task.
As more agents operate concurrently, the infrastructure must support both model serving and the growing number of environments running alongside it. This creates demand for CPU cores, host memory, and predictable access to data, with capacity that can grow according to the application’s balance of reasoning and execution.
Multimodal workloads add further pressure. Audio may require decoding, resampling, speech detection, and feature preparation before model execution. Images and video introduce their own preparation and transfer requirements.
The NVIDIA Vera CPU addresses both roles: supporting accelerated systems as a host CPU and running agentic workloads in dedicated CPU infrastructure. Dedicated Vera CPU systems provide capacity for tool calls, sandbox environments, code execution, and data processing.
A balanced cluster therefore needs sufficient host resources to sustain GPU execution and sufficient application compute to keep agent workflows progressing. The appropriate configuration depends on how much time each workload spends generating tokens, preparing inputs, and executing actions between model calls.
2. Co-Designed Compute for Disaggregated Inference
Language-model inference contains two stages with different resource profiles. Prefill processes the input; decode generates subsequent tokens.
Prefill benefits from substantial parallel compute, while interactive decode requires fast, predictable execution through a repeated token-generation loop. Disaggregating these stages allows their resources to be allocated and scaled according to the workload’s balance of input processing and output generation.
The NVIDIA Vera Rubin platform with Groq 3 LPX extends this principle through co-designed GPU and LPU execution. Rubin GPUs handle prefill and decode attention over the accumulated KV cache. Groq 3 LPX accelerates latency-sensitive feed-forward and mixture-of-experts computations within decode, pairing the GPUs’ compute and memory capacity with the LPUs’ low-latency execution.
At the infrastructure level, realizing the benefits requires coordinated compute placement between GPUs and inference accelerators, sufficient interconnect bandwidth, and predictable transfer latency. Small communication delays can accumulate across generation, making the connections between systems integral to performance.
3. Storage Designed for Model Distribution and Context Reuse
Inference storage supports two distinct performance requirements: bringing models into service and keeping useful context accessible.
Model startup involves moving weights and runtime artifacts into the serving environment. When multiple replicas start together, their combined reads can place substantial pressure on shared storage and the network. Local NVMe can retain frequently used artifacts, while the shared storage tier must support the aggregate demand from deployments and expansion.
During serving, supported runtimes can reuse cached KV data for matching prompt prefixes. Repeated instructions, shared documents, and continuing conversations can then avoid some input processing. The infrastructure must provide an appropriate location for that state.
A practical hierarchy spans GPU memory, host DRAM, local NVMe, and shared storage. Each tier offers a different balance of capacity, access time, and cost. Keeping context in a larger, slower tier is useful only when retrieving it remains worthwhile compared with recomputation.
NVIDIA CMX context memory storage, adds a shared, Ethernet-attached flash tier specifically for reusable KV cache. It retains context beyond individual servers and supports staging that data back into host or GPU memory before it is needed. This extends the context hierarchy while preserving GPU memory for active execution.
Effective context storage allows more reusable state to remain accessible as demand grows. Its capacity, access latency, aggregate bandwidth, and placement relative to compute must be evaluated together, including how it performs while new models are loading and existing replicas are serving requests.
4. Networking Matched to East–West and North–South Traffic
Inference workloads differ substantially in how much communication they require between GPUs and servers.
A service built from independent model replicas can distribute requests among those replicas without the continuous inter-server synchronization associated with large distributed training jobs. For suitable deployments, this creates an opportunity to reduce a separate inter-server GPU backend fabric.
Other inference configurations depend heavily on internal communication. Models distributed across servers, mixture-of-experts deployments, and disaggregated prefill and decode can require substantial east–west bandwidth. Their performance depends on the topology and speed of the connections between participating devices.
The cluster must therefore distinguish between three requirements: GPU-to-GPU communication, access to storage and supporting services, and north–south traffic carrying requests and responses.
Reducing a dedicated GPU fabric does not eliminate internal networking. Model distribution, shared context storage, and application dependencies still generate traffic. Those flows must be sized alongside live serving demand.
NIC capacity, switch topology, and uplink bandwidth should support these flows under concurrent load. Where traffic shares infrastructure, appropriate bandwidth protection helps prevent model loading or cache transfers from disrupting active requests. The resulting network should deliver predictable serving performance at a cost proportionate to the workload.
5. Fine-Grained GPU Allocation
Inference fleets often serve smaller models alongside large language models. Speech recognition, speech synthesis, classification, and embedding services may need only part of a high-end GPU’s resources.
Allocating a whole GPU to each small deployment can leave capacity unused. Increasing batch sizes is not always an acceptable remedy because interactive applications cannot wait indefinitely for more requests.
Where supported, hardware-backed partitioning such as NVIDIA Multi-Instance GPU, allows an accelerator’s compute and memory resources to be divided into smaller allocations. Speech recognition and synthesis are examples where fractional GPU capacity can be effective.
Each allocation still needs sufficient host CPU, memory, and I/O resources. Its size must accommodate model weights, working memory, and expected concurrency, while shared resources outside the GPU must support the combined demand.
Fine-grained allocation makes it possible to expand smaller services in increments that more closely follow their traffic. This improves the economics of serving a diverse model portfolio while preserving whole GPUs and connected multi-GPU groups for workloads that need them.
6. Rapid Replica Readiness
An available GPU becomes useful inference capacity only after the model and runtime are ready to serve requests.
That preparation depends partly on the hardware surrounding the accelerator. Model weights must be read, staged, and transferred into GPU memory. Runtime initialization and warm-up may add further work. Starting many replicas simultaneously multiplies the demand on these paths.
Rapid readiness therefore requires coordinated provisioning across local NVMe, shared storage, host DRAM, CPU capacity, and network interfaces. Frequently used weights and compatible execution artifacts can be cached close to compute, reducing repeated transfers from remote storage.
The important hardware test is concurrent startup. A storage system that loads one model quickly may become a bottleneck when an entire group of servers needs the same files.
Faster readiness shortens the delay between allocating capacity and serving traffic. It helps absorb demand increases, replace failed replicas, and introduce model updates, while reducing the amount of warm spare capacity needed to cover long startup times.
Build Your Inference Cluster with Radiant
Inference architecture increasingly depends on choices made across the physical cluster: how GPUs are grouped, how much CPU capacity supports them, where models and context reside, and which communication paths require the greatest bandwidth.
Radiant brings compute, storage, networking, software and powered land into a coordinated infrastructure program, with facility and cluster designs developed together for inference deployments. That creates a foundation for serving demanding models efficiently, sustaining responsiveness as traffic grows, and introducing new workloads as applications evolve.
