Modern hardware accelerators like GPUs and TPUs are designed for massive parallelism, yet they often sit idle during real-time inference. A single request—such as generating one token in a Large Language Model (LLM) or processing one audio frame—is computationally "thin." It doesn't provide enough work to saturate the thousands of cores on a modern chip.
To bridge this gap, researchers have developed a suite of batching techniques that aggregate independent requests into unified execution blocks. The goal is to transform sequential, latency-sensitive tasks into high-throughput operations by amortizing the high cost of loading model weights across multiple data points.
The Memory Wall and the Case for Batching
The fundamental bottleneck in deep learning inference is often not the raw math (FLOPs), but the speed at which model weights can be moved from memory to the processor. In a single-request scenario, the GPU spends more time "waiting" for weights to arrive than it does performing calculations.
As Noah Fiedel and colleagues noted in the development of TensorFlow-Serving, inter-request batching is the primary mechanism for maximizing cost-effectiveness. By grouping multiple individual inference requests into a single batch, the system loads the model weights once and applies them to many inputs simultaneously. This significantly improves hardware utilization, as the "cost" of the weight loading is shared.
This principle is critical across domains. In speech recognition, Dario Amodei and the Deep Speech 2 team demonstrated that batching concurrent requests allows massive models to serve multiple users in real-time without exceeding tail-latency limits.
From Static Blocks to Continuous Streams
Traditional batching is "static": the system waits for $N$ requests to arrive, processes them together, and returns them all at once. However, this creates a "waiting for the slowest" problem. If one sequence in a batch is much longer than the others, the hardware remains underutilized while the shorter sequences sit idle.
To solve this, systems like vLLM and Baton introduced Continuous Batching. As described by Peizhuang Cong in the BATON framework, inference is treated like a relay race rather than a fixed-distance sprint. Instead of waiting for an entire batch to finish, the system uses "dynamic re-batching" to slot a new request into a processing lane the moment an old one completes. This eliminates idle hardware cycles and can boost throughput by up to 1.75x over static methods.
Continuous Depth-wise Batching in Recursive Models
A more recent innovation involves rethinking the model architecture itself to facilitate better batching. Standard Transformers consist of a stack of distinct layers. In contrast, Recursive Transformers reuse the same physical parameters across multiple iterations.
Sangmin Bae and Adam Fisch introduced Relaxed Recursive Transformers, which enable a paradigm called Continuous Depth-wise Batching. Because the weights are identical across every "loop" of the model, a token that is on its first pass can be batched with a token on its tenth pass.
In standard models, tokens must be at the same layer to be batched. In recursive models, the "depth" becomes a fluid dimension. This allows tokens to exit the loop early if they are "easy" to process, while "harder" tokens continue, with new requests filling the vacated slots immediately. This approach, also explored in Mixture-of-Recursions, effectively eliminates the synchronization bottlenecks that plague traditional adaptive inference.
Centralized Inference for Distributed Systems
The need for efficient batching extends beyond user-facing APIs into the training of Reinforcement Learning (RL) agents. In distributed RL, thousands of "actors" interact with environments to collect data. If each actor performs its own inference locally, the GPU utilization is abysmal.
The SEED RL architecture, developed by Lasse Espeholt and Ke Wang, solves this by moving inference to a centralized server. A dedicated batching layer aggregates asynchronous requests from multiple actors into large minibatches. This design, also utilized in the TLeague framework, allows RL systems to achieve millions of frames per second by aligning the irregular timing of environment steps with the rigid, high-throughput requirements of modern accelerators.
Packing Heterogeneous Workloads
Finally, batching can occur even across different models. Paras Jain and colleagues proposed Dynamic Space-Time Scheduling, which merges kernels from entirely different model instances into a single "super-kernel."
By grouping similar operations (like matrix multiplications) across different user requests—even if they belong to different models—the system can achieve the efficiency of large-batch processing in low-latency environments. Similarly, Graft by Fangming Liu uses "DNN re-alignment" to allow mismatched model fragments to share a common "tail," enabling batching across diverse entry points in edge-cloud environments.