In modern machine learning production, there is a persistent tension between two goals: making a single prediction as fast as possible (latency) and processing as many predictions as possible per second (throughput).
This conflict arises because of a fundamental mismatch between how we want to consume AI—one interactive request at a time—and how the hardware is designed to process it—in massive, parallel chunks.
The Hardware Mismatch
High-performance accelerators like GPUs and TPUs are built for "wide" parallel arithmetic. As Noah Fiedel and colleagues noted in the design of TensorFlow-Serving, these chips require large batches of data to saturate their compute units and hide memory access costs. If you send a single image to a GPU, most of the chip sits idle while the data moves through the circuits.
To fix this, systems use batching: they wait for multiple requests to arrive and group them into a single matrix operation. While this dramatically increases throughput, it introduces a "formation delay." The first request in a batch must sit idle in a queue while the system waits for the last request to arrive. As Daniel Crankshaw explains in InferLine, finding the "sweet spot" for batch size is difficult because every millisecond spent waiting for a larger batch is a millisecond added to the user's response time.
The Cost of the Waiting Room
The relationship between batching and performance is non-linear. Ziyang Zhang and the team behind BCEdge point out that while throughput generally scales with batch size, latency eventually hits a wall where the queuing delay outweighs any computational gain.
Mianyang Yao and others have modeled this as a mathematical trade-off: the total time a request spends in the system is the sum of the waiting time (to form the batch) and the execution time (to run the model). In real-time environments, such as self-driving cars or robotics, Andrew Lavin argues that we must often abandon large batches entirely, necessitating specialized algorithms like Winograd convolutions that remain efficient even at a batch size of one.
The Ragged Tensor Problem in LLMs
Large Language Models (LLMs) add a new layer of complexity: output variability. Unlike an image classifier that always takes the same amount of time to process a picture, an LLM generates tokens one by one. If you batch a request for a 10-word poem with a request for a 500-word essay, the poem request is "held hostage" by the essay, staying in the GPU's memory until the entire batch finishes.
To solve this, Ozgur Guldogan proposed Multi-Bin Batching, which groups requests by their predicted output lengths. However, even this introduces a trade-off: more bins mean more specialized queues, which can increase the time a request spends waiting for a "compatible" partner to arrive. Ranran Haoran Zhang further refined this with EQSPEC, a framework that mathematically verifies how to group these "ragged" sequences without corrupting the model's output.
Navigating the Pareto Frontier
Because you cannot maximize throughput and minimize latency simultaneously, researchers treat this as a multi-objective optimization problem. The goal is to find the Pareto Frontier—the set of configurations where you cannot improve one metric without hurting the other.
Systems like SCOOT, developed by Ke Cheng and colleagues, use Bayesian Optimization to automatically tune parameters like batch size and scheduling factors. By learning the "hidden" constraints of the hardware, these systems ensure that Service-Level Objectives (SLOs)—such as "95% of requests must finish in under 200ms"—are met while still processing the maximum possible volume of queries.
Breaking the Tradeoff
Recent work has attempted to "break" this trade-off rather than just manage it.
- Speculative Decoding: Ranajoy Sadhukhan and the MagicDec team use a smaller "draft" model to predict tokens, which the larger model then verifies in a single batch. This allows for low-latency generation even at high throughput.
- Early Exits: Yinwei Dai introduced Apparate, which allows "easy" requests to leave the model early, freeing up resources for more complex tasks without waiting for the entire batch to complete all layers.
- Model Replication: For smaller models, Pol G. Recasens suggests that replicating the model across multiple instances on a single GPU can be more effective than simply increasing batch size, as it avoids the formation delay while still soaking up idle compute cycles.