Imagine a public transit bus. If the bus departs the station the moment a single passenger arrives, that passenger reaches their destination with zero delay. However, the system operates with terrible efficiency, burning fuel and driver time for one individual—resulting in low overall passenger throughput and high per-capita costs. If the bus instead waits until every single seat is filled, fuel and operational efficiency are maximized, but early-arriving passengers must sit and wait at the station for a long time, suffering severe latency.
This simple tension forms the core of the throughput-latency tradeoff in modern computing systems. From deep learning inference engines to distributed databases, systems must constantly decide whether to process tasks immediately or delay them to form a larger, more efficient batch.
The Anatomy of the Tradeoff
The tension between throughput and latency is driven by two opposing forces: the need to amortize fixed system overhead and the physical time required to accumulate requests.
Force 1: Amortizing Overhead and Saturating Hardware
Every computational task carries a fixed cost that is independent of the task's size. In job scheduling, this might be the time spent installing software or preparing a machine, as explored by Hanna Sumita et al. in Scheduling on Identical Machines with Setup Time and Unknown Execution Time. In distributed databases, it manifests as network coordination and dependency logging costs, which Chang Yao addresses in Scaling Distributed Transaction Processing and Recovery based on Dependency Logging.
In machine learning serving, the overhead is often architectural. Systems must constantly move massive model weights from high-bandwidth memory to processor cores. As noted by Eldar Kurtic et al. in "Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization, memory-bound decoding operations mean that a single request cannot fully utilize the hardware. Similarly, Mianyang Yao et al. highlight the significant "loading cost" of moving deep learning models from disk to RAM in Loading Cost-Aware Model Caching and Request Routing for Cooperative Edge Inference.
Furthermore, massively parallel processors like GPUs and TPUs require large batch sizes to saturate their compute units. As shown by Hua He et al. in Massively Parallel Suffix Array Queries and On-Demand Phrase Extraction for Statistical Machine Translation Using GPUs, processing requests individually leaves parallel hardware severely underutilized, crippling total system throughput.
Force 2: The Accumulation Penalty and Tail Latency
The primary cost of batching is time. To build a batch, a system must introduce a mandatory waiting period, forcing early-arriving requests to linger in a queue. As explained by Yinwei Dai et al. in Apparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in ML Serving, this queuing delay directly degrades the user experience.
If a system aggressively pursues maximum throughput by waiting for massive batches, it will inevitably violate strict Service-Level Objectives (SLOs) for real-time applications. Chengliang Zhang et al. demonstrate this vulnerability in MArk: Exploiting Cloud Services for Cost-Effective, SLO-Aware Machine Learning Inference Serving, showing that unmanaged batching causes severe tail-latency spikes. In production environments, Norman P. Jouppi et al. observe in In-Datacenter Performance Analysis of a Tensor Processing Unit that hardware must often be intentionally run at low utilization simply to guarantee that tail-latency deadlines are met.
Finding the Optimal Frontier
Because throughput and latency operate at cross-purposes, system engineers cannot simply pick an arbitrary batch size. Instead, they must mathematically model the environment to find a Pareto-optimal configuration.
As analyzed by Sounak Kar et al. in On the Throughput Optimization in Large-scale Batch-processing Systems, if batches are too small, the system becomes overwhelmed by processing overhead and queues back up anyway. If batches are too large, the server sits idle waiting for jobs to arrive, wasting valuable execution cycles.
Latency
^
| x (Batch size too small: overwhelmed by overhead)
| /
| / <-- The Pareto Frontier
| /
| x (Optimal balance)
| \
| \
| x (Batch size too large: excessive queuing delay)
+-------------------------------------------------------> Throughput
To dynamically navigate this curve, modern serving layers use adaptive batching algorithms. For example, the framework designed by Jean-François David et al. in StarONNX : Un ordonanceur dynamique pour une inférence rapide et à haut débit sur des ressources hétérogènes uses mathematical optimization to select batch sizes on the fly. By monitoring the arrival rate of incoming requests, the system calculates the exact batch size needed to maximize hardware efficiency without letting individual requests exceed their latency budgets.
Shifting the Frontier: Advanced Mitigations
Rather than merely choosing a compromise point on a static throughput-latency curve, recent research focuses on architectural innovations that shift the entire frontier outward—achieving higher throughput and lower latency simultaneously.
Multi-Bin Batching
A major flaw in standard batching is intra-batch variance, often called the "straggler problem." If a short request is grouped into a batch with an unexpectedly long request, the short request is held hostage until the entire batch finishes.
To solve this, Ozgur Guldogan et al. developed Multi-Bin Batching for Increasing LLM Inference Throughput. Their system predicts the execution length of incoming requests and routes them into specialized "bins" or queues of similar lengths. This prevents short tasks from being delayed by long ones. However, the authors note a secondary trade-off: increasing the number of specialized bins reduces variance but increases the time required to accumulate enough similar requests to fill a batch.
Execution Restructuring (Chunking and Piggybacking)
In Large Language Model (LLM) serving, the throughput-latency tension is particularly acute. Processing the initial user prompt (the prefill phase) is highly compute-bound and efficient, while generating subsequent tokens (the decode phase) is memory-bound and inefficient. Standard systems stall token generation for existing users whenever a new user's heavy prefill request arrives.
In Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve, Amey Agrawal et al. break this bottleneck by "chunking" long prefill requests into smaller blocks. The system then "piggybacks" these compute-heavy chunks onto the memory-bound decode iterations of other users. By blending these two distinct workloads into a single execution cycle, the architecture saturates the GPU's compute units without causing multi-second generation stalls for active users.
Early Exits
Another way to bypass the batching penalty is to dynamically alter the amount of computation a request requires based on its complexity.
In Apparate: Rethinking Early Exits to Tame Latency-Throughput Tensions in ML Serving, Yinwei Dai et al. introduce "early exits" into deep learning models. When a batch of requests passes through a neural network, simpler inputs are allowed to exit the model at earlier layers once a confidence threshold is met. This decouples individual request latency from the total depth of the model, allowing the system to maintain the high-throughput benefits of large batches while dramatically lowering the median latency for simpler queries.