In any shared resource system—whether it is a wireless network, a cloud datacenter, or an AI serving cluster—operators face a fundamental tension between maximizing throughput (the volume of work processed per unit of time) and minimizing delay (the time it takes for an individual task to complete).
To understand why this conflict is inevitable, we must look at the intuition of the backlog.
The Intuition of the Backlog
To achieve maximum throughput, a processing unit (such as a CPU, a router, or a wireless channel) must never sit idle. If a resource becomes available even for a millisecond with no data ready to process, that operational capacity is lost forever. To guard against stochastic fluctuations—such as sudden bursts of incoming traffic or temporary drops in wireless channel quality—systems maintain a buffer, or a queue.
However, by Little’s Law, the average time a packet spends waiting in a system is directly proportional to the average length of the queue. Maximizing resource utilization requires keeping these queues deep enough to absorb any dips in supply or spikes in demand. But deep queues inherently mean that every arriving packet must wait longer behind the data preceding it.
As noted by Kim et al. (2025) in the context of Large Language Model (LLM) serving, this creates a structural misalignment: service providers naturally prioritize high throughput to maximize expensive hardware utilization, while users require low latency for real-time interaction.
The Mathematical Engine: Drift-Plus-Penalty
To systematically navigate this frontier, researchers rely on stochastic optimization frameworks, most notably Lyapunov optimization and the drift-plus-penalty technique.
Consider a network scheduler trying to maximize an objective utility function $U(\gamma)$ (such as total data rate or fairness) while keeping the underlying queues stable. As formalized by Huang et al. (2010) through the Perturbed Max-Weight algorithm, the scheduler does not optimize for throughput in isolation. Instead, it minimizes a joint metric:
$$\text{Minimize: } \Delta(L(t)) - V \cdot U(\gamma(t))$$
Where $\Delta(L(t))$ represents the Lyapunov drift (a mathematical push to shrink queue sizes and maintain stability), $U(\gamma(t))$ is the utility penalty, and $V$ is a non-negative control parameter.
The parameter $V$ acts as a physical "knob" for system designers:
- When $V$ is large: The system heavily prioritizes maximizing utility (throughput). The algorithm pushes the system within an $O(1/V)$ gap of the theoretical maximum throughput. However, the trade-off is severe: the average queue backlog grows linearly as $O(V)$, causing a corresponding surge in packet latency.
- When $V$ is small: The system prioritizes minimizing queue sizes, ensuring rapid packet delivery at the expense of underutilizing available capacity.
This mathematical framework allows operators to precisely tune the system based on traffic types. For instance, Jaramillo et al. (2009) utilize a similar frame-based framework to balance "elastic" (best-effort, throughput-hungry) data with "inelastic" (real-time, delay-sensitive) traffic. By adjusting these internal weights, the network can approach optimal throughput while respecting strict packet deadlines. Similar parameterizations are used to balance operational costs against response times in software-defined networks (Huang et al., 2017; Huang, 2020).
The Perils of Delay-Blindness
What happens if a system ignores this trade-off and greedily optimizes for long-term throughput alone? It becomes "delay-blind," often resulting in catastrophic latency degradation.
In opportunistic spectrum access, traditional protocols wait for peak channel conditions to transmit data, maximizing spectral efficiency. However, Hu (2021) demonstrates that maximizing long-term average throughput fails to minimize the actual time required to complete a specific file transfer. For small or bursty files, the latency incurred while waiting in a buffer for a high-rate channel to open up frequently exceeds the actual transmission speed benefits.
Furthermore, greedily chasing theoretical throughput bounds can cause severe node starvation. Liu et al. (2009) proved that for distributed CSMA algorithms to close the gap to perfect theoretical utility, the average waiting time for an individual link must grow exponentially relative to the desired efficiency. When distributed scheduling algorithms rely on slowly varying weight functions to guarantee throughput stability, they inevitably sacrifice short-term latency due to the time-scale separation required for convergence (Shin, 2012).
Exploiting Slack and Co-Designing the Frontier
To break out of this rigid binary choice, modern architectures focus on exploiting "slack" in system requirements or co-optimizing across different layers of the stack.
One powerful insight is that aiming for absolute perfection (e.g., zero packet drops or zero deadline misses) forces systems into overly conservative, low-throughput regimes. Xia et al. (2008) demonstrate that by targeting a controlled, non-zero deadline miss ratio via fuzzy logic control, a wireless sensor network can dramatically increase its total data throughput while keeping communication delays within an acceptable operational range.
A similar counterintuitive phenomenon occurs at the physical layer of wireless communications. Du et al. (2019) reveal that in Massive MIMO systems, aiming for the lowest possible physical layer error rate actually increases overall end-to-end latency. To achieve near-zero error rates, the system must use conservative, slower transmission speeds, which causes packets to rapidly accumulate in the queue. By accepting a slightly higher error rate, the system can transmit faster; the minor latency penalty from occasional retransmissions is far outweighed by the massive reduction in queueing delay.
[Conservative Transmission] ---> Low Error Rate ---> Deep Queues ---> HIGH LATENCY
[Aggressive Transmission] ---> High Error Rate ---> Short Queues ---> LOW LATENCY (Optimal)
Finally, systems can use precise, real-time feedback to maintain high utilization without building up massive queues. Lee et al. (2015) show that by using accurate, single-packet latency measurements as a direct congestion signal rather than waiting for packet loss, datacenter networks can maintain near-perfect utilization (99.9%) while keeping queueing delays near zero.
By treating latency not as an immutable constraint, but as a managed decision variable with an explicit cost (Maity et al., 2018), modern system designers can map out the exact Pareto frontier of their application and dynamically shift the balance as network conditions change.