TL;DR
eFAIR is an online scheduling framework designed to provide fair throughput for Large Language Model (LLM) requests while strictly adhering to hourly electricity budgets. By intelligently coordinating request dispatching, energy reservations, and GPU frequency scaling, eFAIR improves service for the most underserved requests by up to 31% compared to existing state-of-the-art baselines.
What Problem Does This Paper Address?
Datacenters often purchase electricity in "day-ahead" markets, committing to specific energy budgets for individual hours. For LLM serving, this creates a complex balancing act. If a scheduler simply tries to maximize total throughput, it might favor short, "easy" requests while leaving others waiting indefinitely. Conversely, spending energy too quickly early in the hour might leave the system "bankrupt" when a burst of high-priority requests arrives later.
The core challenge is achieving request-level fairness under a hard temporal energy constraint. This is difficult because:
- Heterogeneity: Requests vary in processing cost (prompt length, output length) and desired service levels (entitlements).
- Uncertainty: The scheduler doesn't know when future requests will arrive or how long they will be.
- Coupled Decisions: Choosing which request to run, which batch to put it in, and what GPU frequency to use all simultaneously affect both fairness and energy consumption.
Methods
The authors introduce eFAIR, which centers around a new metric called normalized throughput. This metric divides a request's actual output rate by its "standalone" throughput (how fast it would run alone) and its "service entitlement" (a weight representing its priority).
eFAIR formulates the problem as a leximin optimization, which essentially means "make the worst-off user as happy as possible, then move to the next worst-off." The framework consists of three main components:
- Request Dispatcher: Prioritizes requests based on their entitlement and current progress. It ensures that "VIP" requests get admitted sooner while maintaining efficient batching for ordinary requests.
- Electricity Budget Allocator: Acts as a "shared account" manager. It tracks past consumption and reserves energy for both currently running tasks and expected future arrivals.
- Frequency Selector: Dynamically adjusts GPU frequencies. It selects the most energy-efficient frequency that meets a minimum service target for a batch. If there is an energy "surplus," it uses that extra budget to boost frequencies for higher-entitlement requests.
Datasets
The system was evaluated using real-world Azure LLM inference traces from May 2024.
- H20 Configuration: Replays a trace with ~35,600 requests.
- H200 Configuration: Replays a heavier trace with ~60,700 requests. Both configurations utilized 32 model replicas (NVIDIA H20/H200 GPUs) and tested varying service entitlements.
Key Insights and Ideas
- Fairness Requires Normalization: Simply giving every request the same number of tokens per second isn't fair if one request is significantly harder to process than another. Normalizing by standalone throughput creates a level playing field.
- Energy Spending Should Follow Demand: Uniformly spending energy across an hour is inefficient. eFAIR uses "demand-aware" allocation, spending more during busy bursts and saving during quiet periods to ensure consistent service quality.
- GPU Frequency as a Fairness Lever: Instead of just using frequency scaling for power saving, eFAIR uses it to "buy" better service for underserved or high-priority requests when the budget allows.
- Leximin is Robust: By focusing on the "tail" (the 10th percentile of users), eFAIR ensures that the system doesn't just look good on average while failing individual users.
Key Results
In head-to-head comparisons against baselines like vLLM and Weighted Virtual Token Counter (VTC), eFAIR demonstrated superior performance:
- Tail Service Improvement: eFAIR improved the 10th percentile (P10) of normalized throughput by 28.0% to 31.0%.
- Budget Precision: It maintained absolute deviations from the hourly electricity commitment below 0.26%, showing remarkable control over energy spend.
- Reduced Starvation: As shown in the throughput distribution, eFAIR significantly reduced the number of requests that received near-zero service.
Limitations and Future Work
- Prediction Dependency: eFAIR relies on estimates of request output lengths. While the authors showed the system is robust to moderate prediction errors, extreme inaccuracies could still degrade performance.
- Homogeneous Clusters: The current evaluation focuses on clusters of identical GPUs. Extending this to heterogeneous environments (e.g., a mix of H100s and A100s) would add another layer of complexity to the "standalone throughput" baseline.
- Fixed Commitments: The paper assumes a fixed energy budget per hour. Future work could explore more dynamic scenarios where electricity prices fluctuate within the hour, requiring the scheduler to shift work to the cheapest minutes.
Summary updated