eFAIR: Fair LLM Serving under Hourly Electricity Commitments

You Peng, Youhe Jiang, Chen Wang, Binhang YuanarXiv 2026 · arXiv

TL;DR

eFAIR is an online scheduling framework designed to provide fair throughput for Large Language Model (LLM) requests while strictly adhering to hourly electricity budgets. By intelligently coordinating request dispatching, energy reservations, and GPU frequency scaling, eFAIR improves service for the most underserved requests by up to 31% compared to existing state-of-the-art baselines.

What Problem Does This Paper Address?

Datacenters often purchase electricity in "day-ahead" markets, committing to specific energy budgets for individual hours. For LLM serving, this creates a complex balancing act. If a scheduler simply tries to maximize total throughput, it might favor short, "easy" requests while leaving others waiting indefinitely. Conversely, spending energy too quickly early in the hour might leave the system "bankrupt" when a burst of high-priority requests arrives later.

The core challenge is achieving request-level fairness under a hard temporal energy constraint. This is difficult because:

  1. Heterogeneity: Requests vary in processing cost (prompt length, output length) and desired service levels (entitlements).
  2. Uncertainty: The scheduler doesn't know when future requests will arrive or how long they will be.
  3. Coupled Decisions: Choosing which request to run, which batch to put it in, and what GPU frequency to use all simultaneously affect both fairness and energy consumption.

Methods

The authors introduce eFAIR, which centers around a new metric called normalized throughput. This metric divides a request's actual output rate by its "standalone" throughput (how fast it would run alone) and its "service entitlement" (a weight representing its priority).

Figure 1: Normalization accounts for differences in request difficulty and priority.

eFAIR formulates the problem as a leximin optimization, which essentially means "make the worst-off user as happy as possible, then move to the next worst-off." The framework consists of three main components:

  1. Request Dispatcher: Prioritizes requests based on their entitlement and current progress. It ensures that "VIP" requests get admitted sooner while maintaining efficient batching for ordinary requests.
  2. Electricity Budget Allocator: Acts as a "shared account" manager. It tracks past consumption and reserves energy for both currently running tasks and expected future arrivals.
  3. Frequency Selector: Dynamically adjusts GPU frequencies. It selects the most energy-efficient frequency that meets a minimum service target for a batch. If there is an energy "surplus," it uses that extra budget to boost frequencies for higher-entitlement requests.
Figure 3: The eFAIR framework coordinates dispatching, budgeting, and frequency selection.

Datasets

The system was evaluated using real-world Azure LLM inference traces from May 2024.

  • H20 Configuration: Replays a trace with ~35,600 requests.
  • H200 Configuration: Replays a heavier trace with ~60,700 requests. Both configurations utilized 32 model replicas (NVIDIA H20/H200 GPUs) and tested varying service entitlements.

Key Insights and Ideas

  • Fairness Requires Normalization: Simply giving every request the same number of tokens per second isn't fair if one request is significantly harder to process than another. Normalizing by standalone throughput creates a level playing field.
  • Energy Spending Should Follow Demand: Uniformly spending energy across an hour is inefficient. eFAIR uses "demand-aware" allocation, spending more during busy bursts and saving during quiet periods to ensure consistent service quality.
Figure 2: Demand-aware spending (bottom) aligns energy use with actual workload bursts compared to uniform spending (middle).
  • GPU Frequency as a Fairness Lever: Instead of just using frequency scaling for power saving, eFAIR uses it to "buy" better service for underserved or high-priority requests when the budget allows.
  • Leximin is Robust: By focusing on the "tail" (the 10th percentile of users), eFAIR ensures that the system doesn't just look good on average while failing individual users.

Key Results

In head-to-head comparisons against baselines like vLLM and Weighted Virtual Token Counter (VTC), eFAIR demonstrated superior performance:

  • Tail Service Improvement: eFAIR improved the 10th percentile (P10) of normalized throughput by 28.0% to 31.0%.
  • Budget Precision: It maintained absolute deviations from the hourly electricity commitment below 0.26%, showing remarkable control over energy spend.
  • Reduced Starvation: As shown in the throughput distribution, eFAIR significantly reduced the number of requests that received near-zero service.
Figure 4: eFAIR (blue) shifts the distribution of service toward higher normalized throughput, especially for the worst-served requests.

Limitations and Future Work

  • Prediction Dependency: eFAIR relies on estimates of request output lengths. While the authors showed the system is robust to moderate prediction errors, extreme inaccuracies could still degrade performance.
  • Homogeneous Clusters: The current evaluation focuses on clusters of identical GPUs. Extending this to heterogeneous environments (e.g., a mix of H100s and A100s) would add another layer of complexity to the "standalone throughput" baseline.
  • Fixed Commitments: The paper assumes a fixed energy budget per hour. Future work could explore more dynamic scenarios where electricity prices fluctuate within the hour, requiring the scheduler to shift work to the cheapest minutes.

Summary updated

Abstract

Datacenters can procure electricity in day-ahead markets by committing to energy purchases for individual hours. For LLM services operating within these hourly budgets, maximizing aggregate throughput alone may still leave individual requests waiting or receiving much slower service than desired. Fair serving therefore requires improving service for poorly served requests while reserving electricity for later arrivals whose timing and demand are unknown. To compare service across requests, we define normalized throughput as each request's actual output rate divided by the product of its standalone throughput and its service entitlement, a positive weight representing its desired service level. Using this measure, we formulate fair serving as leximin optimization under an hourly electricity constraint, prioritizing requests with the lowest normalized throughput. To address this problem, we present eFAIR, an online scheduler that coordinates request dispatching, electricity budget allocation, and GPU frequency selection through a common normalized-throughput target. Specifically, request dispatching prioritizes requests by desired service level and progress; electricity budget allocation reserves electricity for current and expected future work; and frequency selection uses electricity savings to raise GPU frequencies for requests with higher desired service levels. Evaluations on H20 and H200 configurations, each with 32 replicas, show that eFAIR improves the tenth percentile of normalized throughput by 28.0%-31.0% over the best budget-feasible baseline, while keeping absolute deviations from the hourly electricity commitment below 0.26%. eFAIR also improves throughput balance among concurrent requests and aligns electricity allocation more closely with service demand over time. Together, these results demonstrate how eFAIR improves request-level fairness within an hourly commitment.
BibTeX
@misc{peng2026efair,
  author = {You Peng and Youhe Jiang and Chen Wang and Binhang Yuan},
  title = {{eFAIR: Fair LLM Serving under Hourly Electricity Commitments}},
  year = {2026},
  eprint = {2611.24119},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24119},
}
Figure