Distributed LLM Pretraining During Renewable Curtailment Windows: A Feasibility Study

Philipp Wiesner, Sören Becker, Brett Cornick, Dominik Scheinert, Alexander Acker, Odej KaoarXiv 2026 · arXiv · OpenReview

TL;DR

This paper presents a system for training large language models (LLMs) by opportunistically leveraging periods of excess renewable energy (curtailment windows). By dynamically coordinating geo-distributed GPU clusters, they demonstrate that LLM pretraining can be performed using clean, cheap electricity without significantly degrading training quality, reducing operational emissions by up to 95%.

What Problem Does This Paper Address?

Training large language models (LLMs) is incredibly compute- and energy-intensive. Simultaneously, the increasing reliance on renewable energy sources like solar and wind leads to "curtailment" – periods where more clean electricity is generated than the grid can handle, forcing it to be wasted. This presents a prime opportunity: could we align LLM training with these curtailment windows to use clean, low-cost energy? The challenge lies in the tightly coupled and stateful nature of LLM pretraining, which typically requires sustained throughput. Pausing, resuming, or migrating training across geographically distributed sites due to intermittent curtailment windows introduces significant overheads that could negate the benefits. This paper investigates the feasibility of performing full-parameter LLM pretraining across geo-distributed GPU clusters during these renewable curtailment periods.

Methods

The core of the system is a reactive, elastic architecture that dynamically adapts to the availability of training sites. A central coordinator monitors regional curtailment signals.

  • Elastic Provisioning: When a curtailment window is detected in a region, the system provisions a GPU node. Conversely, when the window closes (with a hysteresis period to avoid oscillations), the site is deprovisioned.
  • Dynamic Execution Modes:
    • Single-site training: If only one site is curtailed, it trains locally without synchronization overhead.
  • Federated mode: If multiple sites are curtailed simultaneously, they train in parallel. The coordinator orchestrates periodic synchronization rounds using work-weighted FedAvg [33] to aggregate model states. This approach is chosen for its robustness to dynamic participation and heterogeneous throughput.
  • Curtailment-Aware Provisioning: To mitigate the high cost of provisioning (model transfer, optimizer warm-up), a hysteresis mechanism is employed, meaning sites stay active for a short period even after the curtailment signal ends.
  • Round Sizing: Instead of fixed local steps, training rounds are defined by wall-clock time to better handle straggler effects and dynamic site availability.

Datasets

  • Training Data: The paper uses the FineWebEdu-100B dataset, consisting of 12.8 billion tokens, for pretraining the LLM.
  • Curtailment Data: Real-world marginal carbon intensity traces provided by WattTime [44] are used to derive curtailment windows. These traces are replayed using Vessim [45] for controlled evaluations.

Key Insights and Ideas

  • Curtailment as a Compute Opportunity: The paper firmly establishes that periods of renewable energy surplus are not just for energy savings but can be actively utilized for computationally intensive tasks like LLM pretraining.
  • Federated Learning as an Enabler: The use of federated learning is crucial for handling the dynamic and geographically distributed nature of the training sites, allowing for flexible participation.
  • Systems Overhead is Key: The primary challenge isn't the LLM training algorithm itself, but the systems overhead of provisioning, synchronization, and communication. The paper's success hinges on managing these efficiently.
  • Reactive and Elastic Design is Essential: Given the inherent uncertainty and intermittency of renewable energy, a system that can react quickly and adapt its execution mode is paramount.
  • Trade-offs in Round Sizing: Defining round durations involves a trade-off between amortization of synchronization overhead and the potential for delayed convergence.

Key Results

  • Reduced Operational Emissions: The curtailment-aware approach reduced operational emissions to 5-12% of single-region baselines.
  • High Curtailment Utilization: The system achieved 97% of training energy usage during curtailment windows.
  • Comparable Training Quality: The final EMA-smoothed perplexity of the trained model was 15.1, very close to the centralized baseline of 14.8, demonstrating that training quality is largely preserved.
  • Runtime: The curtailment-aware approach completed training in 14.6 hours, falling between the centralized baseline (17.8 hours) and a continuous two-site FL baseline (11.1 hours).
  • Modest Energy Overhead: While dynamic provisioning and synchronization introduced a slight increase in total energy consumption (37.7 kWh vs. 36.0 kWh for centralized), this was deemed acceptable for the significant carbon savings.

Limitations and Future Work

  • Geographic Optimism: The study used four regions with frequent curtailment, and one (Germany) exhibited no curtailment windows during the experiment. This selection was optimistic and may not reflect all real-world scenarios.
  • Window Duration vs. Overhead: The effectiveness of curtailment-aware training is sensitive to the duration of curtailment windows relative to provisioning, synchronization, and state restoration overheads. Short windows can be dominated by these costs.
  • Scalability Challenges: Scaling to larger models and more sites introduces complexities like wide-area communication, data movement energy, and more robust progress tracking.
  • Carbon Accounting Nuances: The paper acknowledges that using marginal operating emissions rates is a proxy and that ongoing research is needed to refine carbon accounting methods, especially regarding temporal matching and the choice of emissions factors.
  • Future Work: The authors suggest exploring topology-aware synchronization, warm standby capacity, and investigating the use of "dirty" energy for runtime guarantees. They also highlight the potential for curtailment-aware scheduling to benefit other energy-intensive AI workloads beyond pretraining.

Summary updated

Abstract

Training large language models (LLMs) requires substantial compute and energy. At the same time, renewable energy sources regularly produce more electricity than the grid can absorb, leading to curtailment, the deliberate reduction of clean generation that would otherwise go to waste. These periods represent an opportunity: if training is aligned with curtailment windows, LLMs can be pretrained using electricity that is both clean and cheap. This technical report presents a system that performs full-parameter LLM training across geo-distributed GPU clusters during regional curtailment windows, elastically switching between local single-site training and federated multi-site synchronization as sites become available or unavailable. Our prototype trains a 561M-parameter transformer model across three clusters using the Flower federated learning framework, with curtailment periods derived from real-world marginal carbon intensity traces. Preliminary results show that curtailment-aware scheduling preserves training quality while reducing operational emissions to 5-12% of single-site baselines.
BibTeX
@misc{wiesner2026distributed,
  author = {Philipp Wiesner and Sören Becker and Brett Cornick and Dominik Scheinert and Alexander Acker and Odej Kao},
  title = {{Distributed LLM Pretraining During Renewable Curtailment Windows: A Feasibility Study}},
  year = {2026},
  eprint = {2602.22760},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.22760},
}
Figure