Extreme Scaling of Supercomputing with Stranded Power: Costs and Capabilities

Fan Yang, Andrew A. Chien

Abstract

As supercomputing scales toward Exascale and beyond, power consumption and carbon emissions have become critical bottlenecks. We propose the "Zero-Carbon Cloud" (ZCCloud), an architecture that exploits "stranded power"—excess renewable energy that the electrical grid cannot accept. By co-locating containerized supercomputing modules with wind farms, ZCCloud can reduce total cost of ownership (TCO) by up to 45% and enable an 80% increase in peak performance for a fixed budget. This analysis characterizes the temporal and geographic properties of stranded power and models the economic viability of this approach for future extreme-scale systems.

The Power Wall and the Stranded Power Opportunity

High-performance computing (HPC) is hitting a "power wall." Modern systems draw over 25MW, leading to massive operational costs and environmental impacts. As we look toward systems requiring 100MW or more, traditional grid-connected data centers become increasingly unsustainable. Simultaneously, the power grid struggles to integrate variable renewables like wind and solar. Often, these sources produce more electricity than the grid can handle, leading to "stranded power" that is either "curtailed" (wasted) or sold at negative prices.

ZCCloud addresses both issues by moving the computers to the wasted energy. This approach embraces intermittency, using low-cost containerized deployments at renewable energy sites. By bypassing the grid's transmission constraints, supercomputers can tap into a massive, zero-cost energy resource.

System overview of Mira-ZCCloud, showing the connection between traditional data centers and wind-powered containerized modules.

Methodology and Data Characterization

The feasibility of ZCCloud was evaluated through three primary lenses:

  1. Stranded Power Analysis: We analyzed 70 million transactions from the MISO (Midcontinent Independent System Operator) grid across 10 US states (2013-2015). The study identified 1,259 sites, including 200 wind sites. In 2014 alone, MISO had 7.7 TWh of stranded wind power—enough to support multiple Top 10 supercomputers simultaneously.
  2. Workload Simulation: Using the Qsim simulator and real job traces (78,795 jobs) from the Mira supercomputer—a 10-petaflop IBM BlueGene/Q system at Argonne National Laboratory—we modeled how intermittent power affects system throughput.
  3. TCO Modeling: We developed a comprehensive model accounting for computing hardware, specialized modular facilities, networking, and the unique "zero-cost" nature of stranded power.
Monthly wind generation and DIR Dispatch Down (stranded power) in the MISO grid during 2014.

Key Insights: Intermittency as an Asset

While traditional HPC requires 100% uptime, ZCCloud demonstrates that a "duty factor" (uptime) of 60-80% is sufficient if systems are scaled larger to compensate for downtime. A key finding is the "NetPrice" advantage: by looking at average power prices over longer intervals rather than 5-minute spikes, stranded power becomes much more stable. This model provides continuous runtimes often exceeding 10 hours, which is ideal for typical HPC job durations.

Furthermore, ZCCloud eliminates the need for expensive, multi-million dollar raised-floor data centers. By utilizing modular containers and "free cooling" (ambient air) at wind farm sites, the facility cost displacement significantly contributes to the overall TCO reduction.

Results and Scaling Potential

The ZCCloud approach is highly cost-effective in current and future scenarios:

  • TCO Reduction: 21% to 45% compared to traditional centers.
  • Performance Gains: For a fixed $250M annual budget, the ZCCloud model can achieve 80% higher peak PFLOPS than a traditional facility.
  • Cost-Effectiveness: Throughput per dollar improves by 34% in baseline scenarios. In regions with high electricity costs (like Germany or Denmark), this advantage jumps to 55%.

As hardware becomes more "commoditized" (cheaper) and system power density increases, the benefits rise. In a future where power density is 5x higher, ZCCloud could be 116% more cost-effective than traditional builds. For extreme-scale systems beyond terascale, where power requirements exceed 100MW, ZCCloud remains the most viable path for scaling within a fixed budget.

Conclusion and Future Work

ZCCloud offers a viable path to scaling supercomputing sustainably by turning the "problem" of renewable intermittency into an economic opportunity. While challenges remain regarding checkpointing overhead for long-running jobs and the cost of high-speed networking to remote sites, the economic and environmental benefits are compelling. The authors are currently moving beyond simulation to build physical prototypes, including a planned 2,500-core containerized system on the Open Science Grid, to validate these theories in real-world conditions.

Summary updated

Abstract

Power consumption (supply, heat, cost) and associated carbon emissions (environmental impact) are increasingly critical challenges in scaling supercomputing to Exascale and beyond. We proposes to exploit stranded power, renewable energy that has no value to the power grid, for scaling supercomputers, Zero-Carbon Cloud (ZCCloud), and showing that stranded power can be employed effectively to expand computing [1]. We build on those results with a new analysis of stranded power, characterizing temporal, geographic, and interval properties. We simulate production supercomputing workloads and model datacenter total-cost-of-ownership (TCO), assessing the costs and capabilities of stranded-power based supercomputing. Results show that the ZCCloud approach is cost-effective today in regions with high cost power. The ZCCloud approach reduces TCO by 21-45%, and improves cost-effectiveness up to 34%. We study many scenarios. With higher power price, cheaper computing hardware and higher system power density, benefits rise to 55%, 97% and 116% respectively. Finally, we study future extreme-scale systems, showing that beyond terascale, projected power requirements in excess of 100MW make ZCCloud up to 45% lower cost, for a fixed budget, increase peak PFLOPS achievable by 80%.
BibTeX
@misc{yang2016extreme,
  author = {Fan Yang and Andrew A. Chien},
  title = {{Extreme Scaling of Supercomputing with Stranded Power: Costs and Capabilities}},
  year = {2016},
  eprint = {1607.02133},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1607.02133},
}
Figure