Abstract
Ongoing climate change calls for fast and accurate weather and climate modeling. However, when solving large-scale weather prediction simulations, state-of-the-art CPU and GPU implementations suffer from limited performance and high energy consumption. These implementations are dominated by complex irregular memory access patterns and low arithmetic intensity that pose fundamental challenges to acceleration. To overcome these challenges, we propose and evaluate the use of near-memory acceleration using a reconfigurable fabric with high-bandwidth memory (HBM). We focus on compound stencils that are fundamental kernels in weather prediction models. By using high-level synthesis techniques, we develop NERO, an FPGA+HBM-based accelerator connected through IBM CAPI2 (Coherent Accelerator Processor Interface) to an IBM POWER9 host system. Our experimental results show that NERO outperforms a 16-core POWER9 system by 4.2x and 8.3x when running two different compound stencil kernels. NERO reduces the energy consumption by 22x and 29x for the same two kernels over the POWER9 system with an energy efficiency of 1.5 GFLOPS/Watt and 17.3 GFLOPS/Watt. We conclude that employing near-memory acceleration solutions for weather prediction modeling is promising as a means to achieve both high performance and high energy efficiency.
Introduction: The Memory Wall in Climate Modeling
Modern weather and climate models, such as the widely used COSMO model, rely heavily on compound stencils—mathematical operations that update grid points based on neighboring values. These operations are inherently "memory-bound" due to their low arithmetic intensity. In traditional CPU and GPU architectures, the processor often remains idle while waiting for data from memory, leading to significant performance bottlenecks and high energy waste.
As shown in the roofline model above, the attainable performance for kernels like hdiff and vadvc on a POWER9 system is severely constrained by memory bandwidth. To address this, specialized hardware that brings computation closer to high-bandwidth memory is required to break the "memory wall."
The NERO Architecture
NERO (Near-HBM Weather Prediction Accelerator) is designed to overcome these limitations by integrating a reconfigurable FPGA with High-Bandwidth Memory (HBM2). This approach moves the computation closer to the data, significantly reducing the latency and energy costs associated with traditional memory hierarchies.
The architecture leverages several key components:
- FPGA + HBM2: A Xilinx Virtex UltraScale+ FPGA utilizes stacked HBM2 memory, providing massive throughput compared to standard DDR4. This multi-channel architecture allows multiple processing elements to access data in parallel without congestion.
- CAPI2 Interconnect: The accelerator connects to an IBM POWER9 host via the Coherent Accelerator Processor Interface (CAPI2), enabling low-latency, coherent access to host memory as a peer to the CPU.
- Custom Memory Hierarchy: NERO implements a data-centric caching scheme, decomposing 3D grid data into smaller "windows" stored in internal URAM/BRAM. This bridges the latency gap between the HBM and the processing elements.
- Automated Tuning: The system uses OpenTuner (a machine-learning framework) to automatically optimize "tile sizes" for different data precisions, ensuring peak efficiency without manual intervention.
Experimental Evaluation
The system was evaluated using a 256 × 256 × 64-point grid domain, focusing on two critical kernels from the COSMO model: vadvc (Vertical Advection), which has complex vertical dependencies, and hdiff (Horizontal Diffusion), which is highly data-parallel.
Performance and Efficiency Results
Compared to a 16-core IBM POWER9 system running 64 threads, NERO demonstrated substantial improvements:
- Speedup: NERO achieved a 4.2x speedup for vadvc and an 8.3x speedup for hdiff.
- Energy Efficiency: Energy consumption was reduced by 22x and 29x, respectively. NERO reached an energy efficiency of up to 17.3 GFLOPS/Watt, a massive leap over the CPU baseline.
- Scalability: While DDR4-based solutions often plateau early, NERO’s performance scales linearly with the number of processing elements due to the high number of independent HBM ports.
Conclusion
NERO demonstrates that near-memory acceleration using FPGA and HBM technology is a highly effective solution for the memory-bound challenges of weather prediction. By tailoring the memory hierarchy to specific stencil patterns and utilizing high-bandwidth interconnects like CAPI2, NERO achieves orders-of-magnitude improvements in energy efficiency and significant performance gains over general-purpose server CPUs. Future work will focus on integrating the entire dynamical core of the COSMO model into this architecture.
Summary updated