QCD on the Cell Broadband Engine

Francesco Belletti, Gianfranco Bilardi, M. Drochner, N. Eicker, Z. Fodor, Dieter Hierl, H. Kaldass, Thomas Lippert, T. Maurer, Nils Meyer, Andrea Nobile, D. Pleiter, Andreas T. Schaefer, F. Schifano, Hubert Simma, Stefan Solbrig, T. Streuer, R. Tripiccione, Tilo WettigarXiv 2007 · DOI · arXiv

Abstract

We evaluate IBM's Enhanced Cell Broadband Engine (BE) as a possible building block of a new generation of lattice QCD machines. The Enhanced Cell BE will provide full support of double-precision floating-point arithmetics, including IEEE-compliant rounding. We have developed a performance model and applied it to relevant lattice QCD kernels. The performance estimates are supported by micro- and application-benchmarks that have been obtained on currently available Cell BE-based computers, such as IBM QS20 blades and PlayStation 3. The results are encouraging and show that this processor is an interesting option for lattice QCD applications. For a massively parallel machine on the basis of the Cell BE, an application-optimized network needs to be developed.

Introduction: The Quest for Efficient Lattice QCD

Lattice Quantum Chromodynamics (QCD)—the study of the strong nuclear force on a discrete space-time grid—is one of the most computationally demanding fields in physics. Historically, this has required custom-built supercomputers or massive clusters of expensive CPUs. In 2007, the high-performance computing (HPC) community sought a more cost-effective solution, turning their attention to the Cell Broadband Engine (BE).

The heterogeneous architecture of the Cell Broadband Engine, featuring a PowerPC core (PPE) and eight Synergistic Processor Elements (SPEs) linked by the Element Interconnect Bus (EIB).

The Cell BE was an attractive candidate due to its heterogeneous architecture: one general-purpose PowerPC core (PPE) and eight high-speed Synergistic Processor Elements (SPEs). However, the original Cell was notoriously difficult to program and suffered from poor double-precision (DP) performance. This study evaluates the "Enhanced" version of the chip to determine its viability for the rigors of QCD simulations.

Performance Modeling and Methodology

Rather than relying solely on raw benchmarks, the researchers developed a theoretical performance model to predict how the architecture would handle specific QCD kernels, specifically the Wilson-Dirac operator (the 4-d hopping term).

A schematic representation of the performance model, detailing the data transfer timings (T) between the Local Store, Main Memory, and external network.

The model utilized a linear ansatz to calculate execution time ($T_i$) based on data exchange ($I_i$), bandwidth ($\beta_i$), and latency ($\lambda_i$). The team explored two primary memory strategies:

  1. On-chip Local Store (LS): Keeping all data within the 256KB memory of each SPE.
  2. Off-chip Main Memory (MM): Utilizing a "multiple buffering" scheme to stream data from main memory into the SPEs while the processor simultaneously performs calculations.

To validate these models, micro-benchmarks were performed on Sony PlayStation 3 consoles, IBM QS20 Blade Servers, and Mercury Cell Accelerator Boards.

Key Insights and Results

  • The "Enhanced" Cell Advantage: The original Cell BE was optimized for single-precision, but the 2008 "Enhanced" version (PowerXCell 8i) introduced ~100 GFlop/s in double precision with IEEE-compliant rounding, making it a viable scientific tool.
  • Efficiency Constraints: Due to instruction set constraints, the implementation of the Wilson-Dirac operator cannot exceed 78% of the peak performance of the Cell BE. The model predicted a theoretical efficiency of 27% to 34%, which was validated by benchmarks on the IBM QS20 showing actual execution times within 20% of predictions.
  • Alignment and Bottlenecks: Memory alignment is critical; misaligned access (not on a 128-byte boundary) introduces a latency penalty of roughly 16 cycles per block. Furthermore, the bottleneck shifts based on lattice volume: small volumes are limited by node-to-node communication bandwidth ($T_{ext}$), while larger volumes are limited by main memory bandwidth ($T_{mem}$).

Conclusion and Future Outlook

The study concludes that a sustained performance of above 20% is achievable on large-scale Cell BE machines. However, the existing I/O infrastructure (the "southbridge") was not designed for the massive, low-latency parallel communication required for QCD. To scale to a truly massively parallel machine, the authors propose the development of a custom network coprocessor using a 3-d torus topology with bidirectional bandwidth of 6 GB/s and sub-microsecond latency.

Reference: Belletti, F., et al. (2007). QCD on the Cell Broadband Engine. Proceedings of Science (LATTICE 2007).

Summary updated

Abstract

We evaluate IBM's Enhanced Cell Broadband Engine (BE) as a possible building block of a new generation of lattice QCD machines. The Enhanced Cell BE will provide full support of double-precision floating-point arithmetics, including IEEE-compliant rounding. We have developed a performance model and applied it to relevant lattice QCD kernels. The performance estimates are supported by micro- and application-benchmarks that have been obtained on currently available Cell BE-based computers, such as IBM QS20 blades and PlayStation 3. The results are encouraging and show that this processor is an interesting option for lattice QCD applications. For a massively parallel machine on the basis of the Cell BE, an application-optimized network needs to be developed.
BibTeX
@misc{belletti2007qcd,
  author = {Francesco Belletti and Gianfranco Bilardi and M. Drochner and N. Eicker and Z. Fodor and Dieter Hierl and H. Kaldass and Thomas Lippert and T. Maurer and Nils Meyer and Andrea Nobile and D. Pleiter and Andreas T. Schaefer and F. Schifano and Hubert Simma and Stefan Solbrig and T. Streuer and R. Tripiccione and Tilo Wettig},
  title = {{QCD on the Cell Broadband Engine}},
  year = {2007},
  eprint = {0710.2442},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/0710.2442},
  howpublished = {arXiv},
}
Figure