arXiv : A Comparison of CPU and GPU implementations for the LHCb Experiment Run 3 Trigger

R. Aaij, A. Szabelski, Xing Zhu, M. Szymanski, X. Cid Vidal, S. Esen, M. Mazurek, G. Bassi, A. Poluektov, F. Archilli, A. Dzyuba, P. Robbe, A. Dziurda, R. Quagliani, L. Zhang, B. Sciascia, Michael Frank, A. S. Boldyrev, S. Petrucci, M. Santimaria, R. Lane, Y. Amhis, D. Hill, A. Solomin, M. Calvo Gomez, G. Graziani, M. Petruzzo, S. Klaver, L. Dufour, R. J. Hunter, M. Cattaneo, P. Durante, Sokoloff, P. Spradlin, Michele Piero Blago, S. Filippov, S. Vecchi, N. Neufeld, H. Stevens, O. Madejczyk, Tomasz Wojtoń, F. Martínez-Vidal, H. Mu, R. I. Rabadán Trejo, D. P. O’Hanlon, R. Kopecna, R. Schwemmer, D. C. Craik, M. Witek, L. Bian, R. Le Gac, X. R. Lyu, C. Prouvé, B. Mitreska, P. Naik, N. Skidmore, M. Adinolfi, S. Amato, F. Pisani, D. Zhang, M. Lucio Martínez, A. Contu, L. Pica, V. Renaudin, P. A. Günther, J. Albrecht, F. Suljik, G. Punzi, F. Dordei, D. Y. Tou, M. Schiller, X. Vilasis-Cardona, W. D. Hulsbergen, M. Whitehead, B. K. Jashal, S. Borghi, Pei‐Rong Li, S. Hansmann-Menzemer, S. Stahl, R. Vazquez Gomez, W. Krzemień, Louis Henry, P. Fernandez Declara, Dorothea Vom Bruch, P. Gandini, S. Akar, Rangel, Chris Jones, E. Polycarpo, F. Polci, J. F. Marchand, M. Zdybał, B. Couturier, D. H. Cámpora Pérez, S. Malde, N. S. Nolte, T. Boettcher, M. Dorigo, M. Stahl, M. Pepe Altarelli

Abstract

The LHCb experiment at CERN is undergoing an upgrade in preparation for the Run 3 data-taking period of the LHC. As part of this upgrade, the trigger is moving to a fully software implementation operating at the LHC bunch crossing rate. We present an evaluation of a CPU-based and a GPU-based implementation of the first stage of the High Level Trigger (HLT1). After a detailed comparison, both options are found to be viable. This document summarizes the performance and implementation details of these options, the outcome of which has led to the choice of the GPU-based implementation as the baseline.

Introduction

The Large Hadron Collider (LHC) produces a massive volume of data that requires sophisticated filtering in real-time. For the "Run 3" upgrade, the LHCb experiment set an ambitious goal: eliminate the traditional hardware-based trigger and move to a "triggerless" readout. This means the system must process the full 30 MHz bunch crossing rate—roughly 32 Terabits per second—entirely in software. The challenge is to perform real-time reconstruction, including tracking particles and finding collision vertices, with enough precision to identify interesting physics while staying within a realistic budget for power, cooling, and computing hardware.

System Architecture and Data Flow

The core of the upgrade lies in the transition from hardware-based filtering to a high-throughput software trigger. The data flow starts at the detector and moves through a series of processing stages designed to reduce the 32 Tb/s stream to a manageable level for permanent storage.

The LHCb Run 3 trigger architecture, illustrating the data flow from the detector to the HLT1 and HLT2 processing farms.

As illustrated in Figure 2, the architecture involves 173 Event Builder (EB) servers. In a traditional CPU-only setup, the full 32 Tb/s must be distributed over a massive network to a dedicated HLT1 farm. However, the hybrid approach integrates GPU cards directly into these Event Builder servers. By performing HLT1 processing "in-place" on GPUs, the system can reduce the required network bandwidth to the downstream HLT2 farm and disk buffers by a factor of 30 to 60. This architectural shift is a key driver for the efficiency of the GPU-based solution.

Methodology and Implementation

The researchers compared two distinct architectures for the HLT1 stage:

  1. CPU-only: Utilizing dual-socket servers equipped with 32-core AMD EPYC processors.
  2. Hybrid (GPU-based): Utilizing the "Allen" framework, with GPU cards installed directly within the Event Builder servers.

The HLT1 sequence involves several complex reconstruction steps, including Velo reconstruction (decoding signals from the Vertex Locator), Primary Vertex (PV) finding, Forward Tracking (matching tracks with downstream stations), Muon Identification, and Kalman Filtering. To ensure a "like-for-like" comparison, identical physics logic was used across both platforms, though the code was optimized for the specific strengths of each hardware type, such as the massive parallelism inherent in GPUs.

Physics Performance and Results

A significant outcome of the study was the demonstration of physics parity between the two architectures. Despite the complexity of non-uniform tracking algorithms, the GPU-based Allen framework achieved efficiency, resolution, and ghost rates nearly identical to the CPU implementation. Tracking efficiency agreed to within 1% between the two platforms across various signal samples, including $J/\psi \to \mu^+\mu^-$, $B_s^0 \to \phi\phi$, and $Z \to \mu^+\mu^-$.

In terms of throughput, a single GPU card could process HLT1 at approximately 92 kHz, while a dual-socket AMD EPYC node managed 171 kHz. However, the smaller footprint and lower power consumption per event made the GPU approach more cost-effective. The entire 30 MHz stream can be handled by a compact system of roughly 170 CPU servers or 330 GPU cards.

Economic and Operational Impact

The economic evaluation favored the hybrid GPU approach. In units of "Quanta" (a reference server cost), the CPU-only sub-total was 2300 units compared to 1725 units for the hybrid GPU approach—a saving of approximately 25%. Furthermore, the GPU baseline is considered more future-proof, as GPU performance is currently advancing at a faster rate than CPU single-core performance, making it more adaptable to future increases in LHC luminosity.

Conclusion

While both CPU and GPU implementations proved technically viable, the GPU-based HLT1 was selected as the baseline for LHCb Run 3. This decision was driven by the significant infrastructure cost savings, the reduction in networking complexity, and the promising performance trajectory of GPU hardware. Challenges remain, such as the maintenance of two distinct software stacks (CUDA/HIP for HLT1 and C++ for HLT2/Offline) and the ongoing task of porting more complex HLT2 algorithms to GPUs. Nevertheless, the success of the Allen framework in HLT1 marks a transformative step for real-time data processing at the LHC.

Summary updated

Abstract

The LHCb experiment at CERN is undergoing an upgrade in preparation for the Run 3 data taking period of the LHC. As part of this upgrade the trigger is moving to a fully software implementation operating at the LHC bunch crossing rate. We present an evaluation of a CPU-based and a GPU-based implementation of the first stage of the High Level Trigger. After a detailed comparison both options are found to be viable. This document summarizes the performance and implementation details of these options, the outcome of which has led to the choice of the GPU-based implementation as the baseline.
BibTeX
@article{aaij2022arxiv,
  author = {R. Aaij and A. Szabelski and Xing Zhu and M. Szymanski and X. Cid Vidal and S. Esen and M. Mazurek and G. Bassi and A. Poluektov and F. Archilli and A. Dzyuba and P. Robbe and A. Dziurda and R. Quagliani and L. Zhang and B. Sciascia and Michael Frank and A. S. Boldyrev and S. Petrucci and M. Santimaria and R. Lane and Y. Amhis and D. Hill and A. Solomin and M. Calvo Gomez and G. Graziani and M. Petruzzo and S. Klaver and L. Dufour and R. J. Hunter and M. Cattaneo and P. Durante and Sokoloff and P. Spradlin and Michele Piero Blago and S. Filippov and S. Vecchi and N. Neufeld and H. Stevens and O. Madejczyk and Tomasz Wojtoń and F. Martínez-Vidal and H. Mu and R. I. Rabadán Trejo and D. P. O’Hanlon and R. Kopecna and R. Schwemmer and D. C. Craik and M. Witek and L. Bian and R. Le Gac and X. R. Lyu and C. Prouvé and B. Mitreska and P. Naik and N. Skidmore and M. Adinolfi and S. Amato and F. Pisani and D. Zhang and M. Lucio Martínez and A. Contu and L. Pica and V. Renaudin and P. A. Günther and J. Albrecht and F. Suljik and G. Punzi and F. Dordei and D. Y. Tou and M. Schiller and X. Vilasis-Cardona and W. D. Hulsbergen and M. Whitehead and B. K. Jashal and S. Borghi and Pei‐Rong Li and S. Hansmann-Menzemer and S. Stahl and R. Vazquez Gomez and W. Krzemień and Louis Henry and P. Fernandez Declara and Dorothea Vom Bruch and P. Gandini and S. Akar and Rangel and Chris Jones and E. Polycarpo and F. Polci and J. F. Marchand and M. Zdybał and B. Couturier and D. H. Cámpora Pérez and S. Malde and N. S. Nolte and T. Boettcher and M. Dorigo and M. Stahl and M. Pepe Altarelli},
  title = {{arXiv : A Comparison of CPU and GPU implementations for the LHCb Experiment Run 3 Trigger}},
  journal = {Comput. Softw. Big Sci.},
  year = {2022},
  doi = {10.1007/s41781-021-00070-2},
}
Figure