A Comparison of CPU and GPU Implementations for the LHCb Experiment Run 3 Trigger

Roel Aaij, Marta Adinolfi, Salvatore Aiola, S. Akar, Johannes Albrecht, M. Alexander, S. Amato, Yasmine Amhis, F. Archilli, M. Bala, G. Bassi, L. Bian, M. P. Blago, Thomas Boettcher, Alexey Boldyrev, S. Borghi, A. Brea Rodriguez, Lukas Calefice, M. Calvo Gomez, Daniel Hugo Cámpora Pérez, A. Cardini, Marco Cattaneo, V. Chobanova, G. Ciezarek, Xabier Cid Vidal, J. L. Cobbledick, J. A. B. Coelho, Tommaso Colombo, A. Contu, Benjamin Couturier, Daniel C. Craik, R. Currie, P. d'Argent, Michel De Cian, Denis Derkach, F. Dordei, Marco Dorigo, Laurent Dufour, Paolo Durante, A. Dziurda, A. Dzyuba, S. Easo, S. Esen, Placido Fernandez Declara, Sergey Filippov, Conor Fitzpatrick, M. Frank, Paolo Gandini, Vladimir V. Gligorov, Elisabet Golobardes, G. Graziani, L. Grillo, P. A. Günther, S. Hansmann-Menzemer, Arthur M. Hennequin, Louis Henry, D. Hill, S. E. Hollitt, J. Hu, W. Hulsbergen, R. J. Hunter, Mikhail Hushchyn, Brij Kishor Jashal, Christopher Robert Jones, S. Klaver, Konrad Klimaszewski, R. Kopecna, Wojciech Krzemien, Marcin Kucharczyk, R. Lane, Federico Lazzari, R. Le Gac, P. Li, J. H. Lopes, M. Lucio Martinez, Anna Lupato, Oliver Lupton, X. Lyu, F. Machefert, Olga Madejczyk, S. Malde, J. F. Marchand, Saverio Mariani, Carla Marin Benito, D. Martinez Santos, F. Martinez Vidal, Rosen Matev, Michal Mazurek, Biljana Mitreska, Dominik Stefan Mitzel, Michael Joseph Morello, H. Mu, P. Muzzetto, P. Naik, Matthew D. Needham, N. Neri, Niko Neufeld, N. S. Nolte, Daniel O'Hanlon, A. Oyanguren Campos, M. Pepe Altarelli, S. Petrucci, Marco Petruzzo, L. Pica, Flavio Pisani, A. Piucci, Francesco Polci, A. Poluektov, E. Polycarpo, C. Prouve, Giovanni Punzi, Renato Quagliani, R. I. Rabadan Trejo, Miguel Ramos Pernas, M. S. Rangel, Fedor Ratnikov, Gerhard Raven, F. Reiss, V. Renaudin, P. Robbe, Artem Ryzhikov, Marco Santimaria, M. Saur, Manuel T. Schiller, Rainer Schwemmer, Barbara Sciascia, A. Solomin, F. Suljik, Nicola Skidmore, Michael D. Sokoloff, P. Spradlin, M. Stahl, S. Stahl, Holger Stevens, L. Sun, A. Szabelski, Tomasz Szumlak, M. Szymanski, D. Y. Tou, Giulia Tuci, Andrii Usachov, Núria Valls Canudas, R. Vazquez Gomez, Stefania Vecchi, Mika Vesterinen, Xavier Vilasís-Cardona, Dorothea Vom Bruch, Z. Wang, Tomasz Wojton, Mark Whitehead, Mike Williams, Mariusz Witek, Y. Xie, A. Xu, H. Yin, Milosz Zdybal, O. Zenaiev, D. Zhang, L. Zhang, X. Zhu, The LHCb Collaboration

Abstract

The LHCb experiment at CERN is undergoing an upgrade in preparation for the Run 3 data-taking period of the LHC. As part of this upgrade, the trigger is moving to a fully software implementation operating at the LHC bunch crossing rate. We present an evaluation of a CPU-based and a GPU-based implementation of the first stage of the High Level Trigger (HLT1). After a detailed comparison, both options are found to be viable. This document summarizes the performance and implementation details of these options, the outcome of which has led to the choice of the GPU-based implementation as the baseline.

Introduction

The Large Hadron Collider (LHC) produces a massive volume of data that requires sophisticated filtering in real-time. For the "Run 3" upgrade, the LHCb experiment set an ambitious goal: eliminate the traditional hardware-based trigger and move to a "triggerless" readout. This means the system must process the full 30 MHz bunch crossing rate—roughly 32 Terabits per second—entirely in software. The challenge is to perform real-time reconstruction, including tracking particles and finding collision vertices, with enough precision to identify interesting physics while staying within a realistic budget for power, cooling, and computing hardware.

System Architecture and Data Flow

The core of the upgrade lies in the transition from hardware-based filtering to a high-throughput software trigger. The data flow starts at the detector and moves through a series of processing stages designed to reduce the 32 Tb/s stream to a manageable level for permanent storage.

The LHCb Run 3 trigger architecture, illustrating the data flow from the detector to the HLT1 and HLT2 processing farms.

As illustrated in Figure 2, the architecture involves 173 Event Builder (EB) servers. In a traditional CPU-only setup, the full 32 Tb/s must be distributed over a massive network to a dedicated HLT1 farm. However, the hybrid approach integrates GPU cards directly into these Event Builder servers. By performing HLT1 processing "in-place" on GPUs, the system can reduce the required network bandwidth to the downstream HLT2 farm and disk buffers by a factor of 30 to 60. This architectural shift is a key driver for the efficiency of the GPU-based solution.

Methodology and Implementation

The researchers compared two distinct architectures for the HLT1 stage:

  1. CPU-only: Utilizing dual-socket servers equipped with 32-core AMD EPYC processors.
  2. Hybrid (GPU-based): Utilizing the "Allen" framework, with GPU cards installed directly within the Event Builder servers.

The HLT1 sequence involves several complex reconstruction steps, including Velo reconstruction (decoding signals from the Vertex Locator), Primary Vertex (PV) finding, Forward Tracking (matching tracks with downstream stations), Muon Identification, and Kalman Filtering. To ensure a "like-for-like" comparison, identical physics logic was used across both platforms, though the code was optimized for the specific strengths of each hardware type, such as the massive parallelism inherent in GPUs.

Physics Performance and Results

A significant outcome of the study was the demonstration of physics parity between the two architectures. Despite the complexity of non-uniform tracking algorithms, the GPU-based Allen framework achieved efficiency, resolution, and ghost rates nearly identical to the CPU implementation. Tracking efficiency agreed to within 1% between the two platforms across various signal samples, including $J/\psi \to \mu^+\mu^-$, $B_s^0 \to \phi\phi$, and $Z \to \mu^+\mu^-$.

In terms of throughput, a single GPU card could process HLT1 at approximately 92 kHz, while a dual-socket AMD EPYC node managed 171 kHz. However, the smaller footprint and lower power consumption per event made the GPU approach more cost-effective. The entire 30 MHz stream can be handled by a compact system of roughly 170 CPU servers or 330 GPU cards.

Economic and Operational Impact

The economic evaluation favored the hybrid GPU approach. In units of "Quanta" (a reference server cost), the CPU-only sub-total was 2300 units compared to 1725 units for the hybrid GPU approach—a saving of approximately 25%. Furthermore, the GPU baseline is considered more future-proof, as GPU performance is currently advancing at a faster rate than CPU single-core performance, making it more adaptable to future increases in LHC luminosity.

Conclusion

While both CPU and GPU implementations proved technically viable, the GPU-based HLT1 was selected as the baseline for LHCb Run 3. This decision was driven by the significant infrastructure cost savings, the reduction in networking complexity, and the promising performance trajectory of GPU hardware. Challenges remain, such as the maintenance of two distinct software stacks (CUDA/HIP for HLT1 and C++ for HLT2/Offline) and the ongoing task of porting more complex HLT2 algorithms to GPUs. Nevertheless, the success of the Allen framework in HLT1 marks a transformative step for real-time data processing at the LHC.

Summary updated

Abstract

The LHCb experiment at CERN is undergoing an upgrade in preparation for the Run 3 data taking period of the LHC. As part of this upgrade the trigger is moving to a fully software implementation operating at the LHC bunch crossing rate. We present an evaluation of a CPU-based and a GPU-based implementation of the first stage of the High Level Trigger. After a detailed comparison both options are found to be viable. This document summarizes the performance and implementation details of these options, the outcome of which has led to the choice of the GPU-based implementation as the baseline.
BibTeX
@article{aaij2022comparison,
  author = {Roel Aaij and Marta Adinolfi and Salvatore Aiola and S. Akar and Johannes Albrecht and M. Alexander and S. Amato and Yasmine Amhis and F. Archilli and M. Bala and G. Bassi and L. Bian and M. P. Blago and Thomas Boettcher and Alexey Boldyrev and S. Borghi and A. Brea Rodriguez and Lukas Calefice and M. Calvo Gomez and Daniel Hugo Cámpora Pérez and A. Cardini and Marco Cattaneo and V. Chobanova and G. Ciezarek and Xabier Cid Vidal and J. L. Cobbledick and J. A. B. Coelho and Tommaso Colombo and A. Contu and Benjamin Couturier and Daniel C. Craik and R. Currie and P. d'Argent and Michel De Cian and Denis Derkach and F. Dordei and Marco Dorigo and Laurent Dufour and Paolo Durante and A. Dziurda and A. Dzyuba and S. Easo and S. Esen and Placido Fernandez Declara and Sergey Filippov and Conor Fitzpatrick and M. Frank and Paolo Gandini and Vladimir V. Gligorov and Elisabet Golobardes and G. Graziani and L. Grillo and P. A. Günther and S. Hansmann-Menzemer and Arthur M. Hennequin and Louis Henry and D. Hill and S. E. Hollitt and J. Hu and W. Hulsbergen and R. J. Hunter and Mikhail Hushchyn and Brij Kishor Jashal and Christopher Robert Jones and S. Klaver and Konrad Klimaszewski and R. Kopecna and Wojciech Krzemien and Marcin Kucharczyk and R. Lane and Federico Lazzari and R. Le Gac and P. Li and J. H. Lopes and M. Lucio Martinez and Anna Lupato and Oliver Lupton and X. Lyu and F. Machefert and Olga Madejczyk and S. Malde and J. F. Marchand and Saverio Mariani and Carla Marin Benito and D. Martinez Santos and F. Martinez Vidal and Rosen Matev and Michal Mazurek and Biljana Mitreska and Dominik Stefan Mitzel and Michael Joseph Morello and H. Mu and P. Muzzetto and P. Naik and Matthew D. Needham and N. Neri and Niko Neufeld and N. S. Nolte and Daniel O'Hanlon and A. Oyanguren Campos and M. Pepe Altarelli and S. Petrucci and Marco Petruzzo and L. Pica and Flavio Pisani and A. Piucci and Francesco Polci and A. Poluektov and E. Polycarpo and C. Prouve and Giovanni Punzi and Renato Quagliani and R. I. Rabadan Trejo and Miguel Ramos Pernas and M. S. Rangel and Fedor Ratnikov and Gerhard Raven and F. Reiss and V. Renaudin and P. Robbe and Artem Ryzhikov and Marco Santimaria and M. Saur and Manuel T. Schiller and Rainer Schwemmer and Barbara Sciascia and A. Solomin and F. Suljik and Nicola Skidmore and Michael D. Sokoloff and P. Spradlin and M. Stahl and S. Stahl and Holger Stevens and L. Sun and A. Szabelski and Tomasz Szumlak and M. Szymanski and D. Y. Tou and Giulia Tuci and Andrii Usachov and Núria Valls Canudas and R. Vazquez Gomez and Stefania Vecchi and Mika Vesterinen and Xavier Vilasís-Cardona and Dorothea Vom Bruch and Z. Wang and Tomasz Wojton and Mark Whitehead and Mike Williams and Mariusz Witek and Y. Xie and A. Xu and H. Yin and Milosz Zdybal and O. Zenaiev and D. Zhang and L. Zhang and X. Zhu and The LHCb Collaboration},
  title = {{A Comparison of CPU and GPU Implementations for the LHCb Experiment Run 3 Trigger}},
  journal = {Comput. Softw. Big Sci.},
  year = {2022},
  doi = {10.1007/s41781-021-00070-2},
}
Figure