Abstract
The LHCb experiment at CERN is undergoing an upgrade in preparation for the Run 3 data-taking period of the LHC. As part of this upgrade, the trigger is moving to a fully software implementation operating at the LHC bunch crossing rate. We present an evaluation of a CPU-based and a GPU-based implementation of the first stage of the High Level Trigger (HLT1). After a detailed comparison, both options are found to be viable. This document summarizes the performance and implementation details of these options, the outcome of which has led to the choice of the GPU-based implementation as the baseline.
Introduction
The Large Hadron Collider (LHC) produces a massive volume of data that requires sophisticated filtering in real-time. For the "Run 3" upgrade, the LHCb experiment set an ambitious goal: eliminate the traditional hardware-based trigger and move to a "triggerless" readout. This means the system must process the full 30 MHz bunch crossing rate—roughly 32 Terabits per second—entirely in software. The challenge is to perform real-time reconstruction, including tracking particles and finding collision vertices, with enough precision to identify interesting physics while staying within a realistic budget for power, cooling, and computing hardware.
System Architecture and Data Flow
The core of the upgrade lies in the transition from hardware-based filtering to a high-throughput software trigger. The data flow starts at the detector and moves through a series of processing stages designed to reduce the 32 Tb/s stream to a manageable level for permanent storage.
As illustrated in Figure 2, the architecture involves 173 Event Builder (EB) servers. In a traditional CPU-only setup, the full 32 Tb/s must be distributed over a massive network to a dedicated HLT1 farm. However, the hybrid approach integrates GPU cards directly into these Event Builder servers. By performing HLT1 processing "in-place" on GPUs, the system can reduce the required network bandwidth to the downstream HLT2 farm and disk buffers by a factor of 30 to 60. This architectural shift is a key driver for the efficiency of the GPU-based solution.
Methodology and Implementation
The researchers compared two distinct architectures for the HLT1 stage:
- CPU-only: Utilizing dual-socket servers equipped with 32-core AMD EPYC processors.
- Hybrid (GPU-based): Utilizing the "Allen" framework, with GPU cards installed directly within the Event Builder servers.
The HLT1 sequence involves several complex reconstruction steps, including Velo reconstruction (decoding signals from the Vertex Locator), Primary Vertex (PV) finding, Forward Tracking (matching tracks with downstream stations), Muon Identification, and Kalman Filtering. To ensure a "like-for-like" comparison, identical physics logic was used across both platforms, though the code was optimized for the specific strengths of each hardware type, such as the massive parallelism inherent in GPUs.
Physics Performance and Results
A significant outcome of the study was the demonstration of physics parity between the two architectures. Despite the complexity of non-uniform tracking algorithms, the GPU-based Allen framework achieved efficiency, resolution, and ghost rates nearly identical to the CPU implementation. Tracking efficiency agreed to within 1% between the two platforms across various signal samples, including $J/\psi \to \mu^+\mu^-$, $B_s^0 \to \phi\phi$, and $Z \to \mu^+\mu^-$.
In terms of throughput, a single GPU card could process HLT1 at approximately 92 kHz, while a dual-socket AMD EPYC node managed 171 kHz. However, the smaller footprint and lower power consumption per event made the GPU approach more cost-effective. The entire 30 MHz stream can be handled by a compact system of roughly 170 CPU servers or 330 GPU cards.
Economic and Operational Impact
The economic evaluation favored the hybrid GPU approach. In units of "Quanta" (a reference server cost), the CPU-only sub-total was 2300 units compared to 1725 units for the hybrid GPU approach—a saving of approximately 25%. Furthermore, the GPU baseline is considered more future-proof, as GPU performance is currently advancing at a faster rate than CPU single-core performance, making it more adaptable to future increases in LHC luminosity.
Conclusion
While both CPU and GPU implementations proved technically viable, the GPU-based HLT1 was selected as the baseline for LHCb Run 3. This decision was driven by the significant infrastructure cost savings, the reduction in networking complexity, and the promising performance trajectory of GPU hardware. Challenges remain, such as the maintenance of two distinct software stacks (CUDA/HIP for HLT1 and C++ for HLT2/Offline) and the ongoing task of porting more complex HLT2 algorithms to GPUs. Nevertheless, the success of the Allen framework in HLT1 marks a transformative step for real-time data processing at the LHC.
Summary updated