CaloChallenge 2022: A Community Challenge for Fast Calorimeter Simulation

Claudius Krause, Michele Faucci Giannelli, Gregor Kasieczka, Benjamin Nachman, Dalila Salamani, David Shih, Anna Zaborowska, Oz Amram, Kerstin Borras, Matthew R. Buckley, Erik Buhmann, Thorsten Buss, Renato Paulo Da Costa Cardoso, Anthony L. Caterini, Nadezda Chernyavskaya, Federico A. G. Corchia, Jesse C. Cresswell, Sascha Diefenbacher, Etienne Dreyer, Vijay Ekambaram, Engin Eren, Florian Ernst, Luigi Favaro, Matteo Franchini, Frank Gaede, Eilam Gross, Shih-Chieh Hsu, Kristina Jaruskova, Benno Käch, Jayant Kalagnanam, Raghav Kansal, Taewoo Kim, Dmitrii Kobylianskii, Anatolii Korol, William Korcari, Dirk Krücker, Katja Krüger, Marco Letizia, Shu Li, Qibin Liu, Xiulong Liu, Gabriel Loaiza-Ganem, Thandikire Madula, Peter McKeown, Isabell-A. Melzer-Pellmann, Vinicius Mikuni, Nam Nguyen, Ayodele Ore, Sofia Palacios Schweitzer, Ian Pang, Kevin Pedro, Tilman Plehn, Witold Pokorski, Huilin Qu, Piyush Raikwar, John A. Raine, Humberto Reyes-Gonzalez, Lorenzo Rinaldi, Brendan Leigh Ross, Moritz A. W. Scham, Simon Schnake, Chase Shimmin, Eli Shlizerman, Nathalie Soybelman, Mudhakar Srivatsa, Kalliopi Tsolaki, Sofia Vallecorsa, Kyongmin Yeo, Rui Zhang

TL;DR

The CaloChallenge 2022 was a massive community effort to benchmark 31 different generative AI models—including GANs, Diffusion models, and Normalizing Flows—for simulating particle showers in high-energy physics. The results reveal a critical "Pareto front" where researchers must trade off between physics fidelity, generation speed, and model memory footprint to meet the needs of future particle colliders.

What Problem Does This Paper Address?

In the world of experimental particle physics, simulation is the bridge between theory and discovery. Currently, the Large Hadron Collider (LHC) relies on Geant4, a "gold standard" toolkit that simulates particle interactions with matter using first principles. However, Geant4 is computationally expensive. As we move toward the High-Luminosity LHC (HL-LHC) era, the volume of data will explode, and the CPU resources required for traditional simulation are projected to overwhelm existing computing budgets.

Figure 1: Projected CPU requirements. Left: For ATLAS [1]. Right: For CMS [2].

The "Fast Calorimeter Simulation Challenge 2022" (CaloChallenge) was designed to see if Deep Generative Models (DGMs) could replace the slowest parts of this chain—the calorimeter simulations—by learning to "generate" particle showers that are indistinguishable from Geant4 but thousands of times faster.

Methods

The challenge saw 31 individual submissions spanning five major generative architectures:

  • Generative Adversarial Networks (GANs): Known for speed, these train two networks (a generator and a discriminator) in a competitive game.
  • Normalizing Flows (NFs): These use invertible transformations to map simple distributions to complex data, allowing for exact likelihood evaluation.
  • Diffusion Models: A newer class of models that learn to reverse a "noise" process to generate data, often achieving the highest fidelity.
  • Variational Autoencoders (VAEs): These compress data into a latent space and then reconstruct it.
  • Conditional Flow Matching (CFM): A state-of-the-art approach that learns the vector field transporting a simple prior to the data distribution.
Table 1: Models submitted to the CaloChallange.

The challenge evaluated these models not just on how "pretty" the generated showers looked, but on rigorous physics metrics: 1D histograms of energy depositions, correlations between detector layers, and the ability of a "classifier" neural network to tell the difference between AI-generated and real Geant4 data.

Datasets

The challenge provided four datasets of increasing complexity, defined by their dimensionality (the number of "voxels" or readout cells):

  1. Dataset 1 (Photons & Pions): Based on ATLAS open data, featuring ~368-533 voxels.
  2. Dataset 2 (Electrons): An idealized geometry with 6,480 voxels.
  3. Dataset 3 (Electrons): The "hard" mode, featuring a high-granularity geometry with 40,500 voxels.

Key Insights and Ideas

  • No "One Model to Rule Them All": The most significant "aha" moment was the clear trade-off between quality and speed. Diffusion models and Normalizing Flows consistently produced the most accurate physics, but they were often the slowest. GANs and VAEs were lightning-fast but struggled with fine-grained correlations.
  • The Dimensionality Wall: Models that performed perfectly on Dataset 1 often failed to scale to Dataset 3. High-granularity detectors (40k+ voxels) introduce extreme sparsity and complex correlations that require specialized architectures like Transformers or Point Clouds.
  • Evaluation is a Multi-Dimensional Problem: The authors found that standard computer science metrics (like Frechet Physics Distance) are useful but not sufficient. You need a "classifier-based" metric—essentially an AI "critic"—to truly judge if the simulation is high-fidelity enough for real physics analysis.

Key Results

  • Speedup: Generative models achieved generation times ranging from sub-millisecond to a few milliseconds per shower on GPUs. Compared to Geant4, which can take seconds per event, this represents a speedup of several orders of magnitude.
  • Fidelity: In the lower-dimensional datasets, the best models (like CaloDiffusion and CaloINN) produced distributions that were nearly indistinguishable from the reference Geant4 data.
  • Resource Efficiency: Some models, like MDMA, managed to achieve decent results with an order of magnitude fewer parameters than their competitors, making them much easier to deploy in production environments with limited memory.

Limitations and Future Work

While the CaloChallenge proved that generative AI is ready for prime time, several hurdles remain:

  1. Integration: These models were tested as standalone tools. The next step is integrating them into the full, complex software frameworks used by experiments like ATLAS and CMS.
  2. Generalization: Most models were trained on fixed incident positions. Real-world simulations need to handle particles hitting the detector at any angle or location.
  3. The "Out of Memory" Problem: For the 40,500-voxel dataset, many models ran into GPU memory limits during training or inference, suggesting that future research needs to focus on more memory-efficient architectures like graph neural networks or point-cloud-based diffusion.

The CaloChallenge has provided the most comprehensive survey of generative AI in physics to date, establishing a benchmark that will guide the development of fast simulation for years to come.

Summary updated

Abstract

We present the results of the "Fast Calorimeter Simulation Challenge 2022" - the CaloChallenge. We study state-of-the-art generative models on four calorimeter shower datasets of increasing dimensionality, ranging from a few hundred voxels to a few tens of thousand voxels. The 31 individual submissions span a wide range of current popular generative architectures, including Variational AutoEncoders (VAEs), Generative Adversarial Networks (GANs), Normalizing Flows, Diffusion models, and models based on Conditional Flow Matching. We compare all submissions in terms of quality of generated calorimeter showers, as well as shower generation time and model size. To assess the quality we use a broad range of different metrics including differences in 1-dimensional histograms of observables, KPD/FPD scores, AUCs of binary classifiers, and the log-posterior of a multiclass classifier. The results of the CaloChallenge provide the most complete and comprehensive survey of cutting-edge approaches to calorimeter fast simulation to date. In addition, our work provides a uniquely detailed perspective on the important problem of how to evaluate generative models. As such, the results presented here should be applicable for other domains that use generative AI and require fast and faithful generation of samples in a large phase space.
BibTeX
@misc{krause2024calochallenge,
  author = {Claudius Krause and Michele Faucci Giannelli and Gregor Kasieczka and Benjamin Nachman and Dalila Salamani and David Shih and Anna Zaborowska and Oz Amram and Kerstin Borras and Matthew R. Buckley and Erik Buhmann and Thorsten Buss and Renato Paulo Da Costa Cardoso and Anthony L. Caterini and Nadezda Chernyavskaya and Federico A. G. Corchia and Jesse C. Cresswell and Sascha Diefenbacher and Etienne Dreyer and Vijay Ekambaram and Engin Eren and Florian Ernst and Luigi Favaro and Matteo Franchini and Frank Gaede and Eilam Gross and Shih-Chieh Hsu and Kristina Jaruskova and Benno Käch and Jayant Kalagnanam and Raghav Kansal and Taewoo Kim and Dmitrii Kobylianskii and Anatolii Korol and William Korcari and Dirk Krücker and Katja Krüger and Marco Letizia and Shu Li and Qibin Liu and Xiulong Liu and Gabriel Loaiza-Ganem and Thandikire Madula and Peter McKeown and Isabell-A. Melzer-Pellmann and Vinicius Mikuni and Nam Nguyen and Ayodele Ore and Sofia Palacios Schweitzer and Ian Pang and Kevin Pedro and Tilman Plehn and Witold Pokorski and Huilin Qu and Piyush Raikwar and John A. Raine and Humberto Reyes-Gonzalez and Lorenzo Rinaldi and Brendan Leigh Ross and Moritz A. W. Scham and Simon Schnake and Chase Shimmin and Eli Shlizerman and Nathalie Soybelman and Mudhakar Srivatsa and Kalliopi Tsolaki and Sofia Vallecorsa and Kyongmin Yeo and Rui Zhang},
  title = {{CaloChallenge 2022: A Community Challenge for Fast Calorimeter Simulation}},
  year = {2024},
  eprint = {2410.21611},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.21611},
}
Figure