On the Evaluation of Generative Models in High Energy Physics

Raghav Kansal, Anni Li, Javier M. Duarte, Nadezda Chernyavskaya, Maurizio Pierini, Breno Orzari, Thiago TomeiarXiv 2022 · OpenReview

Introduction

In high-energy physics (HEP), simulations are the backbone of discovery. Traditionally, scientists use Monte Carlo (MC) methods like GEANT4 to simulate particle-detector interactions. However, the upcoming High-Luminosity LHC will produce data at a scale that traditional MC cannot computationally sustain. While machine learning generative models (like GANs and Transformers) offer a "fast simulator" alternative, the field lacks a reliable way to validate them. Standard HEP evaluation often relies on 1D histograms that miss complex correlations, while computer vision metrics like FID rely on features irrelevant to physics. This paper introduces two new metrics—the Fréchet Physics Distance (FPD) and Kernel Physics Distance (KPD)—to provide a standardized, physics-grounded evaluation framework.

Methodology

The authors frame evaluation as a two-sample goodness-of-fit test, comparing "real" MC samples to "generated" samples. Instead of using black-box neural network features, they utilize hand-engineered Energy Flow Polynomials (EFPs)—a complete basis for jet observables.

  1. Fréchet Physics Distance (FPD): Based on the Fréchet Inception Distance, it measures the distance between the mean and covariance of these physics features.
  2. Kernel Physics Distance (KPD): An adaptation of the Kernel Inception Distance, it uses a polynomial kernel to compare feature distributions without assuming they are Gaussian.

To benchmark these metrics, the researchers used synthetic 2D Gaussian mixtures to test responses to specific failures like mean shifts and mode collapse, alongside the realistic JetNet Dataset containing simulated gluon jets.

Samples of (mixtures of) Gaussian distributions used for testing evaluation metrics.

Key Results and Insights

The study demonstrates that physics-grounded features are more sensitive to distortions than black-box classifiers. FPD, in particular, proved to be the most sensitive metric for "distribution-level" errors, such as smearing or shifting the overall jet mass.

Jet mass distributions showing various distortions (smearing, shifting, tail removal) used to evaluate metric sensitivity.

Beyond metrics, the authors introduced the Generative Adversarial Particle Transformer (GAPT). This model utilizes self-attention to achieve inference speeds of 9 μs per jet, significantly faster than previous state-of-the-art models like MPGAN (41 μs per jet).

Conclusion

By providing these metrics in the open-source JetNet Python library, the authors aim to move HEP generative modeling toward a unified, interpretable benchmark. While FPD currently assumes Gaussian feature distributions, its sensitivity and grounding in physics make it a powerful tool for validating the next generation of fast simulators. Future work will involve optimizing feature sets for different detector types and extending these metrics to conditional modeling tasks.

Summary updated

BibTeX
@misc{kansal2022evaluation,
  author = {Raghav Kansal and Anni Li and Javier M. Duarte and Nadezda Chernyavskaya and Maurizio Pierini and Breno Orzari and Thiago Tomei},
  title = {{On the Evaluation of Generative Models in High Energy Physics}},
  year = {2022},
  url = {https://openreview.net/forum?id=HF1cD62H6n4},
}
Figure