Report on 2105.14027v2

T. K. Aarrestad, Melissa van Beekveld, M. Bóna, A. Boveia, S. Caron, Joe Davies, Antonio De Simone, C. Doglioni, J. Duarte, A. Farbin, Hoshin V. Gupta, Lizza Hendriks, L. Heinrich, J. Howarth, Pratik Jawahar, Adil Jueid, J. Lastow, Adam Leinweber, J. Mamuzic, Erzsébet Merényi, Alessandro Morandini, P. Moskvitina, C. Nellist, J. Ngadiuba, Bryan Ostdiek, M. Pierini, B. Ravina, Roberto Ruiz de Austri, S. Sekmen, Mary Touranakou, Marija Vaškeviciute, Ricardo Vilalta, Jean-Roch Vlimant, Rob Verheyen, M. J. White, Eric Wulff, Erik Jakob Wallin, K. W. Woźniak, Z. ZhangOpenalex Unknown 2021 · DOI · arXiv · OpenAlex

Abstract

We describe the outcome of a data challenge conducted as part of the Dark Machines initiative and the Les Houches 2019 workshop on Physics at TeV colliders. The challenge aims to detect signals of new physics at the Large Hadron Collider (LHC) using unsupervised machine learning algorithms. First, we propose how an anomaly score could be implemented to define model-independent signal regions in LHC searches. We define and describe a large benchmark dataset, consisting of > 1 billion simulated LHC events corresponding to 10 fb⁻¹ of proton-proton collisions at a center-of-mass energy of 13 TeV. We then review a wide range of anomaly detection and density estimation algorithms, developed in the context of the data challenge, and we measure their performance in a set of realistic analysis environments. We draw a number of useful conclusions that will aid the development of unsupervised new physics searches during the third run of the LHC.

What Problem Does This Paper Address?

The Standard Model of particle physics is incredibly successful, yet it remains incomplete, failing to explain phenomena like dark matter or neutrino masses. Traditionally, physicists search for "New Physics" (Beyond the Standard Model, or BSM) by testing specific theoretical models. However, if the universe contains physics that has not yet been theorized, these model-dependent searches will likely miss it.

This paper tackles the challenge of model-independent event classification. The goal is to develop unsupervised machine learning (ML) algorithms that can analyze Large Hadron Collider (LHC) data, recognize "normal" background processes, and flag "anomalous" events as potential hints of a new discovery.

Comparison between traditional 'expected' signal detection and model-independent 'unexpected' anomaly detection.

As illustrated above, traditional searches rely on predicting a specific signal's distribution (left), whereas anomaly detection uses an "anomaly score" to identify deviations from the predicted background without assuming a specific signal shape (right).

Methods and Algorithmic Approaches

The authors conducted a "data challenge" involving various research groups, resulting in a comparison of 16 distinct algorithmic approaches. Key techniques included:

  • Autoencoders (AEs) & Variational Autoencoders (VAEs): These models learn to compress and then reconstruct "normal" background data. Anomalies are identified by high reconstruction errors—the model cannot accurately rebuild an event it has never encountered before.
  • Normalizing Flows: Used for density estimation, these models learn the exact probability distribution of background data. Events falling into low-probability "voids" are flagged as anomalies.
  • Deep SVDD (Support Vector Data Description): A "fixed-target" approach where the network maps background data into a small sphere in a latent space. Outliers falling outside this sphere are flagged.
  • Hybrid Models: The most successful models often combined density estimation (Flows) with reconstruction or latent space analysis (VAEs/SVDD).
  • Deep Sets: This approach treats events as unordered collections of particles, respecting the permutation symmetry inherent in physics data.

Datasets

The study introduces a massive benchmark dataset, available at phenoMLdata.org:

  • The Hackathon Dataset: Over 1 billion simulated LHC events (10 fb⁻¹ equivalent) representing known Standard Model processes (background), including labeled BSM signals for validation.
  • The Secret Dataset: A blinded dataset containing a mixture of background and "hidden" new physics signals, used to test how well models generalize to truly unknown phenomena.
  • Channels: Data is split into four channels based on experimental signatures, such as events with varying numbers of jets or leptons.

Key Insights and Results

The primary metric used is Significance Improvement (SI), which measures how much the ML filter purifies the signal relative to background noise.

  • The "Fixed Target" Advantage: Models using a "fixed target" (like Deep SVDD or VAEs with $\beta=1$) generalized better to the secret dataset. These models focus on mapping data to a specific mathematical distribution rather than simple reconstruction.
  • Signal Difficulty: While high-energy events like gluino-gluino production are relatively easy to flag, signals with "small mass splittings" remain difficult because they closely resemble standard background processes.
  • Ensemble Power: Combining different anomaly scores (e.g., averaging a Flow score and a VAE score) consistently outperformed single-model approaches.
  • Top Performers: The "Flow-Efficient Likelihood" and combined "DeepSVDD-Flow" models were strongest, achieving median Total Improvement (TI) scores greater than 2.0. For certain signals like RPV SUSY, some models achieved significance improvements of over 100x.

Limitations and Future Work

While promising, these methods face challenges in real-world application. Simulations may not perfectly capture detector noise or systematic uncertainties, which an over-sensitive model might incorrectly flag as anomalies. Furthermore, defining a "signal region" based on an anomaly score requires rigorous statistical treatment to avoid the "Look-Elsewhere Effect" and false discoveries.

Future research will likely focus on Deep Sets and Graph Neural Networks to better handle the variable number of particles in LHC events and improve the validation of these "black box" detectors for real-time hardware triggers.

Summary updated

Abstract

We describe the outcome of a data challenge conducted as part of the Dark Machines (https://www.darkmachines.org) initiative and the Les Houches 2019 workshop on Physics at TeV colliders. The challenged aims to detect signals of new physics at the Large Hadron Collider (LHC) using unsupervised machine learning algorithms. First, we propose how an anomaly score could be implemented to define model-independent signal regions in LHC searches. We define and describe a large benchmark dataset, consisting of > 1 billion simulated LHC events corresponding to 10 fb -1 of proton-proton collisions at a center-of-mass energy of 13 TeV. We then review a wide range of anomaly detection and density estimation algorithms, developed in the context of the data challenge, and we measure their performance in a set of realistic analysis environments. We draw a number of useful conclusions that will aid the development of unsupervised new physics searches during the third run of the LHC, and provide our benchmark dataset for future studies at https://www.phenoMLdata.org.
BibTeX
@inproceedings{aarrestad2021report,
  author = {T. K. Aarrestad and Melissa van Beekveld and M. Bóna and A. Boveia and S. Caron and Joe Davies and Antonio De Simone and C. Doglioni and J. Duarte and A. Farbin and Hoshin V. Gupta and Lizza Hendriks and L. Heinrich and J. Howarth and Pratik Jawahar and Adil Jueid and J. Lastow and Adam Leinweber and J. Mamuzic and Erzsébet Merényi and Alessandro Morandini and P. Moskvitina and C. Nellist and J. Ngadiuba and Bryan Ostdiek and M. Pierini and B. Ravina and Roberto Ruiz de Austri and S. Sekmen and Mary Touranakou and Marija Vaškeviciute and Ricardo Vilalta and Jean-Roch Vlimant and Rob Verheyen and M. J. White and Eric Wulff and Erik Jakob Wallin and K. W. Woźniak and Z. Zhang and Javier Duarte and Jennifer Ngadiuba and Maurizio Pierini},
  title = {{Report on 2105.14027v2}},
  booktitle = {Openalex Unknown},
  year = {2021},
  doi = {10.21468/scipost.report.3866},
}
Figure