Abstract
We describe the outcome of a data challenge conducted as part of the Dark Machines initiative and the Les Houches 2019 workshop on Physics at TeV colliders. The challenge aims to detect signals of new physics at the Large Hadron Collider (LHC) using unsupervised machine learning algorithms. First, we propose how an anomaly score could be implemented to define model-independent signal regions in LHC searches. We define and describe a large benchmark dataset, consisting of > 1 billion simulated LHC events corresponding to 10 fb⁻¹ of proton-proton collisions at a center-of-mass energy of 13 TeV. We then review a wide range of anomaly detection and density estimation algorithms, developed in the context of the data challenge, and we measure their performance in a set of realistic analysis environments. We draw a number of useful conclusions that will aid the development of unsupervised new physics searches during the third run of the LHC.
What Problem Does This Paper Address?
The Standard Model of particle physics is incredibly successful, yet it remains incomplete, failing to explain phenomena like dark matter or neutrino masses. Traditionally, physicists search for "New Physics" (Beyond the Standard Model, or BSM) by testing specific theoretical models. However, if the universe contains physics that has not yet been theorized, these model-dependent searches will likely miss it.
This paper tackles the challenge of model-independent event classification. The goal is to develop unsupervised machine learning (ML) algorithms that can analyze Large Hadron Collider (LHC) data, recognize "normal" background processes, and flag "anomalous" events as potential hints of a new discovery.
As illustrated above, traditional searches rely on predicting a specific signal's distribution (left), whereas anomaly detection uses an "anomaly score" to identify deviations from the predicted background without assuming a specific signal shape (right).
Methods and Algorithmic Approaches
The authors conducted a "data challenge" involving various research groups, resulting in a comparison of 16 distinct algorithmic approaches. Key techniques included:
- Autoencoders (AEs) & Variational Autoencoders (VAEs): These models learn to compress and then reconstruct "normal" background data. Anomalies are identified by high reconstruction errors—the model cannot accurately rebuild an event it has never encountered before.
- Normalizing Flows: Used for density estimation, these models learn the exact probability distribution of background data. Events falling into low-probability "voids" are flagged as anomalies.
- Deep SVDD (Support Vector Data Description): A "fixed-target" approach where the network maps background data into a small sphere in a latent space. Outliers falling outside this sphere are flagged.
- Hybrid Models: The most successful models often combined density estimation (Flows) with reconstruction or latent space analysis (VAEs/SVDD).
- Deep Sets: This approach treats events as unordered collections of particles, respecting the permutation symmetry inherent in physics data.
Datasets
The study introduces a massive benchmark dataset, available at phenoMLdata.org:
- The Hackathon Dataset: Over 1 billion simulated LHC events (10 fb⁻¹ equivalent) representing known Standard Model processes (background), including labeled BSM signals for validation.
- The Secret Dataset: A blinded dataset containing a mixture of background and "hidden" new physics signals, used to test how well models generalize to truly unknown phenomena.
- Channels: Data is split into four channels based on experimental signatures, such as events with varying numbers of jets or leptons.
Key Insights and Results
The primary metric used is Significance Improvement (SI), which measures how much the ML filter purifies the signal relative to background noise.
- The "Fixed Target" Advantage: Models using a "fixed target" (like Deep SVDD or VAEs with $\beta=1$) generalized better to the secret dataset. These models focus on mapping data to a specific mathematical distribution rather than simple reconstruction.
- Signal Difficulty: While high-energy events like gluino-gluino production are relatively easy to flag, signals with "small mass splittings" remain difficult because they closely resemble standard background processes.
- Ensemble Power: Combining different anomaly scores (e.g., averaging a Flow score and a VAE score) consistently outperformed single-model approaches.
- Top Performers: The "Flow-Efficient Likelihood" and combined "DeepSVDD-Flow" models were strongest, achieving median Total Improvement (TI) scores greater than 2.0. For certain signals like RPV SUSY, some models achieved significance improvements of over 100x.
Limitations and Future Work
While promising, these methods face challenges in real-world application. Simulations may not perfectly capture detector noise or systematic uncertainties, which an over-sensitive model might incorrectly flag as anomalies. Furthermore, defining a "signal region" based on an anomaly score requires rigorous statistical treatment to avoid the "Look-Elsewhere Effect" and false discoveries.
Future research will likely focus on Deep Sets and Graph Neural Networks to better handle the variable number of particles in LHC events and improve the validation of these "black box" detectors for real-time hardware triggers.
Summary updated