Abstract
The Neyman-Pearson strategy for hypothesis testing can be employed for goodness of fit if the alternative hypothesis is generic enough not to introduce a significant bias while at the same time avoiding overfitting. A practical implementation of this idea (dubbed NPLM) has been developed in the context of high energy physics, targeting the detection in collider data of new physical effects not foreseen by the Standard Model. In this paper we initiate a comparison of this methodology with other approaches to goodness of fit, and in particular with classifier-based strategies that share strong similarities with NPLM. NPLM emerges from our comparison as more sensitive to small departures of the data from the expected distribution and not biased towards detecting specific types of anomalies while being blind to others. These features make it more suited for agnostic searches for new physics at collider experiments. Its deployment in other contexts should be investigated.
Introduction
In high-energy physics (HEP), researchers traditionally search for "New Physics" by looking for specific, predicted particles. However, this model-dependent approach risks missing phenomena that do not match our current predictions. This challenge calls for Goodness-of-Fit (GoF) tests—agnostic methods that ask whether data deviates from the expected Standard Model distribution without assuming a specific signal model. While machine learning classifiers have been proposed for this task, they often lack the sensitivity needed to spot subtle, localized signals buried under massive backgrounds. This paper evaluates the New Physics Learning Machine (NPLM), a methodology designed to bridge the gap between classical statistics and modern machine learning.
The NPLM Methodology
NPLM treats the GoF problem as a hypothesis test between a reference (the Standard Model) and a flexible, parameterized alternative. Unlike standard classifier-based anomaly detection that might use accuracy or AUC, NPLM utilizes a Likelihood Ratio test statistic. This choice naturally gives more weight to data points that are highly unlikely under the reference model, making it significantly more sensitive to small clusters of anomalous events.
A key technical innovation of NPLM is its in-sample evaluation. Most machine learning workflows require splitting data into training and testing sets to avoid overfitting. However, in HEP, where a signal might consist of only a handful of events, splitting the data can make the signal statistically invisible. NPLM uses the entire dataset for both training and evaluation. To prevent overfitting, the authors use "toy" simulations to establish the expected distribution of the test statistic, effectively calibrating the significance (p-value).
Benchmarking and Datasets
The authors employed three distinct benchmarks to stress-test the methodology:
- Expo (1D): A simple exponential distribution representing energy spectra, including five signal types such as peaks in the tail or bulk and data "defects."
- $μμ$ (5D): A realistic simulation of muon pairs produced at the Large Hadron Collider (LHC), including complex physical effects like $Z'$ bosons.
- DQM (Data Quality Monitoring): Real-world data from a muon detector at CERN, used to identify hardware failures like voltage drops.
Key Insights and Results
NPLM demonstrated several critical advantages over traditional classifier-based tests:
- Sensitivity to Localized Signals: In the 1D "Expo" benchmark, standard classifier tests were often "completely blind" to small peaks in the tail, while NPLM maintained strong detection capability. NPLM's focus on the magnitude of discrepancies, rather than simple classification accuracy, allows it to detect signals that represent only a tiny fraction of the total data.
- Uniform Sensitivity: One of the most striking findings is that NPLM is "uniformly" sensitive. While some traditional tests are optimized for specific distortions (like peaks) but blind to others (like broad deformations), NPLM performs well across a wide variety of anomaly types without needing to be retuned.
- The Advantage of Full Data Utilization: The paper confirms that train-test splitting is detrimental for GoF tests in this context. NPLM’s ability to use 100% of the data for the final test statistic is a major driver of its superior statistical power.
Conclusion
NPLM emerges as a powerful and robust tool for agnostic searches for new physics at collider experiments. While it is computationally intensive—requiring thousands of training runs on "toy" datasets to calculate p-values—its superior sensitivity and lack of bias make it highly suited for detecting unexpected physical phenomena. Its successful application to real-world Data Quality Monitoring at CERN further proves its utility beyond theoretical simulations, suggesting that its deployment in other scientific contexts should be actively investigated.
Summary updated