A Cautionary Tale of Decorrelating Theory Uncertainties

Aishik Ghosh, Benjamin NachmanarXiv 2021 · arXiv

Abstract

Machine learning classifiers in high-energy physics often rely on simulations with inherent theoretical uncertainties. A common mitigation strategy is "decorrelation"—forcing the model to be independent of the specific theoretical model used for training. This paper demonstrates that while decorrelation successfully reduces the estimated uncertainty, it often fails to capture the true underlying physics. This leads to overconfident results that may no longer cover actual experimental data, particularly when faced with models not seen during training.

Introduction

In the search for rare physical processes at the Large Hadron Collider (LHC), machine learning (ML) models are indispensable. However, these models are trained on simulations that incorporate various theoretical assumptions, such as fragmentation models or higher-order perturbative corrections. These assumptions introduce "theory uncertainties." To prevent models from learning features specific to a single simulator, researchers often use decorrelation techniques. The goal is to make the classifier's output invariant to the choice of theoretical model.

Adversarial Decorrelation

The authors focus on Adversarial Decorrelation, where a classifier and an adversary are trained simultaneously. The classifier attempts to distinguish signal from background, while the adversary tries to predict the theoretical model (the "nuisance parameter") from the classifier's output. By penalizing the classifier when the adversary succeeds, the model is forced to rely on features that are common across different simulations.

The study evaluates this approach using two primary scenarios:

  1. Two-point uncertainties: Comparing discrete models, such as the Pythia and Herwig event generators.
  2. Continuous uncertainties: Varying parameters like the "factorization scale" in perturbative calculations.

The Coverage Trap

A central finding of this work is the "Coverage Trap." Decorrelation effectively forces the model to produce consistent results across the training models. However, this consistency is often achieved by artificially shrinking the uncertainty band.

Conceptual illustration of the "coverage trap" where decorrelation shrinks the estimated uncertainty band, potentially excluding the true physical state ("Nature").

As illustrated above, if the real world ("Nature") lies outside the narrowed range of the training models, the decorrelated model becomes precisely wrong. Because theory uncertainties are often not statistical—they represent gaps in our knowledge rather than random noise—treating them as something to be "averaged out" is fundamentally risky.

Experimental Results: The Third-Model Test

The authors tested these concepts on a $W$ boson tagging task, distinguishing boosted $W$ bosons from generic QCD jets. They used Pythia 8 and Herwig 7 for adversarial training and introduced Sherpa 2 as an independent "third model" test.

While adversarial training reduced the performance gap between Pythia and Herwig (from ~40% to ~20%), the discrepancy with Sherpa remained significant. This indicates that the model did not learn a "universal" physical feature; instead, it merely learned to ignore the specific differences between the two training models.

QCD rejection vs. W efficiency (top) and associated uncertainty (bottom). Adversarial training (dotted lines) brings Pythia and Herwig closer but fails to account for the Sherpa model.

In the case of single top production, decorrelation reduced sensitivity to scale variations by half. However, it failed to bridge the gap between Leading Order (LO) and Next-to-Leading Order (NLO) calculations. The model became blind to the specific parameter it was trained on but remained vulnerable to the underlying physics it hadn't encountered.

Conclusion

Decorrelation remains a valuable tool for specific tasks like mass sculpting (preventing the creation of artificial "bumps" in data). However, for mitigating theory uncertainties, it should be used with extreme caution. Without a complete decomposition of uncertainties into statistically meaningful components, decorrelation may simply hide errors rather than removing them. Future research should focus on developing uncertainty-aware loss functions that explicitly account for model differences without misleadingly shrinking error bars.

Summary updated

BibTeX
@misc{ghosh2021cautionary,
  author = {Aishik Ghosh and Benjamin Nachman},
  title = {{A Cautionary Tale of Decorrelating Theory Uncertainties}},
  year = {2021},
  eprint = {2109.08159},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2109.08159},
}
Figure