Punzi-loss: – a non-differentiable metric approximation for sensitivity optimisation in the search for new particles

F. Abudinén, M. Bertemes, S. Bilokin, M. Campajola, G. Casarosa, S. Cunliffe, L. Corona, G. De Pietro, S. Dey, M. Eliachevitch, P. Feichtinger, T. Ferber, J. Gemmler, P. Goldenzweig, A. Gottmann, E. Graziani, H. Haigh, M. Hohmann, T. Humair, G. Inguglia, J. Kahn, Thomas M. Keck, I. Komarov, J. F. Krohn, T. Kuhr, S. Lacaprara, K. Lieret, R. Maiti, A. Martini, F. Meier, F. Metzner, M. Milesi, S. H. Park, M. T. Prim, C. Pulvermacher, M. Ritter, Yuki Sato, C. Schwanda, W. Sutcliffe, U. Tamponi, F. Tenchini, P. Urquijo, L. Zani, R. Žlebčík, A. Zupanc, Zupanc, A.PubMed 2022 · DOI

TL;DR

This paper introduces "Punzi-loss," a novel, non-differentiable loss function for neural networks designed to optimize searches for new particles in high-energy physics. By directly approximating a physics-based figure of merit (the Punzi FOM), Punzi-loss enables a single classifier to achieve superior sensitivity across a wide range of potential new particle masses, simplifying analysis and improving discovery potential.

What Problem Does This Paper Address?

High-energy physics experiments constantly search for new particles beyond the Standard Model. A major challenge is distinguishing rare "signal" events (indicating new physics) from abundant "background" events (explained by known physics). Traditional methods often rely on multivariate analysis (MVA) techniques, like boosted decision trees or neural networks, trained with standard loss functions (e.g., cross-entropy). However, these standard approaches may not optimally balance the trade-off between signal efficiency and background rejection, especially when searching for particles with a wide range of unknown masses. This often necessitates complex analysis strategies, such as training separate classifiers for different mass hypotheses or performing intricate cut optimizations, which can introduce discontinuities and reduce overall sensitivity. The paper aims to address this by developing a single, unified approach that directly optimizes a relevant physics figure of merit.

Methods

The core innovation is the Punzi-loss function, a differentiable approximation of the Punzi figure of merit (FOM). The Punzi FOM is a standard metric in particle physics used to define the sensitivity region for searches where the signal's existence and properties (like mass) are not precisely known beforehand.

The standard Punzi FOM is typically calculated based on discrete event counts and selection efficiencies, making it non-differentiable and thus unsuitable for direct use in neural network training via gradient descent. The authors overcome this by:

  • Approximating discrete counts with continuous functions: They replace the number of signal ($S$) and background ($B$) events passing a selection cut with weighted sums over all training events, where the weights are the neural network's output probabilities. This effectively turns the FOM calculation into a differentiable function of the network's weights and biases.
  • Loss Scheduling: The paper proposes a two-stage training strategy. First, the network is pre-trained using a standard Binary Cross-Entropy (BCE) loss. This initializes the network's weights and biases in a way that already provides some separation between signal and background. Second, the network is fine-tuned using the Punzi-loss function. This "loss scheduling" helps stabilize training and achieve better convergence with the non-differentiable Punzi-loss.
  • Punzi-net: A neural network trained with this Punzi-loss is termed a "Punzi-net."

Datasets

The study uses simulated data for a specific physics search: the search for invisible decays of a hypothetical Z' boson produced in $e^+e^-$ collisions at the Belle II experiment.

  • Signal: Simulated $e^+e^- \rightarrow \mu^+\mu^-Z'$ events were generated for a range of candidate Z' masses (0.1 GeV/c² to 8.9 GeV/c² in 0.1 GeV/c² steps).
  • Background: Simulated events for processes like $e^+e^- \rightarrow e^+e^-\mu^+\mu^-$, $e^+e^- \rightarrow \tau^+\tau^-$, and $e^+e^- \rightarrow \mu^+\mu^-\gamma$ were used.
  • Data Splitting: The dataset was split into training (80%) and validation (20%) sets. Importantly, the validation set included signal hypotheses (masses) that were not used during training to test the network's generalization capabilities.
  • Luminosity Scaling: The background data was scaled to match a target luminosity (50 fb⁻¹) that differed from the simulation's original luminosity, requiring specific scaling factors in the loss calculation.

Key Insights and Ideas

  • Directly optimizing physics metrics is powerful: Instead of relying on proxy loss functions like cross-entropy, directly optimizing a relevant figure of merit (Punzi FOM) leads to a more sensitive classifier for the specific physics goal.
  • A single classifier for all masses: The Punzi-net can provide a coherent and optimal classification across a wide range of signal hypotheses (Z' masses) without needing to train separate models or perform complex cut optimizations for each mass. This significantly simplifies the analysis pipeline.
  • Loss scheduling is crucial: Pre-training with a standard loss function is essential for stabilizing the training of the Punzi-loss, which is inherently more complex and potentially prone to instabilities.
  • Generalization is key: The Punzi-net demonstrates excellent generalization to signal masses it was not explicitly trained on, a critical requirement for new particle searches where the exact mass is unknown.
  • Batch size matters: The paper highlights that batch size can significantly impact the stability of Punzi-loss training, with very small or very large (mono-batch) sizes leading to instability. An optimal batch size is found to be crucial.

Key Results

  • Improved Sensitivity: The Punzi-net consistently achieves higher Punzi FOM values compared to a network pre-trained with BCE loss, particularly for lower Z' masses (below ~5 GeV/c²).
  • Comparable Performance at Higher Masses: Even in regions where the BCE-trained network shows slightly better peak FOM, the Punzi-net's ability to use a single, uniform cut (instead of mass-dependent cuts) provides a more practical and robust solution, avoiding discontinuities in the final analysis distribution.
  • Effective Generalization: The trained Punzi-net shows comparable performance for both signal masses used in training and those reserved for validation, demonstrating its ability to generalize across the search spectrum.
  • Simplified Analysis: The ability to use a single classifier and a single cut value for the entire search range simplifies the analysis workflow and reduces potential sources of error.

Limitations and Future Work

  • Training Instability: While mitigated by loss scheduling and careful batch size selection, training instabilities with Punzi-loss can still occur, especially with very small batch sizes.
  • Mass Hypothesis Dependence: While the method generalizes well, the performance can show some dependence on whether a specific mass hypothesis was included in the training set. The authors suggest that generating a denser grid of signal masses could further improve performance in such regions.
  • Generalizability to Other Searches: The paper focuses on a specific search ($e^+e^- \rightarrow \mu^+\mu^-Z'$). While the method is presented as general, further studies would be needed to confirm its effectiveness across a broader range of high-energy physics searches and different types of neural networks (e.g., CNNs, transformers).
  • Differentiable Frameworks: The authors note that fully differentiable analysis frameworks are an active area of research and could potentially incorporate systematic effects more directly into the optimization process. Future work could explore integrating Punzi-loss within such frameworks.

Summary updated

BibTeX
@inproceedings{abudinen2022punzi,
  author = {F. Abudinén and M. Bertemes and S. Bilokin and M. Campajola and G. Casarosa and S. Cunliffe and L. Corona and G. De Pietro and S. Dey and M. Eliachevitch and P. Feichtinger and T. Ferber and J. Gemmler and P. Goldenzweig and A. Gottmann and E. Graziani and H. Haigh and M. Hohmann and T. Humair and G. Inguglia and J. Kahn and Thomas M. Keck and I. Komarov and J. F. Krohn and T. Kuhr and S. Lacaprara and K. Lieret and R. Maiti and A. Martini and F. Meier and F. Metzner and M. Milesi and S. H. Park and M. T. Prim and C. Pulvermacher and M. Ritter and Yuki Sato and C. Schwanda and W. Sutcliffe and U. Tamponi and F. Tenchini and P. Urquijo and L. Zani and R. Žlebčík and A. Zupanc and Zupanc, A.},
  title = {{Punzi-loss: – a non-differentiable metric approximation for sensitivity optimisation in the search for new particles}},
  booktitle = {PubMed},
  year = {2022},
  doi = {10.5445/ir/1000143424},
}
Figure