Publishing unbinned differential cross section results

M. Arratia, Anja Butter, M. Campanelli, V. Croft, Dag Gillberg, Aishik Ghosh, Kristin Lohwasser, Bogdan Malaescu, V. M. Mikuni, Benjamin Nachman, Juan Rojo, Jesse Thaler, Ramon WinterhalderJournal of Instrumentation 2022 · DOI

TL;DR

This paper proposes a standardized format for publishing "unbinned" differential cross-section measurements in particle physics. This new approach, enabled by machine learning, allows for more flexible analysis and comparison of experimental data with theoretical predictions, moving beyond traditional binned histograms.

What Problem Does This Paper Address?

Traditionally, particle physics experiments present their results as binned histograms. This means that the choices of how to divide the data into bins (the "binning") are made before the analysis is complete. This pre-defined binning can limit the flexibility of how the data is used later, making it difficult to compare results from different experiments or to test new theoretical ideas that might require different binning schemes. Furthermore, it can be challenging to extract information about rare phenomena or the "tails" of distributions when data is aggregated into bins. Recent advances in machine learning have opened the door to analyzing data without pre-defined bins ("unbinned" analysis), but there's no standard way to share these results, hindering their widespread adoption and comparison.

Methods

The paper proposes a general framework for publishing unbinned data, designed to be agnostic to the specific machine learning method used for the unfolding (the process of correcting for detector effects). The core idea is to publish the sampled data points from the unbinned measurement, rather than a fitted function or a histogram. This approach is flexible and can accommodate various methods, including:

  • Density-Based Models: These methods learn a probability density function for the unbinned data. Examples include Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs).
  • Classifier-Based Models: These methods learn a likelihood ratio to reweight events, effectively mimicking the true distribution.

The proposed format involves two files: a submission file (similar to existing HEPData formats) and a data file for each measurement. The data file would contain the particle-level observables for each event, along with associated weights and uncertainties.

Datasets

The paper uses simulated data for the Z + jets production process as an example. The simulations are generated using PYTHIA 8.243 and HERWIG 7.1.5, with detector effects emulated by DELPHES 3.4.2. This simulated dataset is used to demonstrate the proposed format and compare two different unbinned unfolding methods: OMNIFOLD (a classifier-based approach) and a conditional invertible neural network (cINN, a density-based approach).

Key Insights and Ideas

  • Flexibility through Unbinned Data: Unbinned measurements allow for "inference-aware binning," meaning the optimal binning can be chosen after the data is published, tailored to specific analyses or theoretical models.
  • Post-Hoc Observable Definition: Unbinned data enables the definition of new observables after the measurement, which is difficult with traditional binned approaches.
  • Facilitating Comparisons: A standardized format for unbinned data will greatly simplify comparisons between different experiments and with theoretical predictions.
  • Machine Learning as an Enabler: Advances in machine learning are crucial for performing these unbinned measurements, especially in high dimensions.
  • Community Standard is Key: The paper emphasizes the need for community consensus to establish a standard for publishing unbinned results to maximize their scientific impact.

Key Results

The paper demonstrates the proposed format using simulated data for Z + jets production. It shows how both density-based (cINN) and classifier-based (OMNIFOLD) methods can produce unbinned results. The example illustrates how the proposed format can store event-level data, weights, and systematic uncertainties. The paper also shows how new observables (like the ratio of N-subjettiness variables $\tau_2/\tau_1$) can be easily derived from the unbinned output, a key advantage of this approach. The results from the two methods are shown to be comparable, though the paper notes that neither method was fully optimized in this example.

Limitations and Future Work

  • Storage Requirements: Storing unbinned data can require significant storage space (on the order of GBs), necessitating efficient hosting solutions like Zenodo and integration with tools like HDF5.
  • Tool Integration: Existing analysis tools (e.g., RooUnfold, Rivet) will need extensions to fully support unbinned data.
  • Community Adoption: The success of this proposal hinges on widespread community agreement and adoption.
  • Future Work: The authors suggest exploring the possibility of publishing parameterized observables that depend on an input parameter, which could lead to information loss depending on the unfolding method. They also highlight the need for further quantitative comparisons between different unbinned unfolding methods.

Summary updated

Abstract

Abstract Machine learning tools have empowered a qualitatively new way to perform differential cross section measurements whereby the data are unbinned, possibly in many dimensions. Unbinned measurements can enable, improve, or at least simplify comparisons between experiments and with theoretical predictions. Furthermore, many-dimensional measurements can be used to define observables after the measurement instead of before. There is currently no community standard for publishing unbinned data. While there are also essentially no measurements of this type public, unbinned measurements are expected in the near future given recent methodological advances. The purpose of this paper is to propose a scheme for presenting and using unbinned results, which can hopefully form the basis for a community standard to allow for integration into analysis workflows. This is foreseen to be the start of an evolving community dialogue, in order to accommodate future developments in this field that is rapidly evolving.
BibTeX
@article{arratia2022publishing,
  author = {M. Arratia and Anja Butter and M. Campanelli and V. Croft and Dag Gillberg and Aishik Ghosh and Kristin Lohwasser and Bogdan Malaescu and V. M. Mikuni and Benjamin Nachman and Juan Rojo and Jesse Thaler and Ramon Winterhalder},
  title = {{Publishing unbinned differential cross section results}},
  journal = {Journal of Instrumentation},
  year = {2022},
  doi = {10.1088/1748-0221/17/01/p01024},
}
Figure