TL;DR
This paper introduces RINO, a self-supervised learning method that pretrains machine learning models on unlabeled real-world particle collision data. By learning representations invariant to energy scales, RINO significantly improves how well models trained on simulated data generalize to real data, a crucial step for robust analysis in high-energy physics.
What Problem Does This Paper Address?
High-energy physics (HEP) experiments generate vast amounts of data, but training machine learning models for analysis typically relies on simulated data. These simulations, while useful, often don't perfectly capture the complexities of real-world detector responses and physical processes, leading to a "domain shift." This mismatch can degrade model performance when applied to actual experimental data. A major hurdle is that real collision data lacks the precise labels needed for standard supervised training. This paper tackles the challenge of leveraging abundant, unlabeled real collision data to create more robust ML models that bridge the simulation-data gap.
Methods
The core innovation is RINO (Renormalization Group Invariance with NO labels), a self-supervised learning (SSL) approach inspired by DINO. RINO uses a teacher-student architecture where the student model learns to predict representations of the teacher model's output. The key idea is to create different "views" of the same jet by applying a physics-informed augmentation: using the $\kappa_T$ clustering algorithm. This algorithm iteratively groups particles based on their transverse momentum, effectively allowing the model to see jets at different "energy scales" or levels of detail. By learning representations that are invariant across these different views, RINO encourages the model to capture fundamental jet properties rather than simulation-specific artifacts. The pretraining is done on real-like QCD jets, and then the model is fine-tuned on a specific task (top jet tagging) using simulated data.
Datasets
- JETCLASS: Used for pretraining RINO. It's described as emulating real QCD-dominated experimental data.
- JETNET: Used for fine-tuning on the top jet tagging task. It emulates simulation data.
The paper uses these two datasets to simulate the common HEP scenario of training on simulations and evaluating on real data, or vice-versa, to study domain shift.
Key Insights and Ideas
- Physics-informed augmentation is crucial: Unlike standard image augmentations (like cropping or color jitter), RINO uses the $\kappa_T$ clustering history to create physically meaningful variations of jet data. This respects the inherent structure of particle physics data.
- Renormalization Group Invariance as an inductive bias: The paper leverages the concept of renormalization group flow in quantum chromodynamics. By learning representations invariant to energy scales (approximated by $\kappa_T$ clustering depth), RINO introduces a powerful inductive bias that helps models generalize better.
- Pretraining on "real-like" data improves generalization: Pretraining on JETCLASS (emulating real data) and then fine-tuning on JETNET (simulated data) leads to significantly better performance on JETCLASS compared to a purely supervised approach trained only on JETNET. This highlights the value of incorporating real-world data characteristics early in the training process.
- SSL can reduce reliance on simulations: RINO demonstrates that it's possible to achieve robust performance by pretraining on unlabeled real data and then fine-tuning on smaller, high-quality simulated datasets, potentially reducing the burden of extensive simulation generation.
Key Results
- Improved cross-domain generalization: RINO consistently outperforms supervised baselines on the JETCLASS dataset (emulating real data) by a significant margin, with accuracy gains ranging from 15% to 26%.
- Competitive in-domain performance: On the JETNET dataset (emulating simulations), RINO achieves performance gaps of only 4% to 5% compared to supervised baselines, indicating a favorable trade-off for the substantial cross-domain benefits.
- Rich embeddings: The pretrained embeddings from RINO show good separation between top and QCD jets, as visualized by PCA and t-SNE, and achieve high accuracies when used to train Boosted Decision Trees (BDTs).
Limitations and Future Work
- Emulating real data: JETCLASS is described as emulating real data. While this is a practical approach, it's not actual real data, and future work could explore pretraining directly on truly unlabeled experimental datasets.
- Fine-tuning strategies: The paper notes that aggressive fine-tuning can overfit. They propose an LP-FT (Linear Probing then Fine-Tuning) strategy, but suggest that future work will focus on developing better fine-tuning methods to fully leverage the rich embeddings learned by RINO.
- Broader impact: The authors believe RINO has potential for broader impact in fundamental physics by reducing biases from simulation-only training. Future work could explore more rigorous metrics to quantify this robustness.
Summary updated