Particle Transformer for Jet Tagging

Huilin Qu, Congqiao Li, Sitian QianICML 2022

TL;DR

The authors introduce JETCLASS, a massive new dataset for particle physics containing 100 million simulated jets—two orders of magnitude larger than previous public benchmarks. To leverage this unprecedented scale of data, they propose the Particle Transformer (ParT), a novel architecture that incorporates physics-informed pairwise particle interactions directly into the Transformer's attention mechanism, setting a new state-of-the-art in jet tagging performance.

What Problem Does This Paper Address?

In high-energy physics experiments, such as those conducted at the CERN Large Hadron Collider (LHC), "jets" are collimated sprays of particles produced from high-energy collisions. Jet tagging is the critical classification task of identifying which fundamental particle (e.g., a Higgs boson, a top quark, or a gluon) initiated that specific spray.

Illustration of a collision event, jet reconstruction, and the jet tagging classification task.

While deep learning has revolutionized this field, two major bottlenecks have historically limited progress:

  1. Data Scarcity: Existing public datasets were relatively small (around 2 million jets), restricting the ability to train massive, modern neural network architectures effectively.
  2. Architectural Fit: Standard Transformers, while powerful in domains like NLP and vision, do not inherently "understand" the relativistic kinematics and pairwise relationships between particles that are fundamental to physics.

Methods: The Particle Transformer (ParT)

To address the architectural limitations of standard models, the authors propose the Particle Transformer (ParT). While it draws inspiration from the standard Transformer and the "Class Attention" mechanism (CaiT), it introduces a domain-specific modification called Particle Multi-Head Attention (P-MHA).

The Particle Transformer (ParT) architecture, highlighting the Particle Multi-Head Attention (P-MHA) mechanism.

Key features of the ParT architecture include:

  • Pairwise Interaction Features: Instead of solely relying on individual particle properties (like momentum or charge), ParT calculates four specific physical variables for every pair of particles in a jet (e.g., relative distance $\Delta$ and invariant mass $m^2$).
  • Attention Bias: These pairwise features are processed and then added as a bias matrix to the attention weights before the softmax operation. This effectively "tells" the model which particles are physically related before it even begins calculating self-attention.
  • Permutation Invariance: Because jets are unordered sets of particles, the model avoids traditional positional encodings, treating the jet instead as a permutation-invariant "particle cloud."
  • Class Token: Similar to Vision Transformers (ViTs), a global "class token" is used to aggregate information from all particles for the final classification output.

Datasets: Introducing JETCLASS

The core data contribution of the paper is JETCLASS, a new large-scale dataset designed to push the boundaries of jet tagging:

  • Scale: Contains 100 million jets for training, dwarfing the previous standard of ~2 million.
  • Diversity: Includes 10 different types of jets, covering several decay modes (like $H \to 4q$) that were previously unexplored in public tagging benchmarks.
  • Evaluation: Beyond JETCLASS, the authors also rigorously test ParT on established benchmarks, including the Top Quark Tagging dataset and the Quark-Gluon Tagging dataset.

Key Insights and Results

  • The Power of Scale: The paper demonstrates that jet tagging has not yet hit a point of "diminishing returns" with data. Moving from 2M to 100M jets provided a massive boost in performance across all evaluated models.
  • Physics as an Inductive Bias: Simply providing pairwise features as standard inputs isn't enough. By injecting them directly into the attention mechanism via P-MHA, the model learns much more efficiently. A "plain" Transformer (without P-MHA) performed significantly worse, even when given access to the same pairwise information.
  • Pre-training Works in Physics: Much like BERT or GPT in natural language processing, a ParT model pre-trained on the massive JETCLASS dataset can be fine-tuned on smaller, specific datasets to achieve results far superior to training from scratch.
  • State-of-the-Art Performance: On the JETCLASS dataset, ParT achieved an accuracy of 86.1%, significantly outperforming the previous state-of-the-art model, ParticleNet (84.4%).
  • Background Rejection: In experimental physics, "Background Rejection" (the ability to ignore "noise" jets while keeping "signal" jets) is often more critical than raw accuracy. ParT improved background rejection by a factor of 2 to 3 for several important particle types. On the Top Quark Tagging benchmark, fine-tuned ParT increased background rejection by 70% compared to previous bests.

Limitations and Future Work

  • Computational Complexity: The reliance on pairwise interactions means the computational cost scales quadratically ($O(N^2)$) with the number of particles ($N$). While jets typically contain a manageable 30–100 particles, this scaling could become a bottleneck for analyzing larger, full-collision "events" or more complex detector data.
  • Simulation vs. Reality: The JETCLASS dataset is based on Monte Carlo simulations. While the authors used sophisticated tools (DELPHES) to mimic real detector effects, there remains a "domain gap" between simulation and actual LHC data that requires further study.
  • Future Directions: The authors suggest that the JETCLASS dataset could serve as a foundation for broader applications, such as unsupervised learning or generative modeling of particle collisions.

Summary updated

BibTeX
@inproceedings{qu2022particle,
  author = {Huilin Qu and Congqiao Li and Sitian Qian},
  title = {{Particle Transformer for Jet Tagging}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
}
Figure