Blind Tail Audits for Background-Rejection Classifiers

Rare-signal searches in particle physics depend on what a classifier does at the far edge of its operating range. A tagger is useful when it lets through a tiny fraction of background, one jet in ten thousand or fewer, while keeping a fair share of the signal. Yet the models are ranked by accuracy or area under the curve on a clean test set that the authors evaluate themselves. We propose a centralized, blind evaluation service that measures tail rejection, meaning how much background survives at a fixed signal efficiency, under small hidden distortions of the input particles. The distortions are tuned so that standard distribution-comparison tests cannot tell them apart from clean data. The question is whether such "invisible" distortions reorder the leaderboard of jet taggers. If they do, the community is certifying models with a check that misses exactly the effect that matters.

The idea joins two lines of work. Automated evaluation infrastructure, where submitted code runs on held-out data under controlled conditions, has a long track record outside physics. Signal-to-background discrimination has strong deep models and well-known difficulties with mismatch between simulation and real detectors. The new element is a specific measurement that only a blind, centralized evaluator can make credibly: rejection in the extreme tail, with honest error bars, under distortions that the model developer never sees and that pass the usual validation tests.

Core Claim

We claim that bulk metrics and clean-data tail rejection do not predict a model's tail rejection under realistic, small, unannounced changes to particle-level inputs. We also claim these changes can be chosen so that fixed-kernel two-sample tests have essentially no power to detect them. The second half rests on a measured negative result. Two-Sample Testing for Monte Carlo Evaluation in Particle Physics built MCBench and showed that Maximum Mean Discrepancy and Sliced Wasserstein tests on JetNet gluon jets reach power 1.000 for mean and variance shifts at $\epsilon \ge 0.2$. For additive noise smearing, the same tests stay near the 0.05 false-alarm baseline until $\epsilon = 0.5$. That paper motivates this proposal. It shows what validation misses, but it never asks whether a classifier's rejection is hurt by what is missed. We ask that question directly. A positive answer turns a statistics finding into a concrete warning about how taggers are ranked.

What Exists and How This Differs

The closest work is FAIR Universe HiggsML Uncertainty Dataset and Competition. It runs a standardized competition on 280 million tabular collision events, holds 120 million events private, and scores signal-strength estimates and confidence intervals under six explicitly parameterized nuisance parameters. It supplies the evaluation model we reuse: a private test set and server-side scoring. It differs from our proposal in three ways. Its inputs are 28 summary variables rather than particle clouds. Its distortions are announced and parametric. Its target is interval calibration rather than tail rejection.

JetNet supplies the data access and physics metrics we build on, including Fréchet and kernel physics distances and Wasserstein distances on jet observables. JetNet runs on the user's own machine, though, and has no blind or held-out scoring. MCBench, described above, supplies the two-sample tests we use as the "invisible" criterion.

The classifiers under audit come from three papers. Particle Transformer for Jet Tagging adds pairwise kinematic terms to attention and reports clean-data gains. Searching for exotic particles with deep learning and Deep extreme feature extraction disagree about whether engineered high-level features help. Our audit puts that disagreement on firmer ground. Models that lean on engineered or pairwise physics features may be more or less fragile in the tail than models that learn from raw constituents.

On the infrastructure side, Dynaboard and the SEALS-based OAEI 2010 campaign show that running submitted code on uniform hardware removes self-reporting and environment effects. Neither targets rare-event statistics. The inference-as-a-service systems such as FPGA-Accelerated Machine Learning Inference as a Service speed up production inference and do not score competing models.

Research Program

The service accepts a container holding a trained tagger with a fixed input interface: particle clouds in the JetClass and JetNet formats. It runs the tagger on a private pool of simulated jets on fixed hardware. It applies a set of hidden distortion families to the particle features. These include heavy-tailed per-particle momentum smearing, angular resolution degradation, random removal of soft constituents, and injection of low-momentum pileup-like particles.

For each family, we set the strength using MCBench at the evaluation sample size. We choose the largest strength at which fixed-kernel MMD and SWD on JetNet observables still have power at or below 0.1. By construction, these distortions would pass standard simulation validation.

The main metric is background rejection $R = 1/\varepsilon_B$ at fixed signal efficiency $\varepsilon_S$, reported at a standard working point (30%) and a tighter one. Here $\varepsilon_B$ is the fraction of background passing the cut set on clean signal. If $k$ background jets pass out of $n_B$, then $R$ has relative error of about $1/\sqrt{k}$, because the pass count is binomial with small rate. This fixes how large the private pool must be:

$$n_B \gtrsim \frac{k_{\min}}{\varepsilon_B},$$

where $k_{\min}$ is the minimum number of passing background jets needed for the target precision. For example, $k_{\min}=100$ gives 10% relative error. This bound is the reason a large private pool is needed. Small public test splits cannot resolve rejection at $\varepsilon_B \sim 10^{-4}$ or below. We report Clopper–Pearson intervals throughout. We also report a rank-stability score: Kendall's $\tau$ between the clean and distorted tail-rejection rankings, with a bootstrap interval over jets.

Baselines

The naive reference point is current practice. Each model's tail rejection is measured on the clean test set and the models are ranked by it. Whatever ordering this produces is the "do nothing" leaderboard. Our claim is beaten if the distorted ranking matches it within bootstrap error.

The strongest existing model is the Particle Transformer. The cited paper reports 86.1% overall accuracy on JetClass against 84.4% for ParticleNet, background rejection better by a factor of 2 to 3 for several classes, and a 70% increase in rejection on the top-tagging benchmark after fine-tuning. We retrain ParT and ParticleNet with their published configurations on the same JetClass training split, with the same GPU-hour budget and the same number of tuning trials. We first confirm that our clean numbers fall within the reported range before auditing. ParT "wins" our audit if its tail-rejection advantage survives every hidden family. It "loses" if a simpler model overtakes it under distortion, with non-overlapping intervals.

The weakest serious competitor is a boosted decision tree on the four JetNet jet-level features (transverse momentum, pseudorapidity, mass, multiplicity), together with a deep-sets model on constituents. We include them because summary features may average out per-particle smearing. If so, the simple model would be the more robust one. That possibility is the sharpest test of whether more expressive taggers buy fragility.

The key ablation isolates hidden-ness. Each model is also trained with augmentation from a publicly announced subset of distortion families. We then compare its tail rejection on announced versus hidden families. A gap between the two shows what the blind evaluator adds. No gap means it adds nothing beyond a published test suite.

Why not just…?

Why not just evaluate locally with bootstrap error bars on a public test set? For clean-data metrics this is adequate, and we do not claim otherwise. It fails on two points that our experiments check directly. First, the bound on $n_B$ requires pools larger than public splits for $\varepsilon_B$ well below $10^{-3}$. Second, any public distortion set becomes a training target. The announced-versus-hidden ablation measures exactly how much this matters. If the gap is within error, we will report that a published test suite is enough.

Why not just use FAIR Universe-style parameterized nuisance parameters? Those cover "known unknowns" whose form the modeler can write down and profile over. Our families are chosen to be invisible to standard tests and are never parameterized for the submitter. Applying a parameterized jet-energy-scale shift alongside our families will show whether models robust to the known shift remain robust to the hidden ones. If they do, the parametric approach is sufficient.

Why not just fix the validation test with a learned or tail-weighted kernel, so the distortions become visible? This is a real alternative and a complementary one. MCBench itself points to learned kernels as open work. Better tests, however, only tell you that simulation differs from data. They do not tell you which classifier's ranking changes. We will run a classifier-based two-sample test as a check. If it detects every family at our chosen strengths, the "invisible" criterion moves to that stronger test, and the audit question remains.

Proposed Experiments and Decisive Checks

The experiments use JetClass for training and the private pool, the top-quark tagging benchmark for comparison with ParT's reported 70% gain, and JetNet gluon jets for calibrating distortion strength with MCBench. No reported result was found on Lacuna for tail rejection of any tagger under distortions, so all distorted numbers are new measurements.

The first decisive check asks whether at least one distortion family exists that has fixed-kernel MMD and SWD power at or below 0.1 yet changes ParT's tail rejection by more than its 95% interval. A negative answer across all families would largely falsify the core claim. The second check asks whether Kendall's $\tau$ between the clean and distorted tail rankings drops clearly below the $\tau$ between two independent clean resamples. The third is the announced-versus-hidden gap described above.

A secondary output is a reusable container interface and private pool. These could later host digitized detector waveforms, such as the germanium charge pulses used for surface-alpha rejection in α-event characterization in point-contact HPGe detectors. The first study, however, stays on jets, where open data and reported baselines exist.