Early-Stopping Ensembles for Rare-Signal Selection

Physics experiments that hunt for rare processes, such as a Higgs boson decay among ordinary jets or a heavy-quark decay among millions of routine collisions, must throw away almost everything they see. They must also keep nearly every real signal event. Averaging or voting across several independently trained neural networks is a well-tested way to sharpen that kind of decision. Physics already relies on it in a classical form: the MAGIC gamma-ray telescope averages decision trees into a single "hadronness" score (Random Forest for MAGIC). The obstacle is cost. Running $M$ Transformers costs $M$ times as much, and physics Transformers such as the Particle Transformer are already expensive because every particle attends to every other particle (Particle Transformer for Jet Tagging).

This note proposes to evaluate the ensemble members one at a time and stop as soon as the outcome of the full vote can no longer change. The idea exploits the class imbalance that makes the problem hard in the first place. When a trigger runs at a tight working point, a background event that the first member rejects confidently cannot be rescued by the remaining members. So the overwhelming background stream costs roughly one model per event. The rare signal candidates, which are the events where extra opinions matter, receive the whole ensemble. The resulting decision is bit-for-bit identical to the full ensemble's, at a cost close to that of its cheapest member. A relaxed variant trades a bounded, measured amount of disagreement with the full ensemble for further savings.

Core Claim

Each ensemble member outputs a score $s_m \in [0,1]$. The full ensemble accepts an event when the mean score reaches a threshold $\tau$, that is, when $\sum_{m=1}^{M} s_m \ge M\tau$. After $j$ members have run with partial sum $S_j$, two exact stopping rules apply. If $S_j + (M-j) < M\tau$, the event is rejected, because even perfect scores from the remaining members cannot reach the threshold. If $S_j \ge M\tau$, the event is accepted. Majority or $k$-of-$M$ voting is the special case where each $s_m$ is a 0/1 vote.

The point of this rule is how it behaves at the working points physics actually uses. Take $M=5$ and $\tau = 0.9$. A first-member score below 0.5 already decides rejection. Under a unanimity vote ($k=M$), a single background vote decides it. With members ordered from cheapest to most expensive, the expected cost per event is

$$ \mathbb{E}[C] = \sum_{j=1}^{M} c_j \, P(\text{event still undecided after } j-1 \text{ members}), $$

where $c_j$ is the compute cost of member $j$. The probability is taken under the deployment mix of signal and background. Background dominates that mix by orders of magnitude, so $\mathbb{E}[C]$ is governed by how often background survives the first member or two. This is exactly the quantity a good discriminator makes small.

The claim has three parts. Ensembling physics Transformers improves background rejection at fixed signal efficiency. Exact sequential stopping recovers most of the ensemble's cost. A calibrated relaxed rule recovers more, with a stated bound on how much signal efficiency it can lose.

The first part is not established anywhere in the file. No reported result was found on Lacuna for ensembles of the Particle Transformer or any other physics Transformer. If that experiment shows no gain, the project stops there.

Relation to Prior Work

Three lines of work motivate the proposal.

The first shows that the base discriminators are strong but costly. The Particle Transformer adds pairwise kinematic terms to its attention and improves background rejection over ParticleNet by a factor of 2 to 3 on JetClass, but it scales quadratically in particle count (ParT). The Mixture-of-Experts Graph Transformer reaches AUC 0.929 on simulated ATLAS supersymmetry events by routing particles inside one network to specialised sub-networks (MGT). That routing happens within a single model, which is different from deciding whether further independent models are needed at all.

The second line shows that voting helps, but only by running every member. Five-seed RoBERTa voting (AIR-JPMC@SMM4H'22) and majority voting with a confidence veto (Ax-to-Grind Urdu) both evaluate all members on every input. The veto method's reported 0.956 accuracy is bought at triple the inference cost. Combinations of Weak Classifiers supplies the theory that errors fall quickly with ensemble size when members err independently. That assumption is exactly what seed-only diversity may violate, so our design tests heterogeneous members as well.

The third line supplies the mechanisms we borrow and contrast against. The rare-event cascade of Direct Feature Selection already places a majority vote inside each stage and rejects early, but over Haar features on images rather than deep models. Learning to Cascade trains a cheap model to hand off to an expensive one and cuts CIFAR-100 compute by 31–55% with no accuracy loss. It is vision-only, tested on balanced classes, and does not preserve any ensemble's decision. LEAP makes early exit inside a Transformer work, reaching 1.61× wall-clock speedup at batch size 1. It constrains our design in one specific way: per-layer exits lose much of their speedup under batching. Stopping between whole models avoids that problem, because undecided events can be gathered into a fresh dense batch before the next member runs.

Two papers act as constraints rather than methods. The sPHENIX FPGA trigger runs one attention model in 8.82 µs (Real-time AI for sPHENIX), and HEPT shows that efficient attention matters at scale (HEPT). Our method lowers average cost, not worst-case latency. A hard microsecond first-level trigger needs the worst case of all $M$ members to fit its budget. We therefore target GPU-based software triggers and offline selection, and we treat FPGA deployment only as a worst-case resource estimate.

Research Program

The first step is to build ensembles and measure whether they help. We train five Particle Transformers with different seeds. We also build a heterogeneous set: a small Transformer, ParticleNet-style point networks, and the full Particle Transformer. All are trained on JetClass and on the Top Quark Tagging set. We measure background rejection, defined as the inverse of the background acceptance rate, at fixed signal efficiencies, including tight points such as 30% and 50%.

The second step is to implement exact sequential stopping for both score averaging and $k$-of-$M$ voting, with cost-ordered members and batch compaction between members. We report expected floating-point operations per event and measured GPU throughput. Both are reweighted to deployment priors of $10^{-3}$ and $10^{-6}$ signal fraction, because the training sets are not imbalanced in the way a real trigger stream is.

The third step is a relaxed rule. On held-out data we estimate the probability that the remaining members would flip the current partial decision, conditional on the partial sum and members seen so far. We stop when an upper confidence bound on that probability falls below $\varepsilon$. The rule is asymmetric. Rejection is permitted early, but an event provisionally accepted as signal always runs the full ensemble, so any loss of signal efficiency is bounded by $\varepsilon$ and measured directly. Finally, we learn the member order, choosing members by expected information per unit of compute rather than by cost alone.

Baselines

The naive reference point is a single Particle Transformer with its threshold moved to the working point. It defines both the cost unit and the rejection that the ensemble must beat. The strongest reported number on this question is the Particle Transformer's 86.1% JetClass accuracy, against 84.4% for ParticleNet, together with its 2–3× improvement in background rejection over ParticleNet (ParT). On top tagging, fine-tuned ParT increased background rejection by 70% over previous bests. Absolute rejection values for ensembles are not in the file. We will reproduce ParT to within its reported accuracy before comparing anything against it.

The full ensemble, run on every event, is the accuracy reference. The sequential method must match it exactly in the exact mode, and within the stated $\varepsilon$ in the relaxed mode.

The strongest cost competitor is a single student network distilled from the ensemble, with the same architecture and parameter count as one member. Distillation from an ensemble teacher is the route NiuTrans used, retaining 44.3 of 44.5 BLEU.

The second cost competitor is a Learning to Cascade two-stage pipeline, running the small Transformer first and the Particle Transformer second, with its cascade loss. Applying that loss to Transformers is itself untested in the file. A further point of comparison is a single Particle Transformer widened or deepened to match the ensemble's average compute.

All baselines use the same training data, the same total training GPU-hours, and the same hyperparameter search budget. Distillation gets the full ensemble as its teacher plus extra unlabelled-weight epochs equal to our ordering-search budget.

We beat a baseline if, at equal expected compute per event under the deployment prior, we achieve higher background rejection at 30% and 50% signal efficiency. The difference must lie outside the seed-to-seed spread, using bootstrap intervals over test events.

Ablations isolate the key ingredient, which is stopping on the aggregate decision rather than on a single model's confidence. They compare random versus cost-ordered members, exact versus relaxed stopping, homogeneous versus heterogeneous members, and voting versus score averaging.

Why not just…?

Why not just distill the ensemble into one model? This is the real threat. If a distilled student matches the ensemble's background rejection at tight working points, the student wins on simplicity and on worst-case latency. The file gives no evidence on how well distillation preserves the far tails of the background distribution, which is where trigger rejection lives. The comparison at 30% signal efficiency settles it. If the student comes within the seed spread of the ensemble there, we will recommend distillation and report the sequential method as unnecessary.

Why not just train a bigger single model with the same compute? The ensemble's expected cost under a background-dominated prior is close to one member, so the fair comparison is against a model of that compute, not of $M$ times it. We run the compute-matched larger Particle Transformer anyway. If it beats the sequential ensemble at equal expected compute, the claim fails.

Why not just use a confidence cascade or per-layer early exit? Learning to Cascade and LEAP both stop on one model's confidence. Under heavy imbalance, that confidence is skewed toward background, which risks prematurely rejecting subtle signal events. Neither method guarantees agreement with a stronger reference decision. We test this directly by measuring signal efficiency lost relative to the full ensemble at matched compute.

Proposed Experiments and Decisive Checks

The first check is a gate. Does a five-member ensemble improve background rejection over a single Particle Transformer at 30% and 50% signal efficiency on JetClass and top tagging? If the improvement is within seed noise, we stop. The second check measures how much exact stopping saves at deployment priors, and whether the saving grows as the working point tightens, as the stopping rule predicts. The third check runs the head-to-head comparisons against the distilled student and the cascade at equal expected compute. The fourth check measures whether the relaxed rule's observed signal-efficiency loss stays below $\varepsilon$ on held-out data.

On the ATLAS supersymmetry set, we repeat the first and third checks using MGT and Graph Transformer members against MGT's reported 0.929 AUC. This tests whether the result holds for whole-event classification rather than only for jets. Finally, we report measured batched GPU throughput alongside floating-point operation counts, so that compaction overhead cannot hide behind theoretical savings.