Telling Signal from Artifact When Boosted Trees Learn from Mixed Samples

Searches for rare particles often cannot train on clean labels. Instead, physicists train a classifier to tell apart two batches of real events that should differ only in how much signal they contain. Examples are a mass window against its neighbouring windows, or real data against an estimated background. Boosted decision trees (ensembles of small trees, each fit to correct the errors of the ones before) are the workhorse for this. From the classifier's point of view, though, almost every label is wrong: nearly all events in the "signal-rich" batch are background. The robust-boosting literature defends against randomly flipped labels. The noise that actually hurts here is different. It is learnable structure, such as a detector effect or a flaw in a background model, that differs between the batches for reasons other than signal. A booster will fit that structure as eagerly as it fits signal, and robust losses cannot tell the two apart.

We propose using three or more batches with different, unknown signal fractions. When separation comes from genuine signal, it has a precise fingerprint: how well any score separates each pair of batches must scale with that pair's difference in signal fraction. Artifacts generally break this scaling. We build this consistency test into the boosting loop as a gate on which trees are kept. No clean "trusted" labels are needed, and the method targets the failure that matters most for discovery: a spurious excess produced by the classifier itself.

Core Claim

Training on mixed samples was formalized as Classification Without Labels, or CWoLa (Classification without labels: learning from mixed samples in high energy physics). That paper shows that the best classifier separating two mixtures with signal fractions $f_1 \neq f_2$ is also the best signal-versus-background classifier. The guarantee holds only if signal and background look identical in both mixtures. A short calculation turns this assumption into something testable. Write $A(s)$ for the true signal-versus-background area under the ROC curve (AUC) of a score $s$. Write $\Delta_{jk}(s)$ for the AUC of $s$ at separating mixture $j$ from mixture $k$, minus one half. Under the CWoLa assumptions, expanding the pairwise comparison gives

$$\Delta_{jk}(s) = (f_j - f_k)\,\big(A(s) - \tfrac{1}{2}\big).$$

Across all mixture pairs, the vector of $\Delta_{jk}(s)$ therefore lies on one fixed ray, whose direction is set by the unknown fraction differences. This holds for every score, and so for every partial ensemble during boosting. Suppose instead that mixture $k$ contains an extra component $\epsilon_k(x)$ that is not signal, such as a background shape that depends on selection or a template mismatch. That component adds a term to $\Delta_{jk}$ that need not follow the ray.

Our central claim is that measuring how far each new tree pushes the held-out $\Delta$ vector off the ray separates learned signal from learned artifacts. Gating boosting on this measure removes classifier-induced false excesses that early stopping and noise-robust losses leave in place. It should also cost almost nothing in sensitivity when the data are clean.

Why Boosting's Noise Sensitivity Is the Right Lens, and Where It Misleads

Theory on noisy labels in boosting is thorough, but it targets random flips. Boosting can be read as gradient descent on a margin loss (Boosting Algorithms as Gradient Descent; A Brief Introduction to Boosting). Under the exponential loss, mislabeled points gain weight without bound. This reweighting can push the effective noise rate seen by the base learner past one half, which voids the weak-learning guarantee (Boosting in the Presence of Massart Noise). These results motivate our concern about mixture training, where most labels are "wrong."

The proposed fixes are bounded or non-convex losses: SavageBoost (On the Design of Loss Functions for Classification: theory, robustness to outliers, and SavageBoost), tempered weight updates (Boosting with Tempered Exponential Measures), and losses that stop tree growth (Robust Loss Functions for Training Decision Trees with Noisy Labels). Other fixes cap margins through a minimax formulation (Robust Minimax Boosting with Performance Guarantees) or model flip probabilities explicitly (Boosting in the presence of label noise). All of them limit the influence of individual points that cannot be fit.

That defence is aimed at the wrong target here, for two reasons. First, in mixture training the "mislabeled" points are not isolated outliers. They fill the whole feature space, and the true signal events are the ones the classifier most needs to keep. A loss that gives up on badly misclassified points may give up on exactly those events. Second, an artifact present in one mixture is perfectly learnable, so it carries no outlier signature for any loss to suppress.

The physics literature confirms that this second failure is real. The anomaly search in CMS Open Data (Isolating Unisolated Upsilons with Anomaly Detection in CMS Open Data) trains boosted trees on real data against a background estimated by a normalizing flow, a generative model that learns a density. Because the flow is imperfect, a bias-free result is not guaranteed, and the authors needed a separate same-sign control sample to rule out a false peak. The JUNO pulse-shape study (Pulse shape discrimination technique for diffuse supernova neutrino background search with JUNO) shows that background events closely mimicking signal can be present and selection-dependent. CWoLa supplies our key identity. The robust-boosting papers serve as the competitors that our setting is designed to challenge. Their reliance on a trusted clean set, explicit in rBoost, is a limitation we remove.

Research Program

We will build consistency-gated boosting. It wraps standard gradient-boosted trees with logistic loss (XGBoost-style (Higgs Boson Discovery with Boosted Trees)) and is trained on $K \ge 3$ mixtures. The training target is ordinal: each mixture is labelled by its rank in expected signal enrichment, which physicists usually know even when the fractions themselves are unknown. Each held-out mixture pair is also kept as a separate validation stream.

After each candidate tree, we compute the change it causes in the held-out vector $\Delta(s)$. We maintain a running estimate of the signal ray, taken as the leading direction of accumulated increments. We bootstrap the angle between the new increment and that ray. A tree is accepted if its increment lies within the bootstrap tolerance of the ray. It is rejected if its off-ray component is significant. In that case, the leaves responsible for the off-ray part receive a growing penalty. This acts as a "brake" in the spirit of Robust Loss Functions for Training Decision Trees with Noisy Labels, except that it triggers on inconsistency rather than on loss magnitude. A lighter variant only adds the off-ray norm as an early-stopping criterion.

The analysis side has three goals. The first is a sample-size law for when the test has power. Each $\Delta_{jk}$ is shrunk by its fraction difference $\delta_{jk}$ while its estimation error scales roughly as $1/\sqrt{n}$, from standard AUC variance. Detection therefore needs roughly $n \gg 1/\delta^2$ events per pair. The second is a characterization of artifacts that evade the test: any artifact whose strength happens to scale with signal enrichment. A realistic example is a flow interpolation error that grows toward the centre of a signal window. The third is to compare these predictions with the behaviour of robust losses inside the same ordinal setup.

Baselines

The naive reference point is standard XGBoost with logistic loss, trained on two mixtures in CWoLa fashion. It uses early stopping on held-out mixture loss and an ensemble of seeds, which matches how boosted trees are used in Isolating Unisolated Upsilons with Anomaly Detection in CMS Open Data. That pipeline reports raising significance from 1.6σ to 6.4σ on 2016 CMS Open Data, and it is the strongest existing method on the real-data question.

A second variant is the same XGBoost trained on all $K$ mixtures with the ordinal target but without the gate. This ablation isolates the gate itself from the extra data. A third variant adds the "do nothing beyond physics" reference: a one-dimensional cut on the most discriminating variable.

The strongest competitors from the label-noise literature are RMBoost, SavageBoost and $t$-AdaBoost, each run in the same mixture setup. RMBoost is the most serious of these. On Credit with 10% symmetric noise, its error rose from 14% to 16%, against 14% to 18% for AdaBoost (Robust Minimax Boosting with Performance Guarantees). It is also the only one already run at 11-million-event HIGGS scale. SavageBoost reports 25.9% average error at 40% outliers, against 38.2% for AdaBoost (On the Design of Loss Functions for Classification: theory, robustness to outliers, and SavageBoost).

We found no reported result on Lacuna for any booster trained on mixtures of HIGGS or SUSY events. Those numbers will be produced here. The fully supervised XGBoost and the deep network of Searching for exotic particles in high-energy physics with deep learning serve as clean-label ceilings on simulation. That paper reports 4.9σ on low-level HIGGS features for its deep network. A neural network trained in CWoLa fashion is included as the original recipe.

For fairness, all methods see identical mixtures and identical held-out splits. Every method gets the same budget of hyperparameter trials, selected only on held-out mixture metrics, never on truth labels. The tree depth range is shared across methods.

Beating the baselines means three things. Spurious significance on signal-free, artifact-injected data must fall to the level seen on artifact-free data, where the baselines show a clear excess. Background rejection at fixed signal efficiency must stay within statistical error of ungated XGBoost when no artifact is present. And the gated method must match or exceed CATHODE's significance on CMS Open Data while showing no sculpting in the same-sign control.

Why not just…?

Why not just use XGBoost with early stopping, as current analyses do? Early stopping halts when held-out mixture loss stops improving. An artifact that differs between mixtures improves held-out loss exactly as signal does, because it reappears in the held-out events. Early stopping is therefore blind to it by construction. This is a real threat only if artifacts happen to be weak in practice. The signal-free injection experiment settles the question directly. If ungated XGBoost shows no spurious excess there, the gate is unnecessary, and we will report that outcome.

Why not just swap in a noise-robust loss such as RMBoost or SavageBoost? These losses cap or saturate the influence of points that cannot be fit. A learnable artifact is fit easily, so it is never down-weighted. Meanwhile, the scarce true signal events are the ones most likely to be treated as hopeless. The file's own tension analysis of saturating losses under extreme rarity points the same way. Their reported gains come from symmetric or adversarial random flips on balanced data. We run them head to head. If RMBoost suppresses artifact excesses comparably, the proposal reduces to recommending RMBoost, which is an acceptable outcome.

Why not just validate afterwards with a control sample, as the CMS analysis did? A control region, like the same-sign muon sample, can detect sculpting but cannot remove it. It also requires a control sample that shares the artifact but contains no signal, and many searches lack one. The gate uses only the mixtures already in the analysis and acts during training. We still keep the same-sign check as an independent judge on CMS data.

Why not just train a neural network? The failure mode we study depends on the training data, not the model family. A network trained on contaminated mixtures will learn the same artifact. We include neural-network CWoLa to confirm this. The gate itself is model-agnostic and could later wrap networks. Trees remain the preferred deployment model for low-latency triggering and modest training sets, as noted in Intelligent experiments through real-time AI: Fast Data Processing and Autonomous Detector Control for sPHENIX and future EIC detectors.

Proposed Experiments

Controlled studies on HIGGS and SUSY. On HIGGS (11 million events) and SUSY (5 million), we will build $K = 3$ to $5$ mixtures from truth labels. Signal fractions will range from $10^{-1}$ down to $10^{-4}$. We will inject three kinds of contamination. The first is random label flips, the case the robust losses were designed for. The second is a mixture-specific background distortion that does not scale with signal enrichment, such as a shifted lepton-momentum distribution in one mixture, standing in for selection-dependent detector effects. The third is an adversarial distortion that does scale with enrichment, the known blind spot.

The metrics are truth-label AUC, background rejection at 50% and 80% signal efficiency, and $s/\sqrt{b}$ in the selected region. The decisive check is the signal-free run: all $f_k = 0$ with artifacts present. Any significance reported there is classifier-made. We will plot spurious significance against injected artifact strength and sample size, and test whether the gate's detection threshold follows the predicted $n \sim 1/\delta^2$ law.

Discovery metric. On the Kaggle Higgs Challenge data, the reference is XGBoost's fully supervised test Approximate Median Significance (AMS) of 3.72370 (Higgs Boson Discovery with Boosted Trees). We report the AMS each method reaches when trained only on mixtures, to show how much of the supervised ceiling consistency-gated mixture training recovers.

Real data. On the 2016 CMS DoubleMu Open Data, we will rerun the CATHODE-style search with adjacent mass windows and sidebands as the $K$ mixtures. We will compare significance and same-sign sculpting against the reported 6.4σ. Where collaboration simulation is available, the same protocol applies to JUNO pulse-shape data. There, mixtures are formed by distance from preceding muon tracks, which enriches cosmogenic mimics. The reference is the reported operating point of roughly 80% signal efficiency at 1% background leakage.

What would refute the proposal. The proposal is refuted if ungated XGBoost and RMBoost show no artifact-driven excess at realistic sample sizes. It is also refuted if the gate rejects so many trees on clean data that background rejection drops beyond statistical error.