Exact Accuracy–Invariance Paths for Systematics-Robust Classifiers

Particle physicists train classifiers to pick signal events out of large backgrounds. They often also need those classifiers not to depend on a nuisance variable, meaning a quantity that is poorly modeled or must stay untouched, such as a jet's mass in a resonance search or the number of overlapping "pileup" collisions. The standard tool is adversarial pivoting. A second network tries to predict the nuisance from the classifier output, and a weight $\lambda$ sets how hard the classifier is pushed to defeat it. Every published use explores $\lambda$ by training separate, unstable minimax models at a few hand-picked values. Physicists therefore see the tradeoff between separation power and nuisance independence only at scattered points, and they do not know whether the best operating point falls between them.

We propose to make that whole tradeoff curve exact and cheap for one convex classifier. A linear support vector machine (a classifier that maximizes a margin with a hinge loss) is trained on features from a frozen network. We add a penalty on the difference in mean background score between mass bins, and each bin's term can be driven exactly to zero. With this penalty, the optimal classifier weights move along straight-line segments as $\lambda$ increases. The path can be traced from breakpoint to breakpoint with the same active-set machinery that computes the full support vector machine regularization path. The deliverable is the complete accuracy–invariance frontier with no minimax training. It also gives a certificate of which mass bins are exactly decorrelated at each $\lambda$, and it lets us find the $\lambda$ that maximizes the downstream physics metric (discovery significance under a background systematic) along the continuum rather than on a grid.

Core Claim

The adversarial penalty in Learning to Pivot with Adversarial Networks and Decorrelated jet substructure tagging using adversarial neural networks can be replaced, for the final classification layer, by a polyhedral penalty that keeps the problem a convex quadratic program. The parameter $\lambda$ enters that program linearly, so the solution path is exactly piecewise linear. Let $\phi(x)\in\mathbb{R}^d$ be a frozen feature map, and let the score be $s(x)=w^\top\phi(x)+b$. For $K$ background mass bins, define $d_k$ as the mean feature vector of background events in bin $k$ minus the mean over all background. The problem is

$$\min_{w,b}\ \tfrac12\|w\|^2 + C\sum_i \max\big(0,\,1-y_i s(x_i)\big) + \lambda\sum_{k=1}^K |w^\top d_k| ,$$

where $y_i\in\{\pm1\}$ are labels and $C$ is fixed. Its optimality conditions give $w=\sum_i \alpha_i y_i\phi(x_i)-\sum_k \beta_k d_k$, with event multipliers $\alpha_i\in[0,C]$ and bin multipliers $\beta_k\in[-\lambda,\lambda]$.

Within a fixed active set, these conditions are linear in $\lambda$. The active set records which events sit on the margin and which bins are exactly decorrelated. Breakpoints occur in two ways. An event may enter or leave the margin, as in The Entire Regularization Path for the Support Vector Machine. Or a bin's multiplier may leave its box, so that its first-moment constraint $w^\top d_k=0$ becomes exactly active or is released. As $\lambda$ grows, the path therefore shows mass bins locking into decorrelation one at a time. This is an interpretable object that adversarial training cannot produce.

The claim to be tested is empirical. On the boosted $Z'\to q\bar q$ benchmark, this exact path on a frozen embedding should reach an accuracy–sculpting frontier competitive with grid-searched adversarial networks, at a fraction of the cost and with no training instability.

Relation to Prior Work

Adversarial pivoting motivates the problem. Learning to Pivot with Adversarial Networks introduced the minimax formulation with a mixture-density adversary. Decorrelated jet substructure tagging using adversarial neural networks showed that decorrelating jet mass stops the classifier from sculpting the background mass shape. Sculpting means a mass cut creates artificial bumps that corrupt sideband fits. Both papers train independent models at discrete $\lambda$. We keep their goal and benchmark but trade the adversary's generality, since it can target any dependence, for exactness: our penalty controls only first moments per bin. Whether that is enough is the central empirical risk.

Parametric programming supplies the method. The Entire Regularization Path for the Support Vector Machine traced the path in the regularization constant. Multi-parametric solution-path algorithm for instance-weighted support vector machines traced paths along rays in instance-weight space. We add a new class of multiplier, bounded by $\lambda$ itself and attached to decorrelation terms rather than to events. Both papers also constrain the design. They report numerical drift and near-singular systems after many breakpoints, and the number of breakpoints can grow badly with sample size. We therefore measure breakpoint counts on physics-scale samples rather than assume them.

Beyond Discretization: Learning the Optimal Solution Path approximates solution paths with polynomial bases under convexity. Our problem is convex, so its guarantees apply, which makes it a natural approximate competitor to exact tracing.

Two further lines set the evaluation context. Application of Inferno to a Top Pair Cross Section Measurement with CMS Open Data optimizes a summary statistic for final measurement precision under a jet-energy-scale nuisance. We use it as a second test setting in which the bins are replaced by the nominal-versus-shifted template differences. Classification under Nuisance Parameters and Generalized Label Shift in Likelihood-Free Inference calibrates a fixed classifier after training. It is complementary: it could sit on top of any point on our path.

Research Program

The first component is the path algorithm itself. We extend the support vector machine active-set updates to include the bin multipliers $\beta_k$. At each breakpoint we solve a linear system for the slopes of $(w,b,\alpha,\beta)$ and compute the next event or bin event analytically. The system is posed in the primal, with $d$ around 64–256 frozen features plus $K$ around 10–20 bins, so each step's linear algebra scales with $d+K$ rather than with sample size. We add periodic re-solves from scratch to control the drift reported by Hastie et al. and Karasuyama et al.

The second component is exact downstream selection. Within a segment every event's score is affine in $\lambda$. Background rejection at fixed signal efficiency, the sculpting divergence, and discovery significance can therefore be computed from closed-form weights at any $\lambda$, at negligible cost. We scan densely inside each segment and at every breakpoint. The physics-optimal $\lambda$ is chosen on held-out data.

The third component is the choice of features. We use the penultimate layer of a network trained with ordinary cross-entropy, computed on the 11 high-level substructure features. As a representation-free control we also use random Fourier features of those 11 inputs.

Baselines

The naive reference point is the standard network trained with binary cross-entropy and no systematics handling. On this benchmark, Decorrelated jet substructure tagging using adversarial neural networks reports that such a network reaches 2.82σ discovery significance under a 50% background uncertainty and visibly sculpts the mass distribution. We also report our path's own $\lambda=0$ endpoint, which is the same frozen features with a plain support vector machine head. That endpoint separates the effect of the linear head from the effect of the penalty.

The strongest existing method is adversarial pivoting. The same paper reports 3.67σ in that 50%-uncertainty setting. Learning to Pivot with Adversarial Networks reports an approximate median significance of about 7.8 for the pivot at $\lambda=10$ versus about 7.0 for a standard classifier, on its pileup-nuisance example. For a fair comparison we regenerate the benchmark with the described MadGraph5/Pythia8/Delphes signal and Sherpa background pipeline. We train adversarial networks on identical events over a $\lambda$ grid, with a matched GPU-hour budget and several seeds per grid point. We report both the envelope of their frontier and the published numbers, reading the published numbers as reproduction targets rather than direct comparisons.

Beating adversarial pivoting means one of two outcomes. The path's significance-optimal point could match or exceed the best adversarial grid point within seed-to-seed spread. Or, at equal sculpting divergence, the path's frontier could lie on or above the adversarial envelope, and do so at a small fraction of the compute.

Two further baselines isolate the key ingredients. The first is the same convex problem solved independently on a grid of $\lambda$ values, which tests whether exact path tracing matters beyond the convex penalty itself. The second is Beyond Discretization: Learning the Optimal Solution Path's adaptive basis method applied to the same objective, which tests whether an approximate path suffices. It is compared by uniform error to the exact path and by gradient-call budget.

On CMS Open Data, the naive and strongest references are those of Application of Inferno to a Top Pair Cross Section Measurement with CMS Open Data. On Asimov data, that paper reports signal-strength uncertainty of about 10–11% with cross-entropy versus 9% with INFERNO, and a jet-energy-scale correlation of −0.42 versus −0.10.

Why not just…?

Why not just grid-search the adversarial network, since compute is cheap? Grid points are uncoupled and individually unstable. The published studies used only a handful of $\lambda$ values, so the frontier between them is unobserved. The support vector machine path computed 632 exact steps in 1.44 s against 9.28 s for 10 grid points in The Entire Regularization Path for the Support Vector Machine. That is a precedent, not proof at physics scale. Our matched-budget frontier comparison decides whether the adversary's extra flexibility buys anything the grid can reliably find.

Why not just train one network that takes $\lambda$ as an input? This is the most serious threat. Parameterized inputs already work for hypothesis masses in Decorrelated jet substructure tagging using adversarial neural networks, and a $\lambda$-conditioned adversarial network could give a smooth frontier in one training run. It still inherits minimax instability and gives no exactness or per-bin certificate. We run it as an additional baseline under the same budget. If it dominates our frontier by more than seed noise, the honest conclusion is that exactness buys interpretability and certification but not performance, and we will report it that way.

Why not just decorrelate after training, for example with a mass-dependent threshold that keeps background efficiency flat? This flattens sculpting at one working point without changing the ranking of events. It is cheap and a real competitor. It is not decorrelated across working points, however, and it does not trade separation power for invariance. We include it and compare sculpting across the full range of efficiencies, not just at the tuned one.

Why not just control first moments, when sculpting depends on the whole distribution? Mean-matching per bin may leave higher-moment dependence. We measure the full-histogram divergence, not the penalized quantity. If the residual is large, we will extend the penalty to several fixed score quantiles or random projections. Both extensions keep the problem polyhedral.

Proposed Experiments

On the $Z'$ benchmark, with $m_{Z'}$ spanning 20–300 GeV, we trace the full path and plot background rejection at 50% signal efficiency against the sculpting divergence. We report the maximal discovery significance under a 50% background uncertainty, the number of breakpoints as the training size grows from $10^4$ to $10^5$ events, and wall-clock time. We also report the $\lambda$ values at which each mass bin locks into decorrelation.

On CMS Open Data, bins are replaced by the nominal-minus-±1σ jet-energy-scale template differences. We then report signal-strength uncertainty and nuisance correlation at the path's best point against the INFERNO figures above.

Three results would be decisive. The first is whether the exact frontier touches or exceeds the adversarial envelope at equal compute. The second is whether exact tracing recovers a significance optimum strictly between grid points that the convex-grid ablation misses. The third is whether breakpoint counts grow slowly enough, roughly linearly in sample size, for the method to remain practical. A clear failure on the first result with success on the third would reposition the method as a cheap diagnostic of the frontier rather than a replacement for adversarial pivoting.