Retreating Up the Tree: Generator-Robust Prediction Sets for Neutrino Event Topologies

Neutrino detectors classify each interaction from images of particle tracks. Typical labels are which kind of neutrino it was, whether it produced pions, and how many protons and neutrons came out. Every classifier in use is trained on simulation, and the simulation is least trustworthy at exactly the finest labels. Protons and neutrons are knocked out by poorly understood effects inside the struck nucleus. MINERvA found that its tuned simulation overpredicts multi-neutron antineutrino events by about 50% at the peak. DUNE dropped the neutron-counting output of its classifier entirely because the simulation could not be trusted. The effective model GENIE uses for hadrons rescattering inside the nucleus is excluded by ProtoDUNE-SP pion–argon data at $p<0.001$. Coarse labels such as "charged-current muon neutrino, no pions" are far less sensitive to these effects.

We propose to organize event labels as a tree of observable final states and to build prediction sets with a coverage guarantee over that tree. The sets are calibrated against a family of competing nuclear-physics simulators rather than against one. When the simulators disagree about the fine label, the set retreats to a coarser node instead of committing to a fine-grained guess the physics cannot support. The output is a classifier that reports, event by event, the finest topology it can stand behind. We then test whether that honesty survives a change of simulator, which serves as a proxy for the gap between simulation and real data.

Core Claim

Two directions meet here, with different roles. The tree-structured hierarchy supplies the mechanism: a principled meaning for "retreat to a safer, coarser answer." Nuclear effects supply the reason the mechanism is needed and the place it gets tested. The claim has three parts. First, misclassifications caused by simulator error are concentrated at the deep levels of an observable-topology tree, namely nucleon and neutron multiplicity. Second, standard split conformal prediction loses its coverage at those levels when the test simulator differs from the calibration simulator. (Conformal prediction is a method that turns any classifier's scores into label sets containing the true label with a chosen probability, here 90%.) Third, hierarchical conformal sets calibrated by worst case over a simulator family keep coverage across simulator shifts. They should also stay much smaller than simply collapsing the taxonomy to coarse labels.

A design choice matters here. The research file notes that nuclear effects make the map from primary process to final state a directed acyclic graph rather than a tree. An observed muon plus one proton can come from single-nucleon scattering, from two-nucleon knockout with an unseen neutron, or from pion production followed by pion absorption. Tree methods break on such graphs, as noted for hierarchical cross-entropy in Making Better Mistakes and for parent-centred priors in Neal's hierarchy-based prior. We therefore do not put the primary process in the label space. The hierarchy is over what the detector can see. It runs from charged- versus neutral-current, to flavor, to pion content ($0\pi$, $1\pi^\pm$, $1\pi^0$, multi-pion), to proton multiplicity (0, 1, ≥2), and finally to neutron presence. This is a strict tree. The many-to-one ambiguity of the primary process is pushed into the simulator family, where it belongs. Disagreement among simulators about how often absorbed pions produce a given proton count is exactly what makes the proton-count level unreliable.

Research Program

We first build a simulated liquid-argon dataset at Booster-beam-like energies of about 1 GeV, in the style of MicroBooNE, with the same detector simulation throughout. Only the interaction model changes. Variants come from configurations the literature already contrasts. These include default GENIE with the effective hA intranuclear cascade, a microscopic cascade alternative, and the G24 tunes from the combined TKI tuning, which changed the short-range-correlation fraction from about 12% to 15% and suppressed $\pi^0$ charge exchange. We also include pion-reinteraction reweightings constrained by the ProtoDUNE-SP absorption and charge-exchange cross sections from the exclusive π⁺–argon measurement. Each event is labelled with its leaf in the observable tree, which has roughly a hundred leaves.

We train one image classifier over the leaves on the default variant. The architecture is a ResNet-style network like the DUNE CVN in the DUNE far detector paper. Training uses plain cross-entropy, so that any gain comes from the decision layer, not from a new loss.

The prediction sets follow the representation-complexity construction of Mortier et al.. There, a set is a union of at most $r$ subtrees. The internal-node-only version is called CRSVP, and the $r$-bounded version found by dynamic programming is called CRSVP-r. Their method calibrates on held-out data from the training distribution. Our change is the calibration step. Let $s(x,y)$ be the nonconformity score of event $x$ with true leaf $y$, and let $\hat q_k$ be its $(1-\epsilon)$ conformal quantile on calibration events from simulator variant $k$. We set

$$\hat q = \max_{k \in \mathcal{K}} \hat q_k ,$$

where $\mathcal{K}$ is the set of calibration variants. Standard split-conformal arguments then give at least $1-\epsilon$ coverage under every variant in $\mathcal{K}$. Whether coverage extends to unseen variants and to real data is an empirical question, and we test it directly.

Because the sets are subtree unions, a large $\hat q$ caused by variants disagreeing about neutron counts widens the set at the neutron level only. It should not destroy flavor information. We expect, but must show, that this beats flat conformal sets. Flat sets cannot express "any proton count, but definitely $\nu_\mu$ CC $0\pi$" compactly.

We also compute post-hoc risk-optimal point decisions under the generalized tree distance of Cao, Feng and An. That loss has a penalty $c$ which moves the threshold for committing to a child node to $(1+c)/2$. The decisions are computed with the decoding algorithms of To Each Metric Its Decoding. This gives analysts a single "finest safe node" per event alongside the set.

Baselines

The naive reference point is the flat leaf classifier with argmax decisions. It is evaluated both on the calibration simulator and on a held-out simulator variant. This tells us how badly a fine-grained label degrades under nuclear-model shift. No reported result was found on Lacuna for leaf-level observable-topology classification on argon. The nearest reported numbers are coarse. The DUNE CVN reports above 85% $\nu_e$ CC selection efficiency at 2–5 GeV and 90% overall purity (DUNE CVN). The VLM study reports 87% three-class accuracy (VLM neutrino classification).

The strongest existing set-valued method is CRSVP-r from Mortier et al., calibrated in the usual way on the training simulator only. Its reported efficiency is about 44 leaves on average for CRSVP-3, against about 123 for single-node CRSVP, on PlantCLEF 2015 at 90% coverage. That is not a neutrino number, and we will reproduce it on our data. It is the right competitor because it already exploits the hierarchy; our method differs from it only in the calibration step.

The second strong competitor is flat split-conformal prediction calibrated with the same worst-case-over-simulators rule. This is the ablation that isolates the hierarchy. If it matches our coverage at similar size, the tree adds nothing. The third is post-hoc optimal hierarchical decoding of the flat classifier without any set construction (To Each Metric Its Decoding). That paper reports up to a 10% reduction in mistake severity on image benchmarks, so it is the point-prediction bar our decoded "safe node" must meet. The fourth is a classifier trained with hierarchical cross-entropy (Making Better Mistakes), testing whether a training-time loss alone yields the robustness.

All methods share the same backbone, training data, simulator variants and calibration events. The hierarchical-loss baseline gets the same hyperparameter sweep budget as the conformal calibration. Beating them means three things on held-out simulator variants. Coverage must be within one point of nominal where the baselines fall measurably below it. Average set size must be smaller than the flat worst-case sets. Flavor-level purity must be at least that of the flat argmax classifier.

Why not just…?

Why not just do what DUNE did and drop the unreliable outputs? Collapsing the tree to a fixed coarse level, say flavor and pion content, is the real threat because it is simple and robust. Its cost is that it throws away proton and neutron information for every event, including the many events where all simulators agree. Our experiments report the fraction of held-out events whose set stays at or below the proton-multiplicity level while keeping coverage. If that fraction is small, the fixed coarse taxonomy wins and we will say so.

Why not just propagate systematics the standard way, by reweighting simulation and widening uncertainty bands? Reweighting gives an uncertainty on aggregate rates. It does not tell an analysis which individual events have trustworthy fine labels, and it inherits the validity of the parameterized model. The ProtoDUNE-SP result shows the effective cascade model itself is excluded, not merely mistuned (π⁺–argon). The TKI tuning leaves residual tension even after fitting. The two approaches are complementary, and our per-event sets can be fed into a standard reweighting analysis.

Why not just use post-hoc hierarchical decoding on a flat classifier? That method minimizes expected hierarchical risk under the classifier's own probabilities. The authors note it depends on those probabilities being calibrated. A simulator shift is exactly when they are not. We test this directly by comparing decoding-only mistake severity on held-out simulators against our conformal method.

Why not just scale the model or use a vision-language model? Making Better Mistakes found that a decade of architectural progress cut top-1 error without reducing mistake severity. A bigger network trained on one simulator also learns that simulator's nuclear model more faithfully. We include one larger backbone to check that the coverage drop under shift does not shrink with capacity.

Proposed Experiments

The decisive test is leave-one-simulator-out. For each variant $j$, we calibrate on the others and measure coverage and set size on $j$. We break the results down by tree level and by the region of muon transverse momentum and available energy that the MINERvA multi-neutron measurement flags as poorly modelled. The first prediction is that flat and standard hierarchical conformal sets lose coverage mainly through proton- and neutron-level errors. The second is that worst-case-calibrated hierarchical sets hold coverage at a modest cost in size. A third prediction is that sets widen preferentially in mis-modelled phase space. The hypothesis fails if coverage on held-out variants is no better than standard CRSVP-r. It also fails if our sets are no smaller than a fixed coarse taxonomy at matched coverage.

The second check uses electron-neutrino backgrounds, the case where errors matter most for measuring CP violation. We measure how often a neutral-current $\pi^0$ event receives a set or decoded node that excludes the neutral-current branch. We sweep the penalty $c$ to trace purity against completeness.

The third check is an external sanity test against real measurements. The MicroBooNE neutral-current π⁰ measurement reports exclusive $0p$ and $1p$ cross sections of 0.624 and 0.444 ×10⁻³⁸ cm²/Ar. We check whether the simulator variants that bracket this split are the same ones that drive our proton-level set widening. This tests whether the simulator family spans a physically relevant range rather than an arbitrary one.