Charge-Pattern n-grams and Held-Out Detector Variations for Auditing Neutrino Null Results

Several recent experiments report that a light "sterile" neutrino, a hypothetical particle that interacts only by mixing with ordinary neutrinos, does not explain long-standing anomalies. The MicroBooNE detector, which records neutrino interactions as tracks of ionization charge in liquid argon, found no excess of electron-neutrino events where MiniBooNE had seen one. A null result of this kind is only as trustworthy as the event selector that decides which recorded interactions count as signal. That selector is trained entirely on simulation, and the simulation of the detector's response is known to be imperfect. We propose to measure how much a null result depends on the selector learning quirks of the simulation. We then test whether a selector borrowed from text classification is more stable under changes in the simulated detector. That selector is a linear support vector machine on sparse "n-gram" features of the charge deposited along each particle.

The idea rests on a direct analogy. In native-language identification, linear classifiers on character n-grams win benchmarks but also latch onto the essay prompt, and they lose up to 20% accuracy when a prompt is held out. Simulation-trained selectors face the same risk with respect to the detector model. We will treat each simulated detector variation the way that literature treats a prompt: hold one out, measure the drop, and mask the features that betray which variation produced an event. The deliverable is a selection whose expected exclusion of the MiniBooNE excess shifts less across held-out detector models than the current tree-ensemble selection does, at equal or better sensitivity.

Core Claim

Our claim is that the selection systematic in simulation-trained rare-event searches is partly a generalization failure across detector models. It can be measured by held-out-variation evaluation and reduced by variation-aware feature masking. The claim is testable because the per-variation spread of expected sensitivity is a number we can compare across selectors.

The representation we propose is the discretized energy loss per unit length along the first centimeters of each reconstructed shower. In physics shorthand this is dE/dx, and it is the quantity that separates electrons from photons. Each shower's values become a string of symbols, and the string is encoded as hashed n-grams, exactly as essays are encoded in The Power of Character N-grams in Native Language Identification. The result is a sparse, very high-dimensional feature vector, the regime where linear margin classifiers are known to work well.

The masking step borrows the prompt-debiasing motivation from that same paper. A second linear classifier is trained to predict which detector variation generated an event. The n-gram features it weights most heavily are removed or down-weighted before the signal selector is trained.

Relation to Prior Work

The motivating evidence comes from MicroBooNE itself. The deep-learning CCQE search selected single-electron, single-proton events with an ensemble of 20 boosted decision trees (BDTs, sequentially fitted tree ensembles) per run, fed by kinematic summaries and convolutional-network particle scores. It observed 25 events against a constrained expectation of 27.4 ± 3.8 (sys) ± 5.2 (stat). This is our reference system; we reuse its data regime and its signal definition. The Wire-Cell inclusive search reached a stronger rejection with different reconstruction. The gap between the two analyses already hints that the choice of reconstruction and selector matters.

The detector-uncertainty study supplies our variation generator. It modifies simulated wire waveforms to produce detector-response variations, which shift observables by 1–2% and up to 6%, at roughly a tenth of the cost of full re-simulation. That work propagates these variations as uncertainties. It never uses them as held-out domains to train or select classifiers, and that is the step we add.

The text literature supplies the method and a constraint. Question classification using support vector machines found single decision trees among the weakest learners on sparse text. The ILPS Total Recall system found random forests consistently behind a linear model on sparse features. Together they predict that trees will do poorly on raw n-gram vectors, so the fair tree baseline must use the physicists' compact summaries rather than our sparse features.

Stacking a tree model with an SVM, as in the C5 plus one-class SVM intrusion detector, gave only small gains: 83.24% versus 81.53% on NSL-KDD. We therefore treat stacking as a secondary arm, not the headline.

Research Program

We will build the charge-sequence encoder on MicroBooNE-style simulation. Each shower's dE/dx profile is quantized into roughly a dozen symbols, and n-grams of length 1 to 6 are hashed into about a million dimensions. On top of this we train a linear SVM (L2-regularized hinge loss via dual coordinate descent).

We then generate K detector variations with the waveform-modification method and define a leave-one-variation-out protocol. Selectors are trained on the nominal model plus K−1 variations and evaluated on the held-out one.

Evaluation happens at two levels. At the event level, we measure signal efficiency at a fixed background rate. At the physics level, we fold selection efficiencies and background predictions into the collaboration's LEE fit to obtain the median expected excluded LEE normalization. The spread of that exclusion across held-out variations is our selection-robustness metric.

One caveat is practical. Full-rate MicroBooNE simulation with detector variations is collaboration-internal, and the file lists no public equivalent. The program therefore needs either collaboration access or a public liquid-argon simulation as a surrogate. We state this as a dependency rather than paper over it.

Baselines

Naive reference. The simplest reference is a cut-based selection on shower-start dE/dx and two-body kinematics, with no learning. It tells us how much any learned selector buys.

Strongest existing method. The main competitor is the 20-BDT-per-run ensemble of the CCQE search, retrained on the same kinematic and particle-score inputs. It is the validated, published selector on this exact signal. It reported excluding 0.25 (0.38) times the median MiniBooNE LEE strength at 90% (2σ) confidence, with a 2.4σ rejection. No per-variation spread of that exclusion is reported on Lacuna, and measuring it is part of this work.

Fairness. All selectors see the same simulated events and the same held-out variations. Each receives the same hyperparameter search of 50 configurations on a nominal-model validation split. Compute is matched to the BDT ensemble's training cost.

What counts as winning. Beating the BDT ensemble means matching its nominal expected exclusion within statistical error while reducing the spread of expected exclusion across held-out variations by a clearly resolved margin. We would treat a 30% reduction, larger than the bootstrap spread, as a win.

Ablations. We run three ablations, each isolating one ingredient. First, the same linear SVM without variation masking isolates the masking step. Second, logistic regression on the same features isolates the hinge loss. Third, a BDT on the n-gram features isolates the representation from the classifier.

Why not just…?

Why not just propagate detector variations as systematics, as MicroBooNE already does? Propagation reports how much the prediction moves, but it does not choose a selector that moves less. The collaboration's own framework, the waveform-variation method, shows shifts of up to 6% but offers no way to reduce them. If held-out spreads turn out to be equal across all selectors, the BDT standard is vindicated and we will say so. That outcome would still be informative about where the systematic lives.

Why not just add dE/dx sequences to the BDT? This is a real threat. A BDT on a few hand-built shower-start summaries may capture most of what the n-grams encode. The BDT-with-summaries and BDT-on-n-grams arms settle it. If the first matches our SVM in both sensitivity and spread, the contribution reduces to the held-out protocol and masking, which we would then apply to the BDT.

Why not just use logistic regression instead of an SVM? Nothing in our argument depends on the hinge loss. The ILPS result favored logistic regression over forests, not over SVMs, so the file gives no evidence either way. We run both on identical features and report whichever wins. The claim concerns representation and invariance.

Why not just retrain the convolutional networks end-to-end? Convolutional networks can learn simulation artifacts as readily as any model. The same held-out protocol applies to them if compute allows. The high reported accuracy of SparseSSNet, 99.6% on showers, is measured on the nominal simulation and says nothing about held-out variations.

Decisive Checks

We will first validate the masking recipe where ground truth is cheap: leave-one-prompt-out on the NLI Shared Task 2017 data. The published linear SVM reached test F1 0.8756, and the source reports up to a 20% drop when a prompt is removed. If masking prompt-predictive n-grams does not shrink that drop, the physics transfer is unlikely to work, and we stop early.

On the neutrino side, the decisive result is the pair (nominal expected exclusion, spread across held-out variations) for each selector. A second check injects a synthetic single-photon excess, the alternative the Wire-Cell paper leaves open, into held-out variations. This tests whether each selector's null result would have hidden a real signal.