Particle-physics measurements increasingly feed raw detector "images", grids of energy deposits in calorimeter cells and tracker hits, into neural networks that estimate physical parameters. Real detectors always have a few broken or switched-off cells, and the usual fix is to write a zero or an average value into each missing cell and carry on. This note argues that this single-guess fill-in, which statisticians call deterministic imputation, is a systematic error rather than noise. Its bias on a physics parameter stays fixed as the dataset grows, while the statistical error shrinks as $1/\sqrt{N}$. So at high data volumes it should become a residual systematic floor that the analysis does not report. We propose to measure that floor directly. We also propose a remedy that avoids imputation altogether: train the estimator on the observed cells together with the map of which cells are dead. That estimator should keep valid frequentist error bars, meaning the stated confidence intervals contain the true value at the stated rate, for every dead-cell pattern.
In two sentences: filling dead cells with zeros biases a neural parameter estimate by an amount that does not shrink with more data, so its importance relative to the statistical error grows like $\sqrt{N}$. Telling the network which cells are missing, instead of pretending they were measured, should remove that bias and keep the reported uncertainties honest, as long as the simulation reproduces how cells go missing.
Core Claim
The work starts from neural density-ratio estimation. A density-ratio estimator is a network that learns how much more likely an observation is under one hypothesis than under another, and such estimators are now used for simulation-based inference. Frequentist Uncertainties on Neural Density Ratios with wifi Ensembles (companion version here) freezes $M$ trained networks $f_i$ as fixed features and fits their combination weights by maximum likelihood. It then uses the sandwich covariance formula from M-estimation, the standard error formula for estimators defined by an objective, to obtain intervals on a quark fraction $\kappa$ in a jet mixture, with no need for pseudo-experiments. That paper supplies our inference machinery. It also states the constraint that matters here: coverage holds only if the true log-ratio lies in the span of the $f_i$, and it assumes every input is fully observed.
Our claim is that deterministic imputation breaks this assumption in a specific, measurable way. Suppose the zero-filled input $\tilde{x}$ is passed to networks trained on clean images. Then the fitted $\hat\kappa$ converges to a wrong value $\kappa^\ast \neq \kappa$. The standardized bias grows as
$$\frac{|\kappa^\ast - \kappa|}{\sigma_{\hat\kappa}(N)} \propto \sqrt{N},$$
because the numerator is fixed and the sandwich standard error $\sigma_{\hat\kappa}$ shrinks as $1/\sqrt{N}$. This defines a crossover sample size beyond which imputation bias exceeds the statistical error.
The remedy rests on one observation. The dead-cell mask $m$ is itself observed data. If the training simulation reproduces how cells go missing, then the likelihood ratio of the pair (observed cells $x_o$, mask $m$) is the correct quantity for inference. This holds even when missingness depends on the event, for example through tracking losses in dense jet cores. Under this view the systematic is no longer "the imputation was wrong". It is "the simulated missingness mechanism differs from the real one", and that can be bounded with the usual physics tools for systematics.
Relation to Prior Work
Several papers motivate the problem. End-to-End Jet Classification of Quarks and Gluons with the CMS Open Data classifies quark and gluon jets from Tracks+ECAL+HCAL images with a ResNet-15. ECAL and HCAL are the electromagnetic and hadronic calorimeters. The paper reports ROC AUC 0.8076, and it states that deriving uncertainties is hard because detector-level and event-level effects become entangled. Its inputs are complete grids, and it only hard-masks regions outside the jet. Systematics of quark/gluon tagging shows that parton-shower generators disagree by a factor of two on gluon-jet separation power. Those are physics-modeling systematics at generator level. Our work adds a detector-processing systematic and asks whether it is comparable in size.
From the missing-data side, three papers show that single-guess filling distorts what comes downstream. MisConv replaces the first convolution with its expected value over a Gaussian-mixture model of the missing pixels. It reports 75.7% versus 67.9% for zero-filling on SVHN, but only on natural images and only for classification accuracy. Conformal Prediction with Missing Values shows that impute-then-predict intervals under-cover when they are evaluated separately for each missingness pattern. Its proofs assume missingness is completely random. Impute Missing Entries with Uncertainty builds a distributional tabular imputer, U-VAE, that keeps multiple-imputation bias under 3% with 84–91% coverage, where the GAIN imputer falls as low as 27%.
We borrow two things from this line. The first is the evaluation idea of checking coverage separately for each mask pattern, taken from the conformal-prediction paper. The second is the use of distributional imputers as competitors. We do not borrow their conclusion that better imputation is the answer. In a simulation-based setting where the mask is itself part of the data, we expect imputation to be unnecessary rather than merely improvable. Section 2 of the research file lists no work that measures imputation bias in a physics parameter as a function of $N$, or that extends wifi-ensemble coverage to inputs with missing cells.
Research Program
First, we build a dead-cell injection layer for the CMS Open Data jet images. It covers three cases. In the fixed case, one dead-cell map applies to all events. In the run-varying case, maps are drawn from a distribution of isolated dead cells and dead clustered towers covering 0.5–5% of cells. In the event-dependent case, track pixels are dropped with a probability that rises with local hit density, which is not random with respect to the physics. The quark/gluon truth labels in the simulated sample define mixtures with a known $\kappa$.
Second, we train wifi ensembles in two forms. In the deterministic form, networks are trained on clean images and applied to zero-filled or mean-filled images, which is current practice. In the mask-marginalized form, each basis network $f_i(x_o, m)$ takes the mask as an extra input channel and is trained on images with masks drawn from the same mask distribution in both classes. Ensemble weights are fit by the original symmetrized likelihood loss. We test two weight choices: weights shared across masks, and weights fit separately for each mask class. This checks whether the span assumption must hold separately for each mask.
Third, we run pseudo-experiments over $N$ from $10^3$ to $10^6$ jets. For each one we record the bias in $\hat\kappa$, the empirical coverage of the 68% sandwich interval for each mask class, and the interval width. From these we fit the crossover $N$ for each method.
Fourth, we stress the remedy where it should fail. The test masks will come from a different mask distribution than training: denser maps, or event-dependent missingness that the simulation does not model. This measures how large the remaining mechanism-mismodeling systematic is. As a secondary study, we pixelize particle-level Pythia and Herwig jets. This gives a rough comparison between the dead-cell systematic and the generator spread reported by Gras et al.
Baselines
The naive reference is a clean-trained wifi ensemble applied to zero-filled images. This is what most pipelines would do. The second naive reference is mean-filling, using the average deposit in each cell. No reported result found on Lacuna for either method's bias or coverage on missing-cell jet images. The reference point is the clean-data wifi result: coverage converges to the nominal 68% at $M=16$ ensemble members, per the wifi paper, which reports no single scalar coverage figure. We expect zero-filling to reach nominal coverage at small $N$ and fail as $N$ grows.
The strongest imputation-based competitor is multiple imputation. Each dead cell is filled with several draws from a conditional generative imputer trained on clean simulated images, using a U-VAE-style model adapted to the grid. For each draw we fit $\hat\kappa$ and combine within-draw sandwich variance and between-draw variance using Rubin's rules, the standard formula for pooling estimates across imputations. A second strong competitor applies a MisConv-style expected first-layer convolution to the clean-trained networks. MisConv is the strongest published result for grid inputs with missing regions: 75.7% SVHN accuracy against 67.9% for zero-filling. We use the same basis architecture, the same $M=16$, the same training jets, and the same pseudo-experiments for every method. Each method gets the same budget: a fixed number of GPU-hours, split between training the imputer and training the basis networks, and an equal hyperparameter search.
The mask-marginalized estimator beats these competitors if three things hold. It must keep 68% coverage within Monte Carlo error for every mask class at all tested $N$. Its intervals must be no wider than multiple imputation's. Its crossover $N$ must lie beyond the largest sample tested, while zero-filling's falls within the tested range.
The ablation that isolates the key ingredient keeps random-mask training but removes the mask channel, so the network sees zeros without being told they mean "missing". It also swaps in a mask distribution in training that differs from the test distribution. If coverage survives without the mask channel, the gain comes from training-time exposure to missing cells, not from conditioning on the mask.
Why not just…?
Why not just impute better, with a distributional imputer and multiple imputation? This is the most serious threat, and the experiment is designed to settle it. U-VAE's tabular results (bias under 3%) suggest multiple imputation can remove most of the bias. However, the imputer's conditional distribution of a dead cell given its neighbours is learned from one generator. That makes it a new source of modeling error, inherited from the shower simulation that Gras et al. show is uncertain. If multiple imputation matches the mask-marginalized estimator in both coverage and interval width, we will report imputation as an adequate fix and recast our contribution as the crossover diagnostic.
Why not just assign a systematic by varying the fill-in value and taking the envelope? Comparing zero-filling with mean-filling is standard physics practice, but it brackets the bias without removing it. It also has no guarantee of covering the truth, because both fills are wrong in correlated ways: neither restores the variance that was collapsed by filling. The pseudo-experiments measure directly whether the envelope covers $\kappa$.
Why not just drop jets that touch a dead cell? Discarding incomplete events wastes data and, for clustered dead towers, biases the jet sample by its position in the detector. Event-dependent track loss makes the selection depend on the physics. We include it as a baseline and report both the bias in $\kappa$ it induces and the fraction of jets it discards.
Why not just retrain the network on each run's dead-cell map? With a fixed map this is exactly the mask-marginalized estimator restricted to a single mask, and it should work. It becomes impractical when maps change from run to run. Our run-varying scenario compares one mask-conditioned model against per-map retraining at matched total compute.
Decisive Checks
The central test is the plot of standardized bias against $N$. Zero-filling should rise as $\sqrt{N}$, while the mask-marginalized estimator should stay flat near zero. Coverage for each mask class must sit at 68% for the remedy and fall for zero-filling. If zero-filling shows no growing bias at realistic dead fractions, the claimed systematic floor is negligible and the premise fails.
If the remedy fails only when the test missingness mechanism differs from the simulated one, we have confirmed that the systematic has moved from the imputation to the modeling of the missingness mechanism. We will quantify that remaining error alongside the generator spread. AUC on the masked images, compared against 0.8076 on clean images, will be reported only as a secondary check on discrimination power.
References
- Frequentist Uncertainties on Neural Density Ratios with wifi Ensembles
- Frequentist uncertainties on neural density ratios with <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:msub><mml:mi>w</mml:mi><mml:mi>i</mml:mi></mml:msub><mml:mtext> </mml:mtext><mml:msub><mml:mi>f</mml:mi><mml:mi>i</mml:mi></mml:msub></mml:math> ensembles
- End-to-End Jet Classification of Quarks and Gluons with the CMS Open Data
- Systematics of quark/gluon tagging
- MisConv: Convolutional Neural Networks for Missing Data
- Conformal Prediction with Missing Values
- Impute Missing Entries with Uncertainty