Low-Rank Moment Propagation for CNNs on Images with Missing Regions

When part of an image is missing, for example a glare spot on a retinal photograph or an occluded patch in a street photo, the missing pixels are uncertain together rather than one at a time. If one pixel in a hole is bright, its neighbours are probably bright too. Existing deterministic uncertainty methods for convolutional networks discard exactly this structure. These methods are called moment propagation: instead of sampling, they push a mean and a variance for every activation through the network in a single pass. They keep one variance per unit and treat neighbouring units as independent. A convolution sums neighbouring pixels, and the variance of a sum of positively correlated values is much larger than the sum of their variances. So this independence assumption should make the network systematically misjudge its uncertainty in the cases where uncertainty matters most.

We propose to carry a small set of "uncertainty directions" through a standard convolutional network alongside the mean. Each direction is an image-shaped tensor that describes one coherent way the missing content could vary. Linear layers act on these directions exactly. Nonlinearities rescale them using an identity that is exact for Gaussian inputs. A cheap per-unit correction keeps each unit's own variance exact. The starting directions already exist: a factor-analyzer model of the missing pixels, as used in MisConv, produces precisely a mean plus a few factors. The method therefore turns MisConv's first-layer trick into uncertainty that survives through the whole feature hierarchy. Its cost grows with the number of directions $k$ rather than with the square of the feature-map size.

Core Claim

The claim is that most of the error in diagonal moment propagation under spatially structured missingness comes from dropped correlations, and that a rank-$k$ correction recovers most of it. The rank needed should be small, on the order of the number of factors in the input model. We expect the gain to grow with how spatially coherent the missingness is: large for square patches, near zero for scattered single-pixel dropout. That second prediction is the control that makes the claim falsifiable.

Concretely, every layer's activation distribution is represented as mean $\mu$ plus covariance $\Sigma \approx FF^\top + \mathrm{diag}(d)$. Here $F$ holds the $k$ direction tensors as columns and $d$ is a nonnegative per-unit residual variance. A convolution or other linear map $W$ updates these as $F \leftarrow WF$ and $d \leftarrow (W \circ W)\,d$, where $\circ$ is the elementwise product. The first update is exact. The second drops only the off-diagonal part of the residual. Implementing $WF$ means running the same convolution on $k$ extra channels-of-images in one batched call.

For a ReLU, Stein's lemma gives the exact covariance between a Gaussian input and the rectified output. Writing $s$ for the per-unit standard deviation and $\Phi$ for the standard normal cumulative distribution function, the update is

$$F \leftarrow \mathrm{diag}\big(\Phi(\mu/s)\big)\,F, \qquad d \leftarrow \max\big(0,\ v_{\text{ReLU}}(\mu,s) - \lVert F_{i,:}\rVert^2\big).$$

The first update is exact for the cross-covariance and approximate for the output-output covariance. In the second, $v_{\text{ReLU}}$ is the closed-form variance of a rectified Gaussian, so each unit's marginal variance is matched exactly, as in standard assumed density filtering. Max-pooling uses the analogous selection-probability weighting from the Gaussian max formulas. Output variance then decomposes into one contribution per input direction plus a residual. This gives an image-level version of the uncertainty attribution that was previously shown only for tabular data.

Relation to Prior Work

Four lines of work motivate or supply the method. Lightweight Probabilistic Deep Networks runs assumed density filtering through full convolutional networks and shows that full propagation beats a probabilistic output head alone. However, it tracks only diagonal variance, and the research file flags this as at odds with how convolutions pool neighbours. We keep its whole-network propagation and add the low-rank term.

MisConv models missing pixels with a deep mixture of factor analyzers and computes the expected first convolution analytically. It then collapses back to a point tensor, so deeper layers never see the uncertainty. We reuse its input model and continue its factors past layer one. Its mixture structure is handled by propagating each mixture component separately and combining at the output.

Probabilistic Numeric Convolutional Neural Networks keeps full Gaussian-process covariance through every layer. It shows that removing intermediate variance hurts accuracy, which supports propagating beyond the first layer. Its continuous machinery, though, was demonstrated only on small benchmarks. We stay within standard discrete backbones.

An Analytic Solution to Covariance Propagation in Neural Networks derives full inter-neuron covariance propagation, which is infeasible at convolutional scale. Our low-rank form is a structured approximation to it. Exact piecewise-linear moment formulas (Analytic Expressions for Probabilistic Moments of PL-DNN With Gaussian Input) supply the marginal terms.

Understanding Deep Neural Networks through Input Uncertainties traces output variance back to input features in tabular models and explicitly leaves image features open. Our per-direction variance decomposition is one concrete answer to that open question.

Two results constrain the design. Assumed Density Filtering Methods for Learning Bayesian Neural Networks reports that online filtering updates underestimate variance over long training. For that reason we use moments only in the forward pass and train weights by ordinary gradient descent. The lightweight-networks paper reports a threefold runtime overhead for diagonal propagation. We must therefore report wall-clock cost honestly, since rank $k$ adds roughly $k$ extra convolution passes.

Research Program

We will first build the propagation layers in a standard deep-learning library. These cover convolution, batch normalization at inference, ReLU, max-pooling, and global average pooling. The classifier head treats the logits as Gaussian, and its loss is the expected cross-entropy computed with a probit-style approximation. Networks are trained end-to-end on masked images. The input moments come from a MisConv-style factor-analyzer model fit on unmasked training images and conditioned on the observed pixels.

The analysis has three parts. First, we measure moment fidelity layer by layer against a reference computed from 1,000 samples drawn from the conditional input model and pushed through the same fixed network. We report the log-ratio of predicted to reference variance per unit, and the error in pairwise covariance between spatial neighbours. Second, we measure downstream accuracy and calibration. Third, we test whether per-direction variance attributions localize the occluded content that actually drove errors.

Baselines

The naive reference point is zero-imputation with a standard CNN trained on masked images. On SVHN with 25% square patches, MisConv reports 67.9% accuracy for zero-imputation and 37.6% for partial convolution (MisConv).

The strongest existing method on this exact setting is MisConv itself, which reports 75.7% accuracy on SVHN with 25% patch removal (MisConv). We will re-run it with the same backbone, the same factor-analyzer input model, the same masks, and the same tuning budget. Beating it means higher accuracy at equal backbone. More importantly, it means clearly better calibration and error detection, because MisConv outputs no variance at all.

The ablation that isolates our key ingredient is diagonal propagation in the style of Lightweight Probabilistic Deep Networks fed the identical input moments, which amounts to rank $k=0$. If low rank adds nothing over this, the core claim fails. A second ablation keeps the low-rank input but collapses it to diagonal after the first block. This locates where correlations stop mattering.

The real competitor for uncertainty is multiple imputation. In this approach, $S$ samples are drawn from the same conditional input model, each is run through the standard CNN, and the results are averaged. It is matched to our cost by setting $S = k+2$ forward passes. A deep ensemble of equal total compute is also included. Every method gets the same training data, mask distribution, epochs, and hyperparameter search size. Wall-clock time and memory are reported alongside accuracy.

Why not just…?

Why not just draw a few imputations and average the network outputs? This is the most serious alternative. Sampled imputations capture correlations automatically, and they cost about the same as rank-$k$ propagation at matched passes. Our expected advantages are narrower. With only a handful of samples, the Monte Carlo variance estimate is itself noisy, while ours is deterministic. Our training loss also has no sampling noise. Deterministic Moment Propagation for Fast Probabilistic Deep Learning summarizes evidence that removing sampling noise makes probabilistic networks easier to train. However, that evidence comes from weight uncertainty, not input imputation. The matched-compute comparison settles this question. If multiple imputation at $S=k+2$ matches our negative log-likelihood and error-detection scores, we will report that the low-rank method's value is limited to determinism and per-direction attribution, and we will not claim an accuracy or calibration win.

Why not just use diagonal propagation, which already works? Diagonal propagation is exactly correct when inputs are independent, and our random-pixel-dropout control is designed to show this. For coherent holes, a convolution over a patch whose pixels move together has variance that grows with the square of the patch area under full correlation, but only linearly under independence. We therefore predict strong variance underestimation at the first layers, which the fidelity measurements will show directly.

Why not just put a probabilistic head on a MisConv network? The lightweight-networks paper found that full propagation beat an output-only probabilistic layer in error-uncertainty correlation on optical flow. The PNCNN ablation found that dropping intermediate variance hurt accuracy. Neither result was on missing-data images, so we include this head-only variant and expect it to lose mainly on error detection.

Proposed Experiments and Decisive Checks

The core experiments use SVHN, CIFAR-10, and MNIST with three mask families of matched missing fraction. The families are 25% square patches as in MisConv, stripes, and independent pixel dropout. For CIFAR-10 and MNIST under patch masking, no reported result was found on Lacuna; the file states that MisConv's numbers there exist only in plots. We vary rank $k \in \{0, 2, 4, 8, 16\}$. Metrics are accuracy, negative log-likelihood, expected calibration error, AUROC for detecting misclassifications from predicted variance, and the layer-wise fidelity measures described above.

The decisive check is a ranking. The gap between rank $k>0$ and diagonal propagation should be largest for patches, intermediate for stripes, and near zero for pixel dropout. It should also saturate at a small $k$. If the gap is flat across mask types, the correlation hypothesis is wrong.

The application test moves to automated retinal grading on the Kaggle Diabetic Retinopathy fundus images. The reference result there is 73.2% accuracy for the discovery-radiomics sequencer (Discovery Radiomics with CLEAR-DR), which reports that artifacts distract the model. We add synthetic glare and occlusion masks and evaluate selective prediction, meaning accuracy when the most uncertain images are deferred. We also check whether per-direction attributions point at the masked regions more often than occlusion heatmaps do.

A stretch experiment simulates slice dropout in 3D brain MRI from the Alzheimer's Disease Neuroimaging Initiative, using the VoxCNN setting of Residual and Plain Convolutional Neural Networks for 3D Brain MRI Classification. That paper reports an AUC of 0.88 for Alzheimer's disease versus normal controls. We will ask only whether uncertainty flags corrupted scans. We do not expect it to fix the weak early-impairment separation, where that paper reports an AUC of 0.57.