Closed-Form Attribution Variance and the Losses That Shape It

A saliency map is a heatmap that says which input features a neural network's prediction depended on. When the input is slightly noisy, as with any real sensor, the heatmap can change a lot. Today that instability is measured by brute force: add noise many times, recompute the map each time, and look at the spread. We propose computing the mean and variance of a gradient-based saliency map in one deterministic pass. The method pushes the input noise distribution through the network analytically, layer by layer, instead of sampling it. In a ReLU network (one whose units output $\max(0,z)$), the input gradient is a product of weight matrices and on/off gates. So the spread of the map is determined by which units have pre-activations that sit near zero under the noise, where they flip on and off. The analytic pass gives this "gate-flip mass" per unit and per layer. That tells us where in the network an explanation becomes unstable, not just whether it does.

The second half of the proposal uses this tool to answer a training-side question that nobody has tested. Does the loss function's stationary-point structure make explanations more stable? A stationary point is a setting of the weights where the training gradient vanishes. Cross-entropy has no finite such point on the correct side of the boundary, while some alternative losses do and also center the decision boundary between classes. Prior work showed that explanations wobble near curved or close decision boundaries. Separately, it showed that losses with a finite stationary point move the boundary away from the data. We test whether these two findings are the same phenomenon. We also separate "has a finite stationary point" from "centers the boundary," because several cheap losses have the first property without the second.

Core Claim

Our first claim is about computation. For ReLU networks under Gaussian input noise, the expected input gradient and its variance can be approximated in closed form. They come from the propagated mean and covariance of pre-activations, extended to the Heaviside gates that form the backward pass. At matched accuracy, this deterministic estimate should be cheaper than Monte Carlo SmoothGrad-style averaging. It is also fully reproducible.

Our second claim is about mechanism. Per-instance attribution variance should be predicted by the total gate-flip mass. This is the summed probability that hidden units change sign under the noise. It should be at least as predictive as distance to the nearest decision boundary.

Our third claim is about training. Losses that center the boundary should reduce gate-flip mass and attribution variance at matched clean accuracy. A finite stationary point alone may not be enough to produce this effect.

How the Pieces Fit

The mechanism comes from analytic uncertainty propagation. An Analytic Solution to Covariance Propagation in Neural Networks gives sample-free propagation of full means and covariances through ReLU, GELU and Heaviside units. On MNIST it reports mean and variance ratios to Monte Carlo near 1.0, and a 4th-order Taylor maximum absolute error of 3.740 × 10⁻⁴. The paper is motivated by black-box overconfidence, but it never produces an attribution. We reuse its Heaviside treatment, because the ReLU gradient is exactly a Heaviside gate.

Several results constrain the design. Input uncertainty propagation through trained neural networks shows that ReLU zero-regions create point masses and make KL-based fitting blow up. A Complete Characterisation of ReLU-Invariant Distributions shows that no practical parametric family survives ReLU layers. We therefore never claim to track the full output density. We track only the first two moments of pre-activations, which are weighted sums and plausibly closer to Gaussian. From those moments we derive gate probabilities. Errors from this approximation are measured directly, not assumed away.

Exact Upper and Lower Bounds for the Output Distribution of Neural Networks with Random Inputs gives exact output-CDF bounds on small tabular networks. It reports 4.0 seconds versus 504 seconds for a competing method on a 2D Diabetes task, and finds that even 10⁸ Monte Carlo samples fell outside its bounds. We use it as a ground-truth check on low-dimensional problems. Its exponential scaling rules it out as the main method.

For a ReLU network with logit $f$, the input gradient is $\nabla_x f = W_1^\top D_1 W_2^\top D_2 \cdots W_L^\top w$. Here $W_l$ is the weight matrix of layer $l$, $w$ is the readout vector, and $D_l$ is the diagonal matrix of gates $\mathbf{1}[z_l>0]$ on that layer's pre-activations $z_l$. Under input noise, the simplest factorized approximation replaces each gate by its expectation:

$$\mathbb{E}[\nabla_x f] \approx W_1^\top \bar D_1 \cdots W_L^\top w, \qquad \bar D_l = \mathrm{diag}\big(\Phi(\mu_l/\sigma_l)\big),$$

In this equation, $\mu_l$ and $\sigma_l$ are the propagated pre-activation means and standard deviations, and $\Phi$ is the standard normal CDF. The research contribution is the next step beyond this mean-field form. Gates across layers are correlated, so the propagated covariance must be used to correct both the mean and the second moment of the gradient. The per-unit gate-flip mass $\Phi(\mu/\sigma)\,(1-\Phi(\mu/\sigma))$ is a byproduct and serves as our localization signal.

The stationary-point question comes from Stationary Point Losses for Robust Model. That paper argues that cross-entropy's missing stationary point drives weights toward infinity and leaves the boundary near training samples. Its anchored "stationary point" loss creates a finite optimum and provably centers the boundary. On CIFAR-10 with ResNet-18 it reports PGD accuracy of 0.14% for cross-entropy versus 50.96% for SP Focal loss.

Two attribution papers supply the motivation: Boundary-Aware Uncertainty for Feature Attribution Explainers and Uncertainty in Additive Feature Attribution methods. Both find explanation instability rising near curved or close boundaries. The second predicts stable instances from distance to the second-nearest boundary, with 89.2% precision and 95.1% recall. Neither paper changes the training loss, and Gao et al. never look at explanations. We take one constraint from homogeneity. A bias-free ReLU network is positively homogeneous, so scaling all weights uniformly leaves logit-gradient directions unchanged. "Weights diverging" therefore cannot by itself explain unstable logit attributions. If losses matter, it must be through where the boundary and the hidden-unit kinks sit relative to the data. That is what gate-flip mass measures.

Research Program

The first stage builds the estimator. It propagates means and covariances forward, following Wright et al. It then propagates first and second moments of the gradient backward, using the gate probabilities and pairwise gate covariances. We implement three tiers of approximation. The first is the factorized mean-field form above. The second is a diagonal-plus-low-rank covariance that keeps cost near linear in layer width. The third is full covariance, which is feasible only for multilayer perceptrons and small convolutional networks.

Batch normalization is folded into affine layers at test time. Max-pooling is handled by its own gate. We will not claim coverage of attention layers. Verification uses two references: a 10⁵-sample Monte Carlo estimate, and, on tabular networks, the exact CDF bounds of Kofnov et al. applied to individual gradient coordinates.

The second stage tests the mechanism. For every test instance we compute three quantities: gate-flip mass per layer, input-space distance to the decision boundary, and attribution variance. We then ask which quantity predicts instability. We also ask whether the layers carrying most of the flip mass also carry most of the variance, which we check by freezing gates in a single layer and re-measuring.

The third stage is the loss study. We train matched families of models on the same data and architecture with five losses. These are plain cross-entropy, cross-entropy with weight decay, label smoothing, SP Focal loss, and a cross-entropy run with early stopping at matched accuracy. Weight decay and label smoothing are the important controls, because both give a finite stationary point without the boundary-centering argument of Gao et al.

Baselines

The naive reference is the vanilla gradient at the clean input, with no averaging. It has zero cost beyond one backward pass and zero estimated variance. It shows how far the noisy-input mean actually moves from the clean saliency map. If the two agree closely, the whole exercise is moot.

For the measurement claim, the strongest baseline is Monte Carlo averaging of input gradients under the same Gaussian noise. This is the estimator behind SmoothGrad: removing noise by adding noise, with its variance computed from the same samples. It is the strongest competitor because it is unbiased and converges to exactly the quantity we approximate. It is run with an identical noise scale and model, at sample counts from 4 to 10⁴, on the same GPU. We report error versus wall-clock time against the 10⁵-sample reference. Beating it means reaching a given error on mean and variance in less time than Monte Carlo needs for the same error, across noise scales. Matching it with determinism alone is the weaker claim. I found no reported result on Lacuna for SmoothGrad attribution-variance error on these benchmarks, so we establish the curve ourselves.

The second baseline is explanation uncertainty from deep ensembles and MC-Dropout paired with Guided Backpropagation and Integrated Gradients, as benchmarked in Sanity Checks for Explanation Uncertainty. That paper reports pass/fail outcomes on California Housing and CIFAR-10 but no numerical values. These baselines measure a different uncertainty, over weights rather than inputs. We therefore compare them only on the sanity checks and on compute, with five ensemble members and 50 dropout samples.

The third baseline is GPEC from Hill et al., the most direct existing method linking boundary geometry to explanation uncertainty. It runs with its published boundary-sampling settings on Census, German Credit and MNIST. We found no reported numerical result on Lacuna for GPEC. Beating it means equal or better rank correlation with Monte Carlo attribution variance at lower cost, plus applicability to California Housing regression, which GPEC cannot handle.

For the mechanism claim, the baseline predictor is the distance-to-boundary rule of Madaan et al. Our flip-mass predictor must exceed its precision and recall at identifying stable instances on the same models. The ablation that isolates our key ingredient replaces cross-layer gate covariance with the factorized mean-field form. If the factorized form matches the full estimator, the covariance machinery is unnecessary and we will say so.

For the loss study, all five losses share architecture, augmentation, epochs and a fixed hyperparameter search budget per loss. Models are compared at matched clean accuracy within one point.

Why not just…?

Why not just run SmoothGrad with more samples? This is the real threat. Monte Carlo is unbiased and trivially parallel, and if a few dozen samples reach the analytic estimator's accuracy, the estimator offers only determinism. The error-versus-time curves in the first experiment settle this directly. We expect the analytic method to win on variance more than on the mean, because second moments converge slowly under sampling. Kofnov et al. show Monte Carlo misbehaving even at 10⁸ samples for output distributions. If the method does not win, the remaining contribution is localization: flip mass per unit costs one pass analytically but needs per-unit gate statistics from many samples otherwise. We would narrow the claim accordingly.

Why not just use ensembles or MC-Dropout? They pass Valdenegro-Toro and Mulye's sanity checks. However, they quantify weight uncertainty, cost several networks or many passes, and the authors themselves caution that passing does not guarantee correctness. We run both sanity checks on our estimator. Under weight randomization from the input upward, its variance should rise. Under label randomization, it should change in the same direction those methods do. If it fails these checks, the analytic route is not trustworthy, whatever its speed.

Why not just use GPEC? GPEC models boundary-driven explanation uncertainty. It does so with a post-hoc Gaussian process that requires dense boundary sampling, is limited to classification, and sees nothing inside the network. Our comparison on Census, German Credit and MNIST tests whether a layer-internal quantity predicts the same instability more cheaply. California Housing tests the regression case GPEC cannot address.

Why not just use weight decay or label smoothing instead of a special stationary-point loss? This is why both are in the loss study. Both give the training objective a finite stationary point. If they stabilize attributions as much as SP Focal loss does, the operative property is the finite stationary point, and the cheap fixes suffice. If SP loss alone helps, boundary centering is what matters. If none of them differ from cross-entropy at matched accuracy, the hypothesized link fails, and that negative result is worth reporting.

Proposed Experiments

The fidelity experiment runs on MLPs for California Housing, Iris, Wine and Diabetes, MLPs and a small CNN for MNIST, and a small CNN and ResNet-18 (low-rank tier only) for CIFAR-10. We sweep noise standard deviation from 1% to 20% of the input range. Fidelity is measured as relative error of the attribution mean and variance against the 10⁵-sample reference, and as cosine similarity of the mean maps. On the tabular networks we additionally check per-coordinate gradient values against exact bounds. We expect the factorized approximation to degrade as noise grows and more gates flip jointly. Showing where the covariance correction becomes necessary is itself a result.

The mechanism experiment computes per-instance Spearman correlation between attribution variance and each of two predictors: total flip mass and boundary distance. It also reports stable-instance precision and recall against Madaan et al.'s rule. A gate-freezing intervention tests whether per-layer flip mass identifies the layers responsible for the variance.

The loss experiment trains five seeds per loss on two-moons, MNIST and CIFAR-10. For each run we report clean accuracy, input-space margin, total flip mass, attribution variance under fixed noise, and, as a link to Gao et al., PGD accuracy. The decisive check is the ordering at matched accuracy. SP Focal below label smoothing and weight decay, which are in turn below cross-entropy, supports boundary centering as the mechanism. A tie among the three finite-stationary-point losses points to the stationary point itself. No separation from cross-entropy refutes the training-side claim. The flip-mass mediation test asks whether controlling for flip mass removes the loss effect on attribution variance. This tells us whether hidden-unit kinks are the channel through which the loss acts.