Learning Where the Variance Head May Reach Into the Backbone

Many regression networks predict two things for each input: an answer and an estimate of how noisy that answer is likely to be. A blurry ultrasound frame or a chaotic weather state should get a wide error bar, and a clean one a narrow error bar. Modeling noise that changes from input to input is called heteroscedastic regression. The usual recipe adds a second output, a "variance head," to the network and trains both outputs jointly. Current practice offers two bad options. If the variance head is allowed to change the whole shared network during training, optimization becomes unstable: the network can learn to call hard examples "noisy" and stop fitting them. If the shared network is frozen after training on the answer alone, and a variance head is fitted on top afterwards, training is stable. But the frozen features may already have discarded the cues that signal noise. A human designer currently decides by hand where the variance head attaches and how far its training signal reaches.

This note proposes treating that choice as a small architecture-search problem with two separate knobs per layer. The first knob is which backbone layers the variance head reads from. The second is how strongly the variance head's training signal is allowed to write back into each of those layers. Pure post-hoc fitting (read everywhere, write nowhere) and full joint training (write everywhere) are the two corners of this space. The claim is that the well-calibrated models lie in between: the variance signal is allowed to reshape a few intermediate layers and nothing else. That middle region can be found automatically, with roughly the cost of one extra training run.

Core Claim

Two papers motivate the claim from opposite sides. Deep Heteroskedastic Regression: Post-Hoc Variance Estimation from Latent Representations shows that joint training stalls. It also shows that the final layer of a network trained for the mean compresses away noise information, while intermediate layers keep it. Its remedy is to freeze the backbone and fit linear variance heads on hand-picked intermediate layers. On QM9 this matches or beats end-to-end methods such as β-NLL in likelihood and calibration. Understanding Pathologies of Deep Heteroskedastic Regression shows that joint models jump between two failure modes as regularization changes: noise collapsing to zero, or the mean collapsing to a constant. It finds the calibrated region by tuning regularization along a single diagonal.

Neither paper asks whether structure can do the stabilizing. Freezing is a structural choice in disguise: it sets every write gate to zero. The post-hoc remedy therefore silently assumes that a mean-only backbone already contains every feature the variance needs. That assumption fails whenever noise is driven by something the mean does not depend on, such as image blur, sensor saturation, or a nuisance covariate.

The proposal has three parts. Attach a variance head that reads a gated combination of backbone layers. Scale the variance loss's gradient into each layer $l$ by a learned gate $\beta_l \in [0,1]$. Learn the read gates and the write gates by optimizing held-out negative log-likelihood. Concretely, the update that backbone layer $l$ receives is

$$ g_l = \nabla_{\theta_l}\mathcal{L}_{\text{mean}} + \beta_l \,\nabla_{\theta_l}\mathcal{L}_{\text{NLL}}, $$

where $\theta_l$ are that layer's weights. $\mathcal{L}_{\text{mean}}$ is the squared-error loss on the predicted mean. $\mathcal{L}_{\text{NLL}}$ is the Gaussian negative log-likelihood seen through the variance head, with the mean treated as fixed inside it. The mean head itself is always trained by squared error alone, so the variance head can never buy a lower loss by giving up on hard examples. This choice targets the coupling that the pathologies paper identifies.

Research Program

The method borrows its search mechanics from SparseMask: Differentiable Connectivity Learning for Dense Image Prediction. SparseMask relaxes every encoder-to-decoder connection to a continuous weight, pushes the weights toward zero or one with a sparsity loss, and prunes the result. It does this for deterministic segmentation and spends about 18 GPU-hours on search. Here the "decoder" is the variance head, and each connection carries a second weight: the write gate. Read gates $\alpha_l$ and write gates $\beta_l$ are updated on held-out NLL, and network weights on training data, in the two-level alternating style of DARTS: Differentiable Architecture Search.

Two refinements keep the search honest. The write gates enter only through a gradient-scaling operation, so in the forward pass they act as a hyperparameter of the learning dynamics. Their outer-loop gradient is obtained by a one-step lookahead, as in DARTS. After search, gates are binarized and the discovered configuration is retrained from scratch. The deliverable is not a single trick but a map: for each task, which layers the variance needs to read and which it needs to reshape. We then check whether that map is stable across random seeds and backbones.

This differs from existing uncertainty-aware search in what it searches for. μDARTS adds learned dropout and penalizes variance across stochastic passes. This targets model (epistemic) uncertainty in classification and steers search toward flat minima, but it needs many forward passes and never outputs a per-input noise level. NADS searches a distribution over flow-model architectures for out-of-distribution detection. FNAS uses uncertainty only to decide which candidate architectures to skip training. None of the three searches over how a noise-predicting output couples to a shared network.

Neural Architecture Search for Sentence Classification with BERT shows that searched heads beat linear heads on frozen backbones, but only for single-output classification. CUPID attaches a plug-in to frozen intermediate features, which is a fixed read-only design. The present search space contains these frozen designs as its zero-write corner.

Baselines

The naive reference point is a standard dual-head network split at the penultimate layer and trained jointly with plain Gaussian NLL. This is the hand-designed default used, for example, in Bayesian Image Quality Transfer with CNNs and Probabilistic Multilayer Perceptrons for Wind Farm Condition Monitoring. It shows how bad unconstrained coupling is on each benchmark.

The strongest existing method is the post-hoc multi-layer variance ensemble from Deep Heteroskedastic Regression. It is the strongest competitor because it already reads from intermediate layers, is stable, and is cheap. We run it with the same backbone (PaiNN on QM9), the same hold-out split for the variance heads, and both of its aggregation variants (concatenating all layers, and the Gaussian mixture). We give it a fair upgrade: a learned softmax weighting over layers fitted on the same hold-out data. This ensures any win by our method comes from the write gates and not from better reading. I found no reported QM9 numerical result for this method on Lacuna; the file states its scores appear only in figures. We will therefore reproduce it, and count a win as lower test NLL and expected calibration error on a majority of the ten QM9 targets, with mean absolute error no worse than the frozen backbone's.

The second strong competitor is joint training with β-NLL and with the diagonal regularization search of Understanding Pathologies of Deep Heteroskedastic Regression. We give it the same GPU-hour budget as our search plus retraining, spent on points along its diagonal. For depth estimation, the reference is the Self-Teaching model from On the Uncertainty of Self-Supervised Monocular Depth Estimation, which reports an absolute relative error of 0.087 on KITTI against 0.090 for Monodepth2. We run it from mattpoggi/mono-uncertainty. Beating it means a lower area under the sparsification error curve at no worse absolute relative error.

Two ablations isolate the key ingredient. The first is read-gates-only search with all write gates fixed at zero, which tests whether writing matters at all. The second is an exhaustive sweep that lets the variance gradient into exactly the last $k$ backbone blocks, for every $k$, with matched compute. This "simple depth cutoff" is the most dangerous ablation, discussed below.

Why not just…?

Why not just fit post-hoc variance heads on a frozen backbone? On QM9 it may well be enough, and we expect a tie there. Its weakness is structural: it can only use what squared-error training kept. We therefore add a controlled test the post-hoc paper does not run. We take QM9 and UCI targets and inject noise whose scale depends on a nuisance input feature or graph attribute that has no bearing on the mean. If post-hoc fitting matches our method even there, then intermediate layers keep nuisance information anyway, and the write gates are unnecessary. That would be a clean negative result, and we would report it as such.

Why not just tune regularization along the diagonal? The pathologies paper shows this finds calibrated models on Concrete, Housing, Power, Yacht and ClimSim. It still trains a fully coupled network and must re-sweep for every dataset. Our comparison gives it equal compute on the same datasets. If it wins, the conclusion is that coupling problems are a regularization issue rather than a topology issue. We also test the combination, gates plus diagonal tuning, because the two may be complementary rather than rival.

Why not just let the variance gradient into the last few layers, picking the count by grid search? This is the real threat. If the learned gates always converge to a contiguous band of late-intermediate layers, a one-dimensional sweep over $k$ recovers the same model with no search machinery. The matched-compute sweep ablation settles it. If the sweep ties, the contribution shrinks to an empirical finding: "let the variance reshape layers from depth $k^*$ onward." We would then study how $k^*$ scales with depth and task instead of selling the search.

Why not just train a separate small network for the variance from the raw input? A fully decoupled variance network avoids coupling entirely, but it relearns features the backbone already has. This should hurt most on large inputs like KITTI images and on small hold-out sets; the post-hoc paper needed only 200 points, which a from-scratch network cannot use. We include it at matched parameter count.

Proposed Experiments

The experiments run in order of decisiveness. First comes the injected-nuisance-noise variant of UCI regression and QM9. Here the ground-truth noise scale is known, so we can measure the correlation between predicted and true standard deviation directly, alongside NLL and expected calibration error. The prediction is that post-hoc fitting fails on this variant, full joint training destabilizes, and the searched gates succeed by opening write access in a few intermediate layers.

Second, we run unmodified QM9 with PaiNN, UCI including the gap splits used in Benchmarking the Neural Linear Model for Regression, and ClimSim. These show whether the method costs anything when post-hoc fitting is already sufficient. Third, we run KITTI monocular depth, where blur and occlusion are plausible nuisance-driven noise sources.

Across all three, we probe each backbone layer with a linear predictor of the true or empirical residual magnitude. This tells us whether the learned write gates correspond to layers where squared-error training had erased noise information. It is a direct test of the representation-collapse story rather than an assumption of it. We also report seed-to-seed agreement of the binarized gates and total GPU hours for every method.

The proposal is supported if four things hold: the nuisance experiment separates the methods, the depth-cutoff sweep does not tie the search, and the gate pattern is reproducible across seeds. If any of these fails, the note's claim reduces to the corresponding simpler method, and we say so.