Who Should Decide Which Depth Labels to Trust? Label-Side Uncertainty for Training-Only Geometry Heads

Many camera-based driving perception models are taught 3D geometry by a helper branch that exists only during training. The branch takes the backbone's features, renders them back into per-pixel depth or surface maps, and compares the result with LiDAR or with the output of a large pretrained depth model. Once training ends, the branch is thrown away, so the deployed model gains geometric understanding at no inference cost. Every such helper loss used today trusts all of its labels equally. In driving scenes, though, many labels are wrong: LiDAR points projected through a drifting calibration, pedestrians who moved between the LiDAR sweep and the camera exposure, glass, sky, and depth predictions from a foundation model that fails under glare.

The textbook fix is to let the network predict its own per-pixel uncertainty and down-weight the loss wherever that uncertainty is high. We argue this fix is badly matched to training-only branches. A self-predicted uncertainty cannot tell a corrupted label apart from a correct label that is simply hard, such as a distant cyclist. And because the branch is discarded, how well it fits its own task does not matter. The only thing that matters is the gradient it sends into the backbone. If the model is allowed to silence that gradient where geometry is hardest, it removes supervision exactly where the backbone most needs it. We propose instead to estimate uncertainty from the labels' side: measure how much independent supervision sources disagree, compute that weight with no gradient path from the student network, and test directly which scheme discounts corrupted labels while keeping hard-but-correct ones.

Core Claim

In training-only geometric supervision, the weight on each pixel or ray should come from evidence about label reliability that the student network cannot influence. A natural source of that evidence is the disagreement between LiDAR depth, a foundation-model depth estimate, and depth reprojected from a neighboring frame. The claim has two testable parts. First, weighting rendered-ray losses this way improves the deployed task (3D semantic occupancy, and secondarily monocular 3D detection) more than self-predicted aleatoric weighting does, meaning uncertainty that describes noise inherent in the data rather than gaps in the model's knowledge. Second, the gain is concentrated where self-predicted weighting does harm: small, distant, and thin structures. The second part is the diagnostic that would make the result convincing rather than incremental.

Concretely, for a camera ray $r$ let $d^L_r$ be the projected LiDAR depth, $d^F_r$ the foundation-model depth after per-image scale alignment, and $d^T_r$ the depth warped from an adjacent frame. The label-noise estimate is $s_r^2 = \mathrm{Var}(d^L_r, d^F_r, d^T_r)$, computed over whichever sources are present at that ray. The auxiliary rendering loss becomes

$$ \mathcal{L}_{aux} = \sum_r \frac{\sigma_0^2}{\sigma_0^2 + s_r^2}\, \ell(\hat d_r, \bar d_r), $$

where $\hat d_r$ is the depth rendered from the backbone's voxel features, $\bar d_r$ is the inverse-variance fusion of the available sources, $\ell$ is the base depth loss, and $\sigma_0$ is a single scale constant that is tuned once. The weight is a fixed function of the labels. The network cannot inflate it to escape the loss, so no extra term or prior is needed to stop it from claiming high uncertainty everywhere.

Positioning Against Prior Work

The methods this proposal plugs into are the training-only rendering regulators. Vampire renders intermediate 3D feature volumes to 2D depth and semantics through differentiable volume rendering and discards the renderer at test time; it applies an unweighted loss across all rays and reports negative transfer when several tasks share the regulated features. H3O adds rendered depth, normal, and semantic supervision, partly from Metric3D pseudo-labels, on top of a cheap projection architecture, and treats those pseudo-labels as clean. VoxelSplat replaces volume rendering with Gaussian splatting as a training-only loss and weights sampled points by class and speed priors rather than by any estimate of reliability. We keep their architectures unchanged and alter only how their rays are weighted. H3O is the natural host because it already consumes two independent depth sources.

The closest connecting work is Robust Depth Completion with Uncertainty-Driven Loss Functions. It learns per-pixel aleatoric uncertainty under a Jeffrey's prior so that noisy accumulated LiDAR does not corrupt training. There, depth completion is the deployed task, so attenuating hard pixels only costs depth accuracy, which is visible and measurable. In our setting the same mechanism silently removes backbone supervision. We use its loss as our main competitor and do not assume it carries over. Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics supplies the task-level version of the idea: one learned scalar variance per task. We include it to separate "how much to weight the auxiliary branch overall" from "which rays within it to trust."

On the detection side, MonoPixel motivates the proposal from the opposite direction. It predicts depth variance but uses it only to rescale detection confidence at test time, so its auxiliary depth loss stays unweighted and a variance head stays alive at inference. Its analysis finds radial depth error inside foreground objects to be the dominant failure, which is exactly where foreground masking cannot help. MonoDTR and FD3D supervise depth-aware features uniformly, and MonoDTR's authors note sensitivity on small, non-rigid objects. The pseudo-labeling literature, for example Rectifying Pseudo Label Learning via Uncertainty Estimation, estimates reliability from disagreement across the student's own prediction heads. Our estimate differs because it is external to the student, which is the variable we isolate.

Research Program

The work has three parts. The first is building the label-side noise map. LiDAR is projected per camera with dynamic objects compensated using box motion where annotations allow. Metric3D depth is aligned to LiDAR per image with a robust scale fit. Adjacent-frame depth is warped using ego-pose. From these sources we compute $s_r^2$ and cache it offline, so training cost is essentially unchanged. This matters because training cost is already a known burden of rendering branches.

The second part is a controlled corruption study, which supplies ground truth about which labels are bad. On a held-out split we inject three realistic corruptions into the LiDAR labels only: small extrinsic rotations, timing offsets on dynamic objects, and spurious returns on reflective classes. We then record where each weighting scheme places its low weights. For each scheme we measure the fraction of total weight removed from corrupted rays versus from clean rays on small or distant objects. This gives a precision-recall curve for "discounting corruption" that is independent of final accuracy. The prediction is that self-predicted variance discounts both corrupted rays and clean hard rays, while label-side variance discounts mostly corrupted ones.

The third part is downstream training: H3O on Occ3D-nuScenes as the main testbed, and a MonoDTR-style detector on KITTI as a second regime where only LiDAR and Metric3D are available.

Baselines

The naive reference point is the host method exactly as published, with uniform per-ray weights and its original fixed auxiliary weight. On Occ3D-nuScenes, H3O reports 28.45 mIoU (H3O), Vampire reports 28.3 mIoU (Vampire), and CTF-Occ reports 28.53 mIoU (Occ3D). We will reproduce H3O in our own environment and compare against our reproduced number, not the paper's. No reported RayIoU for H3O was found on Lacuna. On KITTI, the uniform reference is MonoDTR at 15.39% moderate Car $AP_{3D}$ (MonoDTR).

The strongest competitor is self-predicted per-ray aleatoric weighting: a variance head attached to the rendering branch and trained with the Jeffrey's-prior loss of Robust Depth Completion with Uncertainty-Driven Loss Functions, and also with the Gaussian-likelihood form. It is the strongest because it is the established answer to noisy LiDAR depth and needs no extra label sources. No reported result found on Lacuna for this loss inside a training-only occupancy or detection branch; its published numbers, 190.88 MAE and 1.98 iRMSE on KITTI depth completion, are for depth completion as the deployed task. The variance head will be discarded with the branch, so inference cost matches.

We also run task-level homoscedastic weighting following Kendall et al., and a combination of that with our per-ray weights. The key ablation removes the stop-gradient: we feed the same disagreement features into a learned weighting head that the backbone can influence. This isolates whether the benefit comes from the extra label sources or from denying the student control over its own supervision. A second ablation uses only the fused target $\bar d_r$ with uniform weights, which separates "better targets" from "weighting."

Fairness rules apply to every arm. All runs use the same backbone, schedule, data, and pseudo-label cache. Each arm gets the same tuning budget of eight runs over its own free constants: $\lambda_{aux}$ for uniform weighting, the prior strength for Jeffrey's, and $\sigma_0$ for ours. We report three seeds. Beating the strongest competitor means a mean mIoU gain larger than the seed spread, together with a non-negative change on every distance band and small-object class. A gain confined to large, near classes would not count.

Why not just…?

Why not just use a robust loss such as Huber or Laplace on every ray? A robust loss caps the influence of large residuals. Large residuals, however, occur both on corrupted labels and on correct labels the backbone has not yet learned, which is the same confusion we attribute to self-predicted variance. This is a real threat, because robust losses are cheap and often competitive. We include Huber with a tuned threshold as an arm under the same budget, and the corruption study shows directly whether it discounts clean hard rays.

Why not just mask disagreeing pixels with a threshold? Binary agreement masking is the discrete version of our weight, so if it matches us the honest conclusion is that the label-side signal, not its continuous form, is what matters. We run it with a tuned threshold. We expect continuous weighting to do better only where sources are partially present, for example far ranges with sparse LiDAR. If it does not, we will report masking as the recommended method.

Why not just tune $\lambda_{aux}$ or use class and speed priors as VoxelSplat does? A global scalar cannot treat one ray differently from another, and VoxelSplat's class and speed priors (VoxelSplat) weight a whole category up or down regardless of whether a particular label is broken. Occ3D's OHEM ablation (traffic-cone mIoU falling from 14.16 to 10.03 without it, Occ3D) shows that emphasizing hard examples helps. That is a reason to keep hard rays weighted, not a reason to weight broken ones. We keep the host's class weighting in every arm, so any gain we report comes on top of it.

Proposed Experiments and Decisive Checks

The main experiment trains H3O on Occ3D-nuScenes under every baseline arm. It reports mIoU and RayIoU overall, per class, and by distance band, plus Vampire's multi-head setting to test whether label-side weighting reduces the negative transfer its authors reported. The secondary experiment repeats the comparison on KITTI monocular detection with a MonoDTR-style training-only depth head, reporting moderate and hard $AP_{3D}$ for Car, Pedestrian, and Cyclist. It also checks whether the gain approaches what MonoPixel gets from test-time variance while keeping no extra component at inference.

The decisive check is the corruption study combined with performance by distance band. The proposal succeeds if two things hold: label-side weighting removes markedly more weight from injected corruptions than from clean distant or small-object rays, and that difference predicts downstream gains on exactly those strata. It fails, and we will say so, if self-predicted Jeffrey's weighting or plain Huber matches it on both measures. In that case, the extra label sources would not be worth the pipeline complexity.