A self-driving car's LiDAR network usually serves several tasks from one shared feature extractor: drawing boxes around cars (detection) and labeling every point as road, sidewalk or pedestrian (segmentation). When fog, rain or a failing sensor corrupts the input, both tasks degrade together, and neither tells you where. We propose a simple change to the attention layer inside that shared backbone. Each 3D cell of the scene (a "voxel") carries a confidence number alongside its features, and attention pools features weighted by those confidences. A low-confidence voxel then contributes less to its neighbours, and every output voxel reports how much trustworthy evidence went into it. Because this confidence lives in the backbone rather than in any task head, it should flag errors for tasks it was never trained on. Because it uses linear attention, which avoids comparing every voxel with every other voxel, it should add almost no runtime.
The concrete bet is this. A precision term (inverse variance) can be carried through the associative computation of scattered linear attention at the cost of one extra vector per window. Trained only through a detection head, this term should still rank segmentation errors correctly on clean and corrupted nuScenes. It should do so better than a variance head bolted onto a deterministic backbone, and at a small fraction of the cost of sampling-based methods. If the cross-task transfer of uncertainty fails, the idea reduces to an expensive variance head, and the experiments are built to reveal that quickly.
Where this sits
Three lines of work each supply one piece and leave the joint problem open.
ScatterFormer supplies the method. It applies linear attention to variable-length sets of voxels inside irregular windows. Linear attention replaces the softmax with feature maps $\phi$ so that a window's keys and values can be summed into a small state once, rather than compared pairwise. This removes the sorting and padding that costs window transformers up to 24% of backbone latency, and the model runs at 23 FPS on an A100. Its attention is fully deterministic, every voxel counts equally, and it was evaluated only on detection.
Kalman Linear Attention motivates the uncertainty side. It recasts linear attention as a Kalman filter in information form, parallelizes the filter with an associative scan, and shows that its internal variance spikes at error-prone tokens. It is strictly a 1D causal language model with diagonal covariance and time-invariant dynamics. Robust Filter Attention makes a related precision-weighted reading of attention but stays quadratic and roughly 2x slower to train. Neither has a notion of spatial windows, LiDAR range or beam sparsity.
On evaluation, Measuring and Modeling Uncertainty Degree for Monocular Depth Estimation constrains how we measure success. It shows that sparsification metrics such as AUSE reward accurate models regardless of uncertainty quality, argues instead for Spearman rank correlation between uncertainty and error, and provides a pairwise ranking loss that we reuse. Lightweight Probabilistic Deep Networks is a cautionary constraint. Propagating full Gaussian distributions through every layer correlates well with error but triples compute and doubles activation memory, so we deliberately carry only a scalar or low-dimensional precision and only through attention.
On the cross-task side, BEVFusion shows that one shared bird's-eye-view backbone can serve detection and map segmentation without mutual degradation, but it has no uncertainty output. Adaptive Dual Uncertainty Optimization shows that under test-time shift, entropy-based adaptation makes geometric heads collapse to uniform outputs. It fixes this with test-time gradient steps that couple semantic and geometric uncertainty. We take its diagnosis, that uncertainty must be shared across semantic and geometric outputs, but try to act on it inside the forward pass with no test-time optimization.
Core Claim
Let voxel $i$ in window $w$ have key $k_i$, value $v_i$ and a nonnegative precision $\lambda_i$. Precision is produced at the first layer from local geometry (point count, range, intensity) and thereafter inherited from the previous layer. Precision-carrying scattered attention accumulates
$$S_w=\sum_{i\in w}\lambda_i\,\phi(k_i)v_i^\top,\qquad z_w=\sum_{i\in w}\lambda_i\,\phi(k_i),$$
and returns, for a query voxel $j$, the feature $y_j=\phi(q_j)^\top S_w / \phi(q_j)^\top z_w$ together with an output precision $\lambda'_j=\alpha\,\lambda_j+\phi(q_j)^\top z_w$. Here $\phi(q_j)^\top z_w$ is the precision-weighted evidence that query $j$ actually draws from the window, and $\alpha\in(0,1)$ is a learned per-layer decay. The decay plays the role of the Ornstein–Uhlenbeck prior in Kalman Linear Attention: it stops precision from growing without bound across depth.
The additive form comes from information-form Gaussian fusion, where precisions of independent measurements add. Both sums are associative, so they fit ScatterFormer's chunk-wise kernel unchanged. The added cost is one length-$d$ vector per window and one scalar per voxel, which is linear in voxel count.
The final per-voxel precision feeds every task head. Each head turns it into a predictive variance for its own output: box residuals for detection, class-probability temperature for segmentation. Training uses heteroscedastic likelihood plus the ranking loss of Measuring and Modeling Uncertainty Degree.
The claim that would be new has two parts. First, a single backbone-level precision map rank-correlates with errors of tasks it received no gradient from. Second, using that precision to weight aggregation, rather than merely reporting it, makes the shared backbone degrade more gracefully under corruption.
Research Program
The work proceeds in three stages.
Stage one: build and profile. We implement the layer as a modification of ScatterFormer's Triton kernel, keeping its cross-window depth-wise convolutions. Precision passes through those convolutions by a fixed nonnegative smoothing so it never goes negative. Before measuring accuracy, we profile latency and memory against the unmodified kernel.
Stage two: confront the obvious confound. Raw evidence counts track point density, so near-range voxels would look "certain" regardless of content. We therefore parameterize the first-layer precision as a learned function of local geometry and check whether the trained precision carries information beyond density. We regress error on density alone and then on density plus precision, and report the partial rank correlation.
Stage three: the cross-task protocol. On nuScenes scans, which carry both box and point-wise segmentation labels, we train a shared backbone with detection and segmentation heads in three regimes. In the first, both heads supply likelihood gradients to precision. In the second, only the detection head does, with the segmentation head trained deterministically and precision detached from its loss. The third is the reverse. Cross-task transfer is measured as the rank correlation between precision and segmentation error in the detection-only regime, and vice versa.
Baselines
The do-nothing reference is ScatterFormer itself, retrained on the same multi-task nuScenes setup, with uncertainty read from what it already outputs: box confidence scores for detection and softmax entropy for segmentation. It anchors accuracy and latency. The proposal must not lose more than about one point of detection score or segmentation IoU against it, and must stay within roughly 10% of its runtime. The published reference points are 72.4 NDS on nuScenes and 73.8 mAP (L2) at 23 FPS on Waymo for ScatterFormer.
The strongest competitor on the actual question is the same deterministic ScatterFormer backbone with a per-voxel variance head added just before the task heads. That head is trained with the identical heteroscedastic likelihood and ranking loss, so it gets the same supervision and the same tuning budget. This is the cheapest way to get uncertainty-error correlation, and it is the one most likely to match us on clean in-task data. To beat it we must show higher rank correlation on the transferred task (detection-trained, evaluated on segmentation error) and under corruption. On clean in-task correlation we would accept parity.
The expensive reference is Monte Carlo dropout over the ScatterFormer backbone with ten samples, plus a five-member deep ensemble where compute allows. It bounds how much correlation is achievable when cost is ignored. Success is matching its rank correlation at one forward pass.
The key ablation keeps the precision readout but sets every $\lambda_i=1$ inside the sums, so aggregation is unweighted. This isolates whether attenuating low-precision voxels matters, or whether the value lies only in having a backbone-level evidence count. A second ablation removes the decay $\alpha$, to test the depth-stability argument borrowed from Kalman Linear Attention.
For the robustness comparison we also run entropy-minimization test-time adaptation on our backbone. The DUO paper reports +18% average gains over existing adaptation methods under real nuScenes weather and lighting shifts and +2.2 AP on KITTI-C. Its setting is monocular, though, so those numbers are context, not a like-for-like target.
For every baseline we use identical data splits, training schedule, augmentation and hyperparameter search budget, and every model is timed with the same kernel and hardware.
Why not just…?
Why not just add a variance head to a deterministic backbone? This is the real threat, and we treat it as the main comparison rather than a straw man. A head sees only the backbone's final features. If those features were already corrupted by fog points averaged in from neighbouring voxels, the head has no record of how much evidence each feature rested on, and a head trained for one task has no reason to track another task's errors. That is the hypothesis, not a known fact; nothing in the file reports cross-task uncertainty transfer for a variance head. If the variance head matches our transferred rank correlation and our corruption robustness, we will conclude the attention change is unnecessary and say so.
Why not just use Monte Carlo dropout or ensembles? They multiply inference cost by the number of samples. Measuring and Modeling Uncertainty Degree reports a single-pass method 2–3x faster than Monte Carlo dropout that reaches a Spearman correlation of 0.3284 on NYU-Depth-v2 while outperforming deep ensembles. We include dropout as an upper reference, but a perception stack at 23 FPS cannot afford ten passes.
Why not just use the linear-attention normalizer or point density as free uncertainty? The unweighted normalizer $\phi(q)^\top \sum\phi(k)$ already exists in ScatterFormer-style attention and costs nothing. The $\lambda=1$ ablation evaluates exactly this proxy. The density regression in stage two tests whether learned precision adds rank information beyond raw point counts. If it does not, the free proxy wins.
Why not adapt at test time instead? Adaptive Dual Uncertainty Optimization shows that naive entropy adaptation collapses geometric outputs. Its fix needs test-time backpropagation and is built for monocular detection, not a shared multi-task LiDAR backbone. Our entropy-adaptation baseline on the same backbone tells us whether a forward-pass mechanism recovers comparable robustness without any gradient steps.
Proposed Experiments
Accuracy and cost. We report detection with NDS and mAP on nuScenes and mAP (L2) on Waymo, and LiDAR segmentation mIoU on nuScenes and SemanticKITTI. The segmentation reference points are 55.9% test mIoU for SqueezeSegV3 on SemanticKITTI and 62.7% map-segmentation mIoU for BEVFusion on nuScenes. The latter is a camera-plus-LiDAR bird's-eye-view task, so it is context rather than a direct target. Latency and peak memory are measured on A100 against the ScatterFormer kernel.
Uncertainty quality. We measure per-voxel and per-box Spearman correlation between uncertainty and error, alongside AUSE for continuity. No reported result was found on Lacuna for Spearman uncertainty-error correlation on nuScenes, SemanticKITTI or any LiDAR benchmark. All baselines will therefore be run by us under one protocol.
Robustness. We evaluate mean corruption error on Robo3D's nuScenes-C, where LiMoE reports a reduction from 91.67% to 88.43% using a three-representation mixture of experts at 8.3 FPS. We also report rank correlation under each corruption type separately, since an uncertainty signal that only works on clean data is not useful.
Decisive checks. Three results settle the question. The first is the detection-only training regime: if the transferred rank correlation for segmentation error is not clearly above the variance-head baseline's transferred correlation, the cross-task claim fails. The second is the $\lambda=1$ ablation under nuScenes-C: if weighting adds no robustness, precision is only a readout and the cheaper normalizer proxy suffices. The third is the partial rank correlation after controlling for point density: if it is near zero, the model has learned density, not uncertainty.
A clean outcome would be a backbone that loses no accuracy, adds under 10% latency, and carries an error signal that transfers across tasks and survives corruption. That would give shared 3D backbones a built-in, single-pass way to report where they are wrong.
References
- ScatterFormer: Efficient Voxel Transformer with Scattered Linear Attention
- Kalman Linear Attention: Parallel Bayesian Filtering For Efficient Language Modeling and State Tracking
- Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation
- Measuring and Modeling Uncertainty Degree for Monocular Depth Estimation
- Lightweight Probabilistic Deep Networks
- BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird’s-Eye View Representation
- Adaptive Dual Uncertainty Optimization: Boosting Monocular 3D Object Detection under Test-Time Shifts
- SqueezeSegV3: Spatially-Adaptive Convolution for Efficient Point-Cloud Segmentation
- LiMoE: Mixture of LiDAR Representation Learners from Automotive Scenes