Fast 3D detectors for self-driving flatten a LiDAR point cloud into a top-down image, called a bird's-eye view (BEV), so that ordinary 2D convolutional networks can process it. Flattening merges everything in a vertical column into one feature vector. A pedestrian under an overpass and the overpass itself become a single cell. Today nothing downstream is told how much was lost in each cell. We propose to treat this flattening the way probabilistic numerics treats a truncated linear solver. The flattening is a known linear projection of a hidden, height-resolved column. A learned Gaussian prior over that column then yields a posterior covariance, which states per cell how much vertical structure the projection destroyed. That number has two uses. It is a calibrated diagnostic that can be checked against held-out 3D features. It is also a budget signal: it chooses the small fraction of cells worth sending through an expensive 3D voxel branch.
In two sentences: make the height-collapse step an explicit linear map, put a context-dependent Gaussian prior on what it discards, and read off how uncertain each flattened cell is about its own vertical content. Then spend 3D compute only where that uncertainty is high, and test whether the uncertainty is calibrated and whether it beats cheap heuristics for deciding where to spend.
Core Claim
Probabilistic linear algebra has shown that classical projection methods are posterior means of Gaussian inference. Probabilistic linear solvers: a unifying view. proves this for conjugate gradients and GMRES. Uncertainty calibration for probabilistic projection methods extends it to arbitrary projections by placing prior mass on the null space of the projection operator. Posterior and Computational Uncertainty in Gaussian Processes shows that the uncertainty created by cutting computation short can be tracked separately from ordinary data uncertainty. All of this work lives inside abstract systems $Ax=b$ or Gaussian-process regression. None of it touches geometric projections. On the perception side, Unifying Voxel-based Representation with Transformer for 3D Object Detection measured the cost of flattening: going from one height bin to eleven raised camera-based nuScenes detection score (NDS) by 4.2%. Its response, like Voxel or Pillar, is to keep 3D features everywhere or fuse them through fixed pooling. In neither case is there a measure of where the loss happened.
Our claim is specific. Suppose the flattening is made exactly linear and the prior over the discarded part depends on context. Then the posterior variance of each column is a non-trivial, checkable estimate of reconstruction error. Allocating 3D computation by that variance should recover most of the accuracy of a full voxel-pillar hybrid at a small fraction of lifted cells. It should also do better than allocating by point-height statistics or by an unstructured learned gate.
One subtlety decides whether the idea has content. In a noise-free linear-Gaussian model the posterior covariance does not depend on the observed values, only on the observation operator and the prior. If every pillar used the same projection and the same prior, every cell would report identical uncertainty, and the method would be vacuous. The covariance becomes informative only if the prior is conditioned on observed context, or the operator varies with which height bins contain points. We build in both, and we keep the fixed-prior version as an ablation that should fail.
Research Program
We redesign the pillar encoder so that the height collapse is an explicit linear map. Each column first produces per-height-bin features $z \in \mathbb{R}^{Hd}$, where $H$ is the number of height bins and $d$ the per-bin feature width. The 2D pillar feature is then $p = W^\top z$ for a learned $W \in \mathbb{R}^{Hd \times d}$. Empty height bins are masked out, so the effective operator $W_m$ differs from cell to cell.
A small network looks at the BEV neighbourhood $c$ and predicts a prior mean $\mu(c)$ and a low-rank-plus-diagonal covariance $\Sigma(c)$. Conditioning on $p$ gives the usual Gaussian posterior, whose covariance is
$$\Sigma_{\text{post}} = \Sigma(c) - \Sigma(c) W_m \left(W_m^\top \Sigma(c) W_m\right)^{-1} W_m^\top \Sigma(c).$$
Its trace is the predicted squared error of reconstructing the column's full 3D feature from the flat one. The structure follows Fanaskov et al.'s null-space prior, with $W_m$ in place of their $W^\top A$. The inner matrix is only $d \times d$ per cell, so the computation is batched and cheap compared with the backbone.
The prior is trained by maximising the Gaussian likelihood of "teacher" column features $z$. These come from a sparse 3D voxel branch run at training time, in the spirit of the voxel-to-student distillation in Point-to-Voxel Knowledge Distillation for LiDAR Semantic Segmentation, whose code (cardwing/Codes-for-PVKD) we reuse for the teacher pipeline.
To let one lifted cell inform its neighbours, the prior couples each 3×3 neighbourhood through a shared low-rank factor. At inference, a budget of $K$% of cells is chosen by greedy variance reduction. Each chosen cell receives its true 3D feature from the voxel branch, and its neighbours' posteriors are updated with rank-one corrections. Unlifted cells pass their posterior mean and log-variance to the detection head. Lifted cells pass the fused voxel feature.
The program has a scientific question alongside the engineering one. The probabilistic numerics literature disagrees about which way these estimates fail:
- Probabilistic linear solvers: a unifying view. finds BayesGMRES overly conservative.
- Statistical properties of BayesCG under the Krylov prior finds low-rank posteriors slightly optimistic, and proves strict calibration is impossible because Krylov search directions depend on the right-hand side.
- IterGP is provably conservative.
Our setting differs in a way that matters. The lifting policy depends only on already-observed pillar features, not on the hidden column. Choosing what to observe next based on what has already been seen leaves Bayesian inference valid. The Krylov obstruction therefore does not apply, and any miscalibration we see must come from mismatch in the learned prior or from its low rank. We predict mild optimism that grows as the rank shrinks, matching Reid et al.'s direction. Measuring this is a contribution in its own right.
Baselines
The naive reference is a plain PointPillars detector with no lifting. It represents what you get for free at high frame rate. Its reported KITTI result is 86.10 bird's-eye-view AP on Moderate cars at 62 Hz. We retrain it on our splits with the same backbone, schedule and augmentation as every other arm. It is "beaten" trivially by any lifting. Its role is to anchor the 0% end of the budget curve.
The strongest existing method is Voxel or Pillar (VPF). It runs voxel and pillar branches everywhere and links them with a bidirectional sparse fusion layer. VPF reports 73.8 NDS on nuScenes, 73.9% Level-2 mAPH on Waymo validation, and 17 FPS for its Lite variants. VPF is the 100% end of our curve and the accuracy ceiling for a hybrid design. We run it with its published configuration, and also re-implemented on our shared pillar backbone so that both arms use identical voxel-branch capacity. We count it as beaten if we match its NDS within 0.5 points while lifting 25% of cells or fewer, at measurably higher frame rate on the same GPU. We do not claim to exceed it outright.
Between the two ends sit the allocation baselines, all given the identical lifting budget, voxel branch and head:
- Random lifting. Cells are lifted uniformly at random.
- Height-statistics lifting. Cells are ranked by the spread of point heights in the column and the number of distinct height clusters. This is the obvious cheap proxy for vertical ambiguity, extending the hand-crafted channels of Complex-YOLO.
- Learned gate. A small network trained end to end with the detection loss and a budget penalty decides which cells to lift.
- Learned error-regression head. A network directly regresses $\lVert z - \hat z \rVert^2$ from the same context the prior network sees.
Every learned arm gets the same parameter count and the same hyperparameter-search budget as our prior network. Our method beats these baselines if its accuracy-versus-budget curve dominates theirs at budgets of 10–30%, with the gap concentrated on the vertically ambiguous subset described below.
Three ablations isolate the key ingredients. The first uses a fixed, context-free prior; it should collapse to near-random allocation, confirming the vacuity argument. The second drops cross-cell coupling, which isolates the value of joint updates. The third replaces the linear collapse with max-pooling, which removes the exact null-space structure.
Why not just…?
Why not just run 3D voxels everywhere? That is the 100% point of our curve, so the experiments include it directly. PointPillars reports full-3D pipelines running at a few FPS against its 62–105 Hz. UVTR names lifting into dense 3D grids as its main cost. The question is not whether 3D helps but how little of it buys most of the gain. A full-voxel arm answers that.
Why not just pack height statistics into extra channels, or lift where point heights are spread out? This is the most serious threat. Height spread is free, and it may correlate strongly with true ambiguity. Its known weakness, documented by Domain Adaptation for Vehicle Detection from Bird's Eye View LiDAR Point Cloud Data, is that max-height encodings hide what lies beneath the top surface. A spread statistic also cannot tell an informative tall column (a tree trunk) from an ambiguous one (a person under a canopy). Our prior is trained on teacher features and can learn that difference from context. The height-statistics lifting arm settles it. If that arm's curve matches ours, we will report that the probabilistic machinery adds nothing for allocation, and keep only the calibration study.
Why not just learn a gate or regress the error directly? A direct error-regression head yields a scalar per cell. It cannot say how lifting one cell changes its neighbours' uncertainty, which is what the coupled Gaussian posterior provides for greedy, non-myopic selection. It also gives no structured covariance whose calibration can be tested with the standard statistics from probabilistic numerics. Whether these properties matter for detection accuracy is an empirical question. The comparison of the error-regression head against our model with coupling ablated against our full model is designed to separate the value of a good per-cell score from the value of joint updates. If the error-regression head matches the full model, the honest conclusion is that a learned scalar suffices.
Why not just distill a 3D teacher into the 2D student? PVD does exactly this and reaches 68.9% mIoU on SemanticKITTI with a 75% reduction in multiply-accumulate operations. At test time, though, the student reports nothing about where it is likely wrong, and it cannot choose to spend more compute there. We use distillation to train our prior, so the two are complementary. We also include a PVD-style distilled pillar student at 0% budget, to check that any gain is not simply better pillar features.
Proposed Experiments
The main testbed is nuScenes LiDAR detection, measured by NDS and mAP. Waymo Level-2 mAPH serves as a second dataset. On KITTI we report bird's-eye-view AP for the Pedestrian and Cyclist classes, where FVNet documents that small objects vanish under grid flattening. For all three we plot accuracy and wall-clock FPS against lifting budgets of 0, 5, 10, 25, 50 and 100%.
Because aggregate scores dilute the effect we target, we also build a vertically ambiguous evaluation subset from validation scenes. A ground-truth box qualifies if its BEV footprint overlaps a column containing non-ground points more than about 1 m above the box top: objects under bridges, trees or signage. We report recall on this subset separately. No published number exists for this subset; we found no reported result on Lacuna for vertically ambiguous subsets of nuScenes or KITTI.
The calibration experiment compares predicted posterior variance against the actual teacher-feature reconstruction error on held-out frames. It uses the $Z$-statistic of Probabilistic linear solvers: a unifying view., which should follow a chi-squared distribution under calibration, and the $S$-statistic of Statistical properties of BayesCG under the Krylov prior. We sweep prior rank to test the predicted drift toward optimism, and we compare the uncalibrated posterior against Fanaskov et al.'s covariance rescaling.
Three outcomes would be decisive. The fixed-prior ablation should allocate no better than random. If it does better, the variance is not doing what we claim. Our curve should dominate height-statistics and learned-gate allocation on the ambiguous subset at 10–30% budget. Finally, calibration should hold well enough that cells in the top variance decile show at least several times the median reconstruction error. If the second check fails, the result is still a clean negative: cheap geometry beats Bayesian bookkeeping for this projection.
References
- Probabilistic linear solvers: a unifying view.
- Uncertainty calibration for probabilistic projection methods
- Posterior and Computational Uncertainty in Gaussian Processes
- Unifying Voxel-based Representation with Transformer for 3D Object Detection
- Voxel or Pillar: Exploring Efficient Point Cloud Representation for 3D Object Detection
- Point-to-Voxel Knowledge Distillation for LiDAR Semantic Segmentation
- Statistical properties of BayesCG under the Krylov prior
- PointPillars: Fast Encoders for Object Detection From Point Clouds.
- Complex-YOLO: An Euler-Region-Proposal for Real-Time 3D Object Detection on Point Clouds
- Domain Adaptation for Vehicle Detection from Bird's Eye View LiDAR Point Cloud Data
- FVNet: 3D Front-View Proposal Generation for Real-Time Object Detection from Point Clouds