Object Shape Priors as Depth Anchors Along Camera Rays in Bird's-Eye View

A single camera cannot tell how far away a surface is. Everything that lies along one line of sight lands on the same pixel. Camera-only driving systems build a top-down map of the scene, called a bird's-eye view (BEV), by guessing a depth for every pixel and dropping its features into the map. When that guess is wrong, features land in the wrong map cell, smeared along the radial line from the camera. Current methods treat each pixel's depth as its own problem and, at best, attach an error bar to it. We propose to couple the depths of all pixels on one object through a learned model of what that object category looks like in 3D, at its real metric size. A car is not a thousand independent depth questions. It is one question with a handful of unknowns: where the car is, which way it faces, how big it is, and which car shape it has. Answering that one question pins down every pixel ray that hits the car. It also says where the car's hidden back half is.

Concretely, each detected vehicle gets a probabilistic shape code from a fast encoder (no test-time optimization). The decoded 3D shape is intersected with each pixel ray to give an object-implied depth distribution. That distribution is multiplied with the network's own per-pixel depth distribution before features are splatted into BEV. The claim we want to test is narrow and falsifiable. Object-level coupling under a metric size prior should reduce the along-ray component of localization error and improve recall of partly occluded vehicles. It should do little for the across-ray component, which monocular cues already handle well. A key ablation asks whether a detailed shape model is needed at all, or whether a cuboid with a learned size distribution captures the same gain.

Where this sits

Two lines of BEV work attack the depth bottleneck from the pixel side. Toward Real-world BEV Perception: Depth Uncertainty Estimation via Gaussian Splatting (GaussianLSS) predicts a Gaussian depth per pixel and splats 3D ellipsoids into BEV. Parametric Depth Based Feature Representation Learning for Object Detection and Segmentation in Bird's-Eye View uses Laplacian depth plus an analytic visibility map that suppresses features in occluded space. FB-BEV gates attention-based backward sampling with 2D depth distributions to remove the "ghost" features that query methods like BEVFormer place along rays.

These papers motivate our proposal. All of them show that ray placement is the failure mode. Every one of them, however, still decides depth pixel by pixel, so a gate built on noisy per-pixel depth inherits that noise. The visibility-masking approach also conflicts with the driving requirement, stressed by BSH-Det3D for LiDAR, that the hidden extent of occluded vehicles be inferred rather than erased.

A second line supplies our method. DeepSDF represents category shapes as a decoder from a latent code to a signed distance function, a field whose zero level is the object surface. Its test-time latent optimization is too slow for driving, by its own account. Multi-View 3D Object Reconstruction and Uncertainty Modelling With Neural Shape Prior replaces that optimization with a feed-forward encoder that outputs a Gaussian over latent codes and carries the uncertainty into the distance field. It works only on cropped single objects in canonical frames (Pix3D, ShapeNet), with no world or BEV frame. Disp R-CNN is the closest precedent for using a statistical car model to sharpen per-pixel depth inside an object region. It relies on calibrated stereo, so its scale comes from geometry rather than from the prior. It also operates in camera-centric regions of interest with no BEV representation.

GitNet does inject geometric priors into the view transform. Those priors are a flat ground plane and ray-restricted attention, not volumetric object shapes.

None of this work combines a pixel-level depth distribution with an object-level shape distribution in one BEV lifting step. That combination is what we build.

Core Claim

For a pixel $u$ that falls inside the image mask of detected object $k$, the depth used for lifting is the product of two densities:

$$p(d \mid u) \;\propto\; \mathcal{N}\big(d;\, \mu_u, \sigma_u^2\big)\cdot \mathbb{E}_{z\sim\mathcal{N}(\mu_z,\Sigma_z),\; s\sim p(s\mid c)}\Big[\mathcal{N}\big(d;\, D_u(z, T_k, s),\, \tau^2\big)\Big].$$

The terms are defined as follows. $\mu_u$ and $\sigma_u^2$ are the network's per-pixel depth mean and variance, as in GaussianLSS. $z$ is the object's shape code, drawn from the encoder's Gaussian with mean $\mu_z$ and covariance $\Sigma_z$. $T_k$ is the predicted yaw and ground-plane position of object $k$. $s$ is a metric scale drawn from a size distribution $p(s \mid c)$ learned for category $c$. $D_u(\cdot)$ is the depth at which ray $u$ first crosses the decoded surface. $\tau$ is a small fixed bandwidth.

Pixels outside any object mask keep the plain pixel distribution. Vehicle cells that are occluded in the image are filled by splatting the interior of the decoded shape. Each filled cell carries an opacity equal to the Monte Carlo agreement across samples of $z$ and $s$, so uncertain completions are marked as uncertain rather than hidden.

The size prior is how the method addresses scale ambiguity. Category size statistics are an imperfect but real metric anchor, the same logic Geo-Supervised Visual Odometry uses with human height. Their known failure mode is atypical instances such as unusual trucks. We handle that by learning $p(s \mid c)$ as a distribution, not a constant, and by letting the pixel term override the prior wherever it is confident.

Research Program

The shape prior is a DeepSDF-style decoder. It is pretrained on ShapeNet vehicle categories and then adapted to real data. The adaptation supervises the latent encoder with nuScenes LiDAR points accumulated over sweeps inside each annotated box. This directly targets the synthetic-to-real gap flagged in the neural-shape-prior paper. Pedestrians and other classes with weak shape regularity use a cuboid prior only.

The encoder reads image features cropped by a first-pass BEV detection head. This two-stage structure mirrors FB-BEV's proposal-then-refine design. The ray-surface crossing $D_u$ is computed by a fixed, small number of samples along each ray inside the proposal's depth interval. We use eight latent samples, so the cost scales with the number of object pixels rather than the image size.

Lifting and rasterization reuse GaussianLSS's splatting. The system is trained end to end on nuScenes using the standard detection and segmentation losses, plus a LiDAR depth loss on the fused density.

Three analyses are planned. The first decomposes each object's center error into radial (along the camera ray) and tangential components, stratified by range and by nuScenes visibility level. The second measures completion quality of occluded vehicle footprints in BEV against annotated boxes. The third tests calibration of the completion opacity, asking whether high-opacity cells are in fact occupied more often.

Baselines

The naive reference is GaussianLSS itself, retrained with our backbone, resolution, and schedule. It is the exact model our method extends with nothing removed, so any gain is attributable to the object term. The comparison runs on the same data, uses the same number of epochs, and gets a hyperparameter sweep of equal size. The paper reports 38.3% vehicle IoU on nuScenes. Beating our own retrained copy matters more than beating that printed figure, and we will report both.

The strongest existing competitor for detection is FB-BEV. It already targets ray ghosts with depth-aware gating and reports 62.4% NDS on the nuScenes test set. That figure likely comes from a larger backbone and test-time setup than we will use. We will therefore retrain FB-BEV and our method at a matched backbone and image size on the validation split. Beating it means higher NDS, and specifically a lower radial component of translation error, at comparable frame rate.

We also report BEVFormer, which reports 56.9% test NDS and over 6% gains in recall of heavily occluded objects from temporal attention. It is retrained matched, with and without temporal history. The parametric-depth method serves as the masking-based occlusion reference; it reports 0.468 mAP and 0.495 NDS on nuScenes.

Two ablations isolate the key ingredient. The first replaces the signed-distance shape with an oriented cuboid that keeps the learned size distribution $p(s \mid c)$. The second removes the size distribution so the shape is fit in a scale-free way. The cuboid ablation is the most important experiment in the proposal. If it matches the full model, the conclusion is that object-level coupling with a metric size prior is what helps, and detailed shape is unnecessary. We will report that outcome as the finding.

Why not just…?

Why not just rely on the detection head's class-specific box sizes? Box heads do learn typical sizes. However, they use them only at the output, after features have already been splatted into the wrong cells. Our method uses size and shape at lifting time, where the smearing happens. Whether an output-side prior is enough is exactly what the cuboid-at-lifting ablation versus the plain baseline measures. If the plain baseline's radial error already matches, the idea fails.

Why not just run Metric3D v2 and lift its depth? Metric3D v2 reports zero-shot absolute relative error of 0.044 on KITTI, so it is a serious threat for visible-surface depth. It predicts depth only for visible surfaces, so it cannot place the hidden half of an occluded car. Its per-pixel errors also still misplace features in BEV. We will run it as a frozen depth source feeding the same splatting, and also as an initialization for $\mu_u$. If our object term adds nothing on top of Metric3D v2 depth, then the prior is redundant once depth is good enough. That result would be informative rather than fatal.

Why not just add sparse LiDAR or distill from a LiDAR teacher? Sparse metric points fix global scale with a least-squares fit, as In-Place Zero-Shot Metric Depth Completion shows. BEVDistill transfers LiDAR geometry to camera students during training. Neither sees behind occluders, and sparse points thin out at long range. We will add BEVDistill-style distillation to both the baseline and our model, to test whether the gains are complementary rather than overlapping.

Why not just use temporal attention? Temporal attention helps only if the object was visible earlier. We will report results separately on objects whose visibility never exceeds the lowest nuScenes visibility level within the history window. In that subset, temporal attention should not help and the shape prior should.

Proposed Experiments

All main experiments use nuScenes. Detection is scored by mAP, NDS, and our radial versus tangential split of translation error. BEV vehicle segmentation is scored by IoU, and occluded-footprint IoU is scored per visibility bin. Frame rate is reported on one GPU type. KITTI 3D detection at the Moderate level gives a secondary monocular check of transfer. Pix3D-MV is used only to verify that the adapted encoder still reconstructs shapes reasonably; the neural-shape-prior paper reports 0.3902 IoU there.

The decisive checks are three. First, gains should concentrate in radial error and in low-visibility objects; uniform small gains would suggest extra capacity rather than ray disambiguation. Second, the full shape prior should beat the lifting-time cuboid prior on occluded footprint IoU, or we drop the claim about shape detail. Third, completion opacity should be calibrated. A last stress test evaluates trucks and buses, the atypical-size classes, separately. This shows whether the learned size distribution adds bias where category statistics are weakest.