Ray-Conditioned Normalization for Shared Multi-Camera Backbones

Self-driving cars perceive their surroundings by running every camera image through one shared image network and then combining the resulting features into a 3D map of the scene. Sharing one network across cameras is cheap and data-efficient. It also means a front camera looking down the road and a side camera looking across traffic are processed identically, even though the same pixel row means very different things in each: a different distance to the ground, a different amount of perspective squeeze, a different likely object size. We propose to keep every convolution shared and give the network a small, continuous per-pixel signal that tells it what that pixel's viewing ray looks like. The ray is described by its elevation above or below the horizon, the camera's focal length, and its pitch and roll. The signal is turned into normalization scales and shifts at every layer. It deliberately ignores which way the camera points around the car (its heading, or "yaw"), so cameras that look in different directions share their statistics.

The bet is this. Earlier work found that making features aware of camera geometry helps, but it injected that geometry after the image network had already produced its features. Putting geometry inside the shared network lets the image features be shaped for the 2D-to-3D lifting step. Because the conditioning is continuous and does not depend on heading, a model trained on one camera rig should degrade gracefully when the cameras are re-mounted, re-zoomed, or tilted. A model that learns one set of parameters per camera cannot do that.

Core Claim

Our central claim is narrow and testable. On a fixed rig, ray-conditioned normalization should roughly match the simpler "one learned affine per camera" approach. Under rig changes it should clearly beat that approach, and also beat a shared backbone that receives the same geometry only after feature extraction. This holds both for 3D detection and for dense 3D occupancy, where the cost of putting features in the wrong place along a ray is highest.

We make the in-distribution concession up front. On nuScenes the six cameras never move, so a per-camera ID carries almost the same information as continuous geometry. The proposal stands or falls on the rig-shift and calibration-shift experiments.

Where This Sits

The mechanism comes from medical imaging. Reliable Liver Fibrosis Assessment from Ultrasound using Global Hetero-Image Fusion and View-Specific Parameterization keeps all convolutions shared across ultrasound views but gives each discrete anatomical view its own Instance Normalization scale and shift. It reports partial AUC rising from 0.710 to 0.783. The method needs a known discrete view label, pools features globally, and does no 3D reasoning. We replace the per-view lookup table with a function of continuous ray geometry, and we feed the resulting features into 3D lifting. A Pose-Sensitive Embedding for Person Re-Identification similarly modulates a shared network with front, back and side bins predicted by an internal branch. Its discretization and its dependence on a predictor are exactly what we avoid: calibrated cameras already give the ray geometry for free.

The geometry-aware 3D detectors motivate the gap and also constrain the design. CAPE shows that expressing positional embeddings in each camera's local frame rather than the vehicle frame gains 1.4 NDS and 2.4 mAP. NDS is the nuScenes Detection Score, a composite of detection precision and box-error terms. That result is evidence that heading-dependent coordinates confuse a shared network, and it is why our conditioning drops yaw. VEDet adds Fourier encodings of camera pose and pixel rays to backbone outputs and trains with virtual-view equivariance. AeDet makes the top-down convolutions rotate with azimuth and stays stable under a 60° revolving test, where BEVDepth drops 4.6 NDS. All three leave the image backbone unconditioned. Our question is whether moving the geometry into the backbone's normalization adds anything beyond these post-backbone fixes. We answer it with a matched ablation rather than by assumption.

For aggregation we reuse existing machinery unchanged. BEVFormer lifts features with deformable cross-attention from bird's-eye-view pillar queries, a flat top-down grid in which each query stands for a vertical column of 3D points. SurroundOcc does the same with full 3D volume queries. Occupancy as Set of Points queries arbitrary 3D points. MvACon observes that these projected sampling points often land on sky or road, and it adds 2D clustering context to address this. We test a sharper version of that diagnosis: features that know their ray elevation should give less weight to pixels that cannot plausibly contain the queried 3D point.

Research Program

The module is spatially varying normalization. At each stage of a ResNet-style backbone, we compute a per-pixel conditioning vector $g(u,v)$ for pixel coordinates $(u,v)$. It contains Fourier features of four quantities: the ray's elevation angle in a gravity-aligned camera frame, the ray's in-camera azimuth relative to the optical axis, the log focal length, and the camera's pitch and roll. Heading around the vehicle is excluded. A small shared network maps $g$ to scale and shift maps $\gamma, \beta$. These modulate the normalized activations:

$$ h' = \gamma(g)\odot \frac{h-\mu}{\sigma} + \beta(g) $$

Here $h$ is a feature map, and $\mu,\sigma$ are its normalization statistics. Both $\gamma$ and $\beta$ are initialized to the identity, so the pretrained backbone is unchanged at the start of training. The added parameters are a few small networks, a negligible fraction of the backbone.

We plug the module into BEVFormer for nuScenes detection and into BEVFormer-based occupancy for Occ3D-nuScenes, holding every other component fixed. To make rig shift measurable, we exploit a geometric fact. A pure camera rotation or focal change is an exact image homography, a 2D warp. We can therefore build test rigs with tilted, rolled, or re-zoomed cameras from real nuScenes images, with exactly known new calibration. We also run the AeDet-style revolving test, which permutes heading. Beyond accuracy, we measure where cross-attention mass lands using LiDAR depth. Specifically, we measure the fraction of sampling weight placed on pixels whose LiDAR depth is inconsistent with the query's 3D position. This tests whether the module changes lifting behavior or merely adds capacity.

Baselines

The naive reference is the unmodified shared backbone inside BEVFormer. On the nuScenes test set, BEVFormer reports 56.9 NDS. On Occ3D-nuScenes, Occupancy as Set of Points reports a BEVFormer occupancy model at 37.84 mIoU (mean intersection-over-union across semantic classes). We retrain this baseline ourselves at the same, smaller scale we can afford, rather than comparing against these headline numbers. Every model in this note shares the same data, schedule, image resolution, and hyperparameter search budget.

The strongest discrete competitor is per-camera View-Specific Parameterization in the style of the liver-ultrasound paper: one learned normalization affine per camera slot, otherwise identical to our model. It is the most direct and cheapest way to give a shared backbone view awareness. We expect it to tie us in-distribution. Beating it means a clear margin on the homography-shifted rigs, without losing ground on the unshifted validation set.

The strongest geometry-aware competitor is post-backbone ray encoding. We use either CAPE-style local-frame positional embeddings, for which CAPE reports 61.0 NDS and 52.5 mAP overall, or VEDet-style Fourier ray features, for which VEDet reports 45.1 mAP and 52.7 NDS on validation. We reimplement these on our BEVFormer base so that the only difference from our model is where the identical geometry signal enters. If our model is no better than this variant, the proposal's key ingredient adds nothing.

For occupancy, SurroundOcc reports 20.30 semantic scene-completion mIoU on its nuScenes validation protocol. We cite it as the volumetric reference and do not rerun it. Two ablations isolate the ingredient. The first adds heading back into $g$; if heading-invariance matters, the revolving test should suffer. The second conditions only the last backbone stage rather than all stages.

Why not just…?

Why not just give each camera its own normalization parameters? On nuScenes this is the real threat, and we expect it to match us in-distribution, because a fixed rig makes camera ID and geometry interchangeable. It has two weaknesses. It cannot represent a camera it has never seen, and it treats the front-left and back-left cameras as unrelated even though their ray elevations are nearly identical. The homography-shifted rig test settles this directly. If per-camera parameters hold up there too, we will report that the continuous version is unnecessary.

Why not just feed ray maps as extra input channels? This passes the same information as our module and is simpler, so we run it with identical conditioning features. Information entering only at the first layer must be carried forward through features shaped by ImageNet pretraining. Multiplicative modulation at every stage can instead change feature statistics directly. That is a hypothesis, not established evidence. If input concatenation matches us, we will adopt it and frame the contribution around ray-conditioning in general rather than normalization specifically.

Why not just add positional encodings after the backbone, as CAPE and VEDet do? These methods show that geometry helps, but they cannot change which image features the backbone computes. The attention-mass diagnostic and the matched placement ablation in the Baselines section answer this. A gain in-distribution and under shift, together with less attention mass on depth-inconsistent pixels, would show that placement matters.

Why not just train a plain backbone with random intrinsic and extrinsic augmentation? Augmentation could buy rig robustness without any architectural change. We therefore apply the same homography augmentation to every model, including the plain baseline. Our module must win on top of augmentation, not instead of it.

Proposed Experiments

The first decisive check is the in-distribution parity test. Our model should be within noise of per-camera parameters, and at least as good as post-backbone ray encoding, on nuScenes validation NDS and mAP and on Occ3D-nuScenes mIoU. If it loses here, the added mechanism is harmful.

The second check is the shifted-rig curve. We plot NDS and occupancy mIoU against camera pitch perturbation of 0–10°, focal change of 0.7–1.4×, and AeDet's revolving test. The prediction is a visibly flatter curve for our model than for both strongest competitors. We also expect the orientation-error term (mAOE) to benefit most, since AeDet's 5.2% orientation-accuracy gain suggests orientation is where geometry mismatch shows up.

The third check is mechanistic. We measure the fraction of cross-attention weight on LiDAR-inconsistent pixels, and expect it to drop relative to post-backbone encoding. A drop with no accuracy gain would point to a decoder bottleneck. An accuracy gain with no drop would mean the benefit comes from something other than better lifting.

Finally, as an external-rig test, we fine-tune nuScenes-trained models on a small fraction of Waymo, which uses a different camera layout. There, per-camera parameters must be re-learned from scratch, while ray-conditioned parameters transfer. We report LET-mAPL, Waymo's detection metric that tolerates errors along the viewing direction, against the plain backbone. We found no reported BEVFormer Waymo number on Lacuna, and MvACon's +0.8 LET-mAPL plug-in gain on Waymo-mini serves as a rough scale for what counts as meaningful.