Self-driving cars increasingly describe their surroundings as a 3D occupancy grid: a stack of small cubes (voxels), each labeled empty or as road, car, pedestrian, and so on. Today's best camera-only systems build this grid by "lifting" image features into 3D, and they do it one way. Each pixel is spread along its camera ray according to a predicted depth, and the camera's mounting pose (its extrinsics) decides where that ray lands in the world. This proposal adds a second lifting route that is geometrically different from the first. Pixels are placed by their predicted height above the ground, and both routes keep the full vertical axis instead of flattening it. A learned gate compares the two volumes voxel by voxel. The claim is that when the camera tilts or vibrates, the two volumes disagree in predictable places, and the model can use that disagreement to lean on whichever route the tilt has corrupted less.
This matters because both directions in the source literature point at the same failure without jointly addressing it. Dual-branch view fusion has shown that pairing two representations helps. The strongest occupancy result, ProtoOcc, pairs a small-kernel 3D voxel branch with a large-kernel bird's-eye-view (BEV) branch, but both branches come from the same depth-based lifting. They therefore share its fragility to pose error, and ProtoOcc is evaluated only on clean calibration. Separately, BEVHeight++ showed that fusing a height-based branch with a depth-based branch keeps 3D detection stable under camera jitter, where depth-only models fell from 41% to 5% accuracy. That fusion happens in a BEV plane that collapses height, and it was tested on boxes, not dense volumes. Height lifting is naturally volumetric, because a height bin is a vertical coordinate. It is the obvious partner for a native 3D branch, and nobody in this literature has paired them.
Core Claim
The robustness gain from dual-branch fusion comes from the branches having different geometric error modes, not from having two branches. A depth-lifted volume and a height-lifted volume, both uncollapsed and fused by a gate that sees their disagreement, should keep most of their clean-data occupancy accuracy under camera-pose perturbation. Two ablations should fail to match it. The first is a dual-branch model whose branches share one lifting geometry, as in ProtoOcc. The second is the same model trained with pose-noise augmentation. On clean data, the model should cost essentially nothing.
Research Program
The build starts from the ProtoOcc codebase and training recipe so that everything except the lifting stays fixed.
The depth branch is ProtoOcc's voxel branch as published. Image features are spread along rays by a predicted depth distribution and processed with small 3D kernels.
The height branch predicts, for each pixel, a distribution over height above a reference ground plane, following BEVHeight++. It places the feature at the voxel column where the ray reaches that height. The height range in Occ3D is only a few meters, so this branch can use a coarse vertical resolution. That keeps its memory cost well below a second full-resolution volume and respects the cubic-cost constraint documented in Occ3D.
ProtoOcc's BEV branch, with its large receptive field, is kept and fed from the fused volume. Its prototype decoder is also unchanged.
Fusion is where the proposal differs from ProtoOcc's multi-scale concatenation. At each voxel, a small network receives both features and their difference and outputs a mixing weight. The weight is trained under the ordinary occupancy loss, while each training batch sees randomly perturbed extrinsics in only some samples. The intuition behind the mechanism is simple geometry. A small pitch error moves a depth-lifted far point by roughly depth times the angle error, which becomes large at range. Height-lifted placement depends on the ground-plane intersection, so it is dominated by the height prior near the camera but degrades differently. The two errors are therefore not aligned, and their disagreement carries information about the perturbation.
As a secondary, clearly optional analysis, we will test whether fitting a single small rotation that best aligns the two volumes at test time recovers the injected extrinsic error. This would turn the disagreement signal into a self-calibration estimate.
The medical side of the file constrains which designs are worth trying rather than supplying the method. MA-SAM and OmniCT build volumetric context from learned coordinate embeddings instead of rigid rays. We include one variant in which the height branch is replaced by a branch that places features using only learned tri-axial position embeddings. This tests whether "different geometry" can simply mean "no explicit geometry."
Baselines
The naive reference point is ProtoOcc trained exactly as published and evaluated under the same inference-time pose perturbations as every other model. It shows how much a clean-trained dual-branch occupancy model loses when the camera tilts. ProtoOcc reports 39.56% mIoU single-frame (77.9 ms on an RTX 3090) and 45.02% mIoU multi-frame on Occ3D-nuScenes. These are the clean numbers we must reproduce first.
The strongest existing method on this question is that same ProtoOcc, retrained with the identical pose-noise augmentation our model sees. It is the direct competitor because it tests whether robustness comes from the data or from the second geometry. It gets the same schedule, GPU budget, and augmentation magnitudes as our model, plus a matched hyperparameter sweep over noise strength. For clean data we also report the older references CTF-Occ at 28.53 mIoU and BEVFormer at 26.88 mIoU from Occ3D.
A height-only baseline, meaning ProtoOcc with its depth lifting swapped for BEVHeight++-style height lifting, tests whether the height route alone is enough.
The key ablation replaces the height branch with a second depth-lifted branch of the same parameter count and a different initialization. This model is two-branch in the ProtoOcc sense but geometrically uniform, and it isolates the proposal's central ingredient.
A second ablation replaces the learned gate with fixed averaging.
No reported result was found on Lacuna for any occupancy method on Occ3D-nuScenes under extrinsic perturbation, so all perturbed numbers will be produced here under one shared protocol. Beating the baselines means three things together. Clean mIoU must stay within 0.5 points of reproduced ProtoOcc. Under perturbation, mIoU must exceed the augmentation-trained ProtoOcc at every tested magnitude. And the gap to the same-geometry ablation must be larger than seed-to-seed variation.
Why not just…?
Why not just train the existing model with pose-noise augmentation? This is the real threat. Augmentation might teach a single depth-lifted volume to blur features enough to tolerate tilt. We cannot rule this out from the file. BEVHeight++'s robustness evidence compares architectures, not augmentation regimes. The augmented-ProtoOcc baseline is therefore the central comparison. If it matches our model under perturbation, the proposal's claim is wrong. We would then report that the second geometry adds nothing beyond data, which is itself useful to know.
Why not just use height lifting alone? BEVHeight++ itself fuses height with depth rather than replacing depth. That suggests height alone gives up accuracy on clean data, plausibly at long range and on tall structures. The height-only baseline measures this directly on occupancy. If it matches the dual model on both clean and perturbed data, the fusion is unnecessary.
Why not just ensemble two ordinary models? An ensemble also has two branches and might smooth errors. But if both members lift by depth, a tilt shifts them in the same direction, and averaging cannot undo a shared bias. The same-geometry dual-branch ablation is a cheaper and stricter version of this ensemble. If it recovers the robustness gain, our explanation based on decorrelated errors fails.
Why not just flatten to BEV as most driving stacks do? ProtoOcc's own ablation shows that removing its voxel branch costs 1.69 mIoU. That is evidence that collapsing height loses occupancy structure, which is why both of our volumes keep the vertical axis.
Proposed Experiments
The main benchmark is Occ3D-nuScenes, in both single-frame and multi-frame settings, scored by mIoU over 17 classes, with latency measured on matched hardware. Perturbation is applied only at inference. It consists of pitch, roll, and yaw errors at several fixed magnitudes on the order of one to a few degrees, plus a random-jitter condition. The same seeds are used for every model, and we report mIoU as a function of perturbation magnitude, broken down by distance band and class.
A secondary check on Occ3D-Waymo, where CTF-Occ reports 18.73 mIoU, tests whether the effect survives a different sensor setup. A camera-only semantic scene completion run on SemanticKITTI tests the single-camera case. The camera-only SDFormer reports 17.42 mIoU there (SDFormer).
The decisive checks are few.
First, the gate's weight on the height branch should rise with perturbation magnitude, and most strongly at long range, where depth-lift error grows. A gate that does not respond to perturbation would mean the gain comes from capacity.
Second, the same-geometry ablation should lose most of the robustness gain while matching clean accuracy.
Third, our model must beat augmentation-trained ProtoOcc under perturbation without losing more than 0.5 mIoU on clean data.
If the optional test-time alignment recovers the injected rotation to within a fraction of its magnitude, that would be evidence the disagreement signal can do self-calibration. Failure there would not affect the main claim.
References
- ProtoOcc: Accurate, Efficient 3D Occupancy Prediction Using Dual Branch Encoder-Prototype Query Decoder
- BEVHeight++: Toward Robust Visual Centric 3D Object Detection
- Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving
- MA-SAM: Modality-agnostic SAM adaptation for 3D medical image segmentation
- OmniCT: Towards a Unified Slice-Volume LVLM for Comprehensive CT Analysis
- SDFormer: Vision-based 3D Semantic Scene Completion via SAM-assisted Dual-channel Voxel Transformer