Turning one camera image into a full 3D map of a scene runs into two problems that are usually studied separately. The first is cost. A dense grid of 3D cells (voxels) grows with the cube of its resolution, so modern systems only process the small fraction of cells likely to contain surfaces. The second is ambiguity. A single image cannot tell a small nearby object from a proportionally larger one farther away, so any depth it yields carries an uncertain overall scale. The two problems interact badly. Sparse scene-completion networks decide which cells to process by trusting a monocular depth map. When that map has the wrong scale, the network places its entire compute budget in the wrong part of space, and the true surfaces are never examined.
We propose to treat global metric scale as an explicit, small set of hypotheses inside the sparse pipeline, rather than a fixed property of the input depth map. For each image, the method places a sparse set of 3D queries at a handful of candidate scales. A query is a cell that the network will actually process. A lightweight scorer then picks the scale that best agrees with image features, and only the winning hypothesis is completed. The bet is that most monocular depth error is one per-image scale factor rather than independent per-pixel noise. If that holds, covering K scales costs about K times the queries of one depth map. That is far cheaper than widening the search band along every ray, and much cheaper than a dense grid.
Core Claim
Under a fixed query budget, sparse monocular semantic scene completion becomes more robust to camera changes and scale errors when a few global-scale hypotheses decide where queries go. Spreading queries uniformly along each ray, or trusting a single better depth map, should do worse. The claim has two parts, and each can fail on its own. The first part is empirical: most of the depth error that misplaces queries is captured by a single per-image scale factor. The second part is architectural: the network can recover that factor from its own early features, without a focal-length tag or extra sensors.
Formally, let $d_i$ be the predicted depth at pixel $i$. We assume true depth $z_i \approx s\, d_i (1+\epsilon_i)$, where $s$ is a per-image scale and $\epsilon_i$ is residual per-pixel error. Queries are placed at the voxels hit by $s_k d_i$ for a grid of candidate scales $s_1,\dots,s_K$, and the scorer outputs a distribution over $k$. The query count grows as $K \cdot P$ for $P$ lifted pixels. Uniform band widening must grow with the band width in voxels along each ray, which becomes large at long range, where monocular error is greatest.
Where This Sits
Several papers motivate the problem. VoxFormer lifts pixels with an off-the-shelf monocular depth estimator and turns occupied cells into sparse queries. Its authors acknowledge that depth errors misplace those queries and that the queries become too thin beyond roughly 30–50 m. SparseOcc prunes empty queries greedily, from coarse to fine, down to about 5% occupancy. A cell pruned early can never return, and SparseOcc relies on calibrated multi-camera video to get depth right. We keep the sparsity of both systems. What we change is that the first allocation step stops committing to a single metric scale.
Other work supplies the scale machinery. Metric3D and Metric3D v2 show that focal length is the main source of metric ambiguity. They normalize images to a canonical camera, which gives low zero-shot error but requires correct focal-length metadata at inference. We use Metric3D v2 as a front-end and test what happens when that metadata is wrong or missing. Monocular Visual-Inertial Depth Estimation supports our error model in spirit: a closed-form global scale fit does most of the work, and a local refinement network adds per-pixel corrections. That paper gets its scale from inertial-odometry points. We have to infer it from the image and the 3D context.
The closest conceptual relative is Ray-ONet. It separates shape from absolute scale using a scale factor computed from focal length and an estimated object distance, and it predicts along rays to avoid cubic query cost. Ray-ONet works on single objects, takes the scale as given, and does not complete hidden geometry. We work on whole scenes, treat the scale as an unknown that is selected among hypotheses, and keep VoxFormer-style completion of occluded space.
Dense methods constrain the design from the other side. MonoScene and NDC-Scene never commit to a depth map, so they cannot misplace queries, but they pay dense-grid memory. NDC-Scene's camera-aligned frustum also motivates our choice to sweep scale along rays rather than in world space. VPOcc and MonoOcc are the strongest reported single-camera competitors on the KITTI-family benchmarks.
Research Program
The work proceeds in three stages, and the first stage can falsify the idea before any architecture is built.
The first stage is an error-decomposition study. On SemanticKITTI and SSCBench-KITTI360, we run several front-ends on the validation splits: VoxFormer's default estimator, Metric3D v2 with correct focal length, and Metric3D v2 with focal length perturbed by ±20–40% or replaced by a dataset-default guess. For each image we fit the best single scale against projected LiDAR. We then report what fraction of query misplacement is removed by that one scalar. Query misplacement is measured as the share of ground-truth occupied voxels that fall outside the query set, broken out by distance band.
If one scalar recovers most of the lost recall, the method proceeds. If the residual per-pixel error dominates, scale hypotheses cannot help much beyond band widening. In that case we would report the negative result and redirect the effort toward per-ray uncertainty instead.
The second stage builds the method on VoxFormer's two-stage design.
Stage 1 places queries at $K$ log-spaced scales, about 3–7 of them, around the front-end depth. The scale range is set from the spread measured in the first study. Each query attends to image features through deformable cross-attention, as in VoxFormer. A small scorer then pools the features of each hypothesis and outputs a softmax over scales. We expect visible-surface hypotheses at the wrong scale to sample features inconsistently, for example road texture landing on cells that should hold cars, or ground cells floating above the true ground plane.
The scorer is trained with a cross-entropy target equal to the hypothesis nearest the LiDAR-fit scale, computed only at training time. At inference, the top hypothesis, or the top two if their probabilities are close, is passed to VoxFormer's completion stage. That stage spreads structure from the visible queries into hidden cells.
A continuous refinement of the selected scale is an optional add-on, and we will report its value separately.
The third stage is a cross-camera and robustness evaluation, which is where the method should matter most. We train on KITTI and test on KITTI-360's different camera. We also synthesize focal-length shifts by crop-and-resize and corrupt the metadata given to Metric3D v2. Finally, on NYUv2 we check whether the same idea transfers indoors, where scale cues come from furniture rather than roads.
Baselines
The do-nothing reference is VoxFormer exactly as published, with its own depth front-end. It anchors the cost of a single committed scale. The cited SemanticKITTI test figure is 12.20 mIoU, reported in MonoOcc. We retrain it in our codebase so that all comparisons share data splits, schedule, and GPU budget.
The strongest simple competitor is VoxFormer with Metric3D v2 as its front-end, given correct focal length. It tests whether a better single depth map already closes the gap. We tune its schedule with the same number of trials as our method. It also serves as the fair base for our method, which uses the same front-end.
The strongest reported single-camera numbers in the file are MonoOcc-L at 15.63 mIoU on the SemanticKITTI test set (MonoOcc) and VPOcc at 46.39 IoU and 19.80 mIoU on SSCBench-KITTI360 (VPOcc). Our in-distribution aim is not to beat these. The aim is to show a gain over the matched VoxFormer–Metric3D v2 baseline, and to show that the same module added to MonoOcc's sparse stage does not lower MonoOcc's score.
Out of distribution, under shifted focal length or a new camera, the claim is that our method loses substantially less mIoU than every baseline. "Beating" means a smaller drop at equal query count and memory, with seed variance reported.
The ablation that isolates the key ingredient is uniform ray-band dilation at a matched query budget. It spends the same number of extra queries spread evenly along each ray around the front-end depth, rather than at discrete global scales.
Two further ablations separate allocation from selection. One takes the union of all K hypotheses with no scorer. The other uses K=1. Finally, an oracle that selects the LiDAR-fit scale bounds how much the scorer could ever recover. A dense reference, NDC-Scene at 12.70 SemanticKITTI mIoU and 29.03 NYUv2 mIoU (NDC-Scene), is reported alongside its memory use to show the accuracy–cost frontier.
Why not just…?
Why not just plug Metric3D v2 into VoxFormer and stop there? This is the most serious threat. With correct focal length, Metric3D v2 reports zero-shot Abs Rel of 0.044 on KITTI (Metric3D v2), which may leave little scale error to fix in-distribution. We expect our in-distribution gain over this baseline to be small and will say so. The case for the method rests on the setting the Metric3D authors themselves flag: wrong or missing focal-length metadata, where canonical normalization inverts incorrectly. The focal-perturbation and cross-camera experiments settle this directly. If the plain Metric3D v2 pipeline degrades no more than ours there, the idea has no value.
Why not just widen the query band along every ray? Band widening makes no assumption about error structure, so it is the natural rival. Its cost grows with band width, and monocular error grows with distance, so at long range it should either exceed the budget or thin the queries out, which is the far-field failure VoxFormer already reports. The matched-budget ablation decides this. The first-stage error decomposition predicts in advance which approach should win: if residual per-pixel error dominates, band widening should tie or win, and we will report that outcome.
Why not just use a dense grid and avoid query misplacement entirely? Dense frustum methods cannot drop cells, but MonoScene uses about 150M parameters against VoxFormer's 60M and needs more GPU memory (VoxFormer). SparseOcc reports that occupancy near 5% is enough and that dense alternatives run 4–6x slower (SparseOcc). We report NDC-Scene's accuracy and memory next to ours. If dense methods prove much more robust to camera shift at acceptable cost, that is a meaningful counterpoint, and we will state it.
Why not just train with heavy scale and focal-length augmentation? Augmentation can teach the depth front-end a range of cameras. Metric3D notes that mixing cameras without normalization yields relative rather than metric depth, because a single image cannot reveal focal length. Augmentation therefore makes the front-end hedge, not resolve. It does not reach the sparse allocation step, where any hedging is lost once one depth map is chosen. As a further baseline we include VoxFormer trained with the same crop-and-resize focal augmentation used to build our test shifts.
Proposed Experiments and Decisive Checks
Metrics are geometric completion IoU and semantic mIoU on SemanticKITTI, SSCBench-KITTI360, and NYUv2. We also report RayIoU from SparseOcc, which penalizes artificially thick surfaces, a likely way to cheat when queries are spread over several scales. Each is broken down by distance band. Two diagnostics trace the mechanism: query recall, meaning the share of ground-truth occupied voxels inside the query set, and scorer accuracy against the LiDAR-fit scale. Peak memory and frames per second are reported for every method at matched query count.
Three checks decide the outcome. The first is the error-decomposition study, which tests whether one per-image scale explains most of the lost query recall. The second is the matched-budget comparison against band dilation, which tests whether allocating by hypotheses beats spreading queries evenly at long range. The third is the focal-shift curve, which tests whether the method's mIoU falls more slowly than the Metric3D v2–VoxFormer pipeline as focal error grows from 0 to 40%. If the first check fails, we stop and report the error structure. If the second fails, the contribution reduces to an analysis paper. If the third fails, canonical camera normalization is already enough, and that is worth knowing.
References
- VoxFormer: Sparse Voxel Transformer for Camera-Based 3D Semantic Scene Completion
- Fully Sparse 3D Panoptic Occupancy Prediction
- Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Image
- Metric3D v2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimation
- Monocular Visual-Inertial Depth Estimation
- Ray-ONet: Efficient 3D Reconstruction From A Single RGB Image
- MonoScene: Monocular 3D Semantic Scene Completion
- NDC-Scene: Boost Monocular 3D Semantic Scene Completion in Normalized Device Coordinates Space
- VPOcc: Exploiting Vanishing Point for 3D Semantic Occupancy Prediction
- MonoOcc: Digging into Monocular Semantic Occupancy Prediction