Systems that watch a scene through several calibrated cameras and predict which points in 3D space are occupied, such as a crowded plaza or the surroundings of a car, are increasingly trained on synthetic renders, because simulators label every voxel for free. They then lose accuracy on real footage, and they are too slow for small embedded computers. This proposal replaces the deep stack of distinct 3D layers that current systems use with one small update block applied over and over. Each pass re-reads evidence from every camera and refines the 3D estimate. Because the same block runs at every step, the model can be told to run more passes on hard inputs and fewer on easy ones, all with a single set of weights.
The central bet is that real images behave like "harder instances" of the synthetic training problem. They have sensor noise, imperfect calibration, and unfamiliar textures. A solver that learned a reusable rule for making the camera views agree with each other should improve on real scenes when it is given extra passes, while a feed-forward network trained to a fixed depth cannot. Adaptive stopping, where empty space halts after one or two passes and ambiguous regions keep iterating, turns the same mechanism into an average-latency saving on edge hardware.
Core Claim
A weight-tied update block that re-samples multi-camera image features at every iteration should do three things when trained only on synthetic multi-view data, compared with an unshared decoder of equal per-pass cost. It should transfer at least as well zero-shot to real multi-camera footage. It should keep improving on real footage when run for more iterations than it was trained with. It should spend fewer iterations on average when allowed to halt each 3D query independently. If the extra-iterations effect does not appear on real data, the proposal reduces to parameter compression, which on its own is not worth doing (see below).
Writing $q_t$ for the set of 3D query features (one per candidate location) after $t$ passes, the update is
$$q_{t+1} = q_t + f_\theta\big(q_t,\; S(F_1,\dots,F_V; P, q_t),\; e_t\big),$$
where $F_1,\dots,F_V$ are the 2D feature maps from the $V$ cameras, $P$ the camera projection matrices, $S$ a sampling operator that projects each query's current 3D position into every view and gathers features there, $e_t$ a small embedding of the iteration index, and $f_\theta$ the single shared block. A query stops updating once $\lVert q_{t+1}-q_t\rVert$ falls below a threshold $\epsilon$. Occupancy is decoded from whatever $q_t$ a query ends with.
Where this sits in prior work
The two dominant ways of fusing views motivate the design. Forward methods push 2D features along camera rays into a voxel grid and then clean the grid with deep 3D convolutions. These include the voxel fusion in Self-supervised surround-view depth estimation with volumetric feature fusion, TesseTrack, which shows that fusing in 3D beats triangulating per-view predictions, and OmniOcc in Multi-View Pedestrian Occupancy Prediction with a Novel Synthetic Dataset, which lifts ResNet features with a parameter-free projection and refines them with a 3D U-Net. Backward methods let 3D queries look up image features through attention guided by camera geometry. These include BEVFormer, SurroundOcc, Occupancy as Set of Points, and the iterative proposal refinement in SimMOD. Several of the backward methods already refine in stages, but every stage has its own weights and the number of stages is fixed. We keep the backward re-sampling and make the stage a shared, repeatable operator.
The method comes from the weight-sharing literature. Iterative residual refinement for optical flow (IRR) and unrolled optimization for demosaicking (cascade of residual denoisers) show that one shared block can learn a consistent update rule for dense vision problems. Can You Learn an Algorithm? and Expressive Power of Implicit Models show that weight-tied models can solve harder instances by running longer at test time. That is the property we want to transfer from "harder puzzle" to "real rather than synthetic image". Two papers constrain the design. On Expressive Power of Looped Transformers proves that pure weight tying limits what the model can express and fixes this by injecting the loop index into normalization scales; we adopt this as $e_t$. Fixed-Point Reasoners warns that looped models drift or explode without contractive training; we adopt its stabilization and treat divergence as a failure mode to measure. There is a real tension here. An index embedding helps expressivity but could break running past the trained depth. We therefore hold $e_t$ fixed at its last trained value beyond the training horizon, and we ablate this choice.
The sim-to-real work sets the evaluation. OmniOcc is the only file entry that trains multi-view occupancy on synthetic data (MVP-Occ) and tests on real multi-camera footage (WildTrack). It is our direct comparison, and it uses no weight sharing and no adaptive compute. CTS and Driving in the Matrix show that synthetic-trained models overfit to simulator geometry and appearance. They address this with pseudo-label adaptation and data scale, both of which could be layered on top of our model. EUREKA / IRAIS shows that physically complete synthetic supervision can transfer, but it works on single views.
Edge efficiency is a constraint, not the headline. MiniViT and MiniALBERT both state that weight sharing cuts parameters but not compute. This is exactly why parameter savings alone would not justify the project, and why per-query halting is essential. SparseOcc and ToC3D save compute by permanently discarding empty voxels or background tokens, and SparseOcc notes that discarded thin or distant structure cannot be recovered. Our halting is softer. A halted query keeps its state and can be reawakened if a neighbor's update changes the evidence it samples, in the spirit of the reversible idling in IdleViT.
Research Program
We would build the solver on OmniOcc's front end, keeping the same ResNet+FPN backbone, the same parameter-free projection to an initial voxel grid, and the same output heads. The 3D U-Net is replaced by the shared block $f_\theta$. The block is a small transformer layer that does four things: attention among nearby queries, deformable sampling of each query's projection into all $V$ views, a feed-forward sublayer, and index-modulated normalization. Keeping the front end fixed means any change in transfer comes from the aggregation stage.
Training uses MVP-Occ only, with a randomly drawn iteration count per batch, between a small minimum and $T_{train}$. This forces every intermediate state to be decodable, which is what makes halting and extrapolation meaningful. A per-step contractive penalty follows Fixed-Point Reasoners. The halting threshold $\epsilon$ is not trained; it is swept at test time.
The analysis has three parts. First, we measure zero-shot WildTrack accuracy as a function of the test iteration count $T_{test}$, from 1 to several times $T_{train}$. Second, we measure how iterations are distributed across queries on synthetic versus real inputs, to check whether real scenes demand more passes and where those passes go: occlusion boundaries, distant people, view-overlap seams. Third, we deliberately perturb the camera extrinsics on synthetic data to test whether extra iterations absorb calibration error. The file flags calibration error as a weakness of all projection-based methods (SimMOD). We would also time mean and worst-case latency on an embedded GPU at batch size one. Worst case matters because adaptive compute can raise it.
Baselines
The naive reference is OmniOcc's front end with no 3D refinement at all: the lifted grid is decoded directly. It tells us how much any refinement stage contributes on WildTrack.
The strongest existing method is OmniOcc itself, from Multi-View Pedestrian Occupancy Prediction with a Novel Synthetic Dataset. It reports 75.4% MODA and 87.5% F1 on WildTrack for 2D occupancy when trained on the synthetic Facade scene. It is the right competitor because it is the only published multi-view synthetic-to-real occupancy system in the file, and it defines the benchmark protocol. We would retrain it with its released recipe and report our reproduced number next to the published one. Our model uses the same backbone initialization, the same Facade training scene, the same augmentation, the same number of training steps, and an equal hyperparameter search budget. Beating it means exceeding the reproduced MODA and F1 at equal or lower mean per-frame compute, with seed variance reported so that a difference inside run-to-run noise is not counted.
The key ablation is an unshared decoder built from $T_{train}$ copies of the same block with distinct weights, trained identically. It has the same per-pass compute and the same re-sampling, so it isolates weight tying as the only difference. We give it per-layer output heads so it can also early-exit, which isolates the one thing it cannot do: run past its depth. A second ablation keeps weight tying but removes re-sampling, so the block iterates on the lifted grid alone. This tests whether re-reading the cameras is what makes extra iterations useful. A third ablation removes the index embedding $e_t$, and a fourth lets $e_t$ continue beyond the training horizon, to measure the expressivity-versus-extrapolation tension.
For in-domain efficiency, we would port the solver to Occ3D-nuScenes and compare it with SparseOcc. SparseOcc reports 34.0 RayIoU with 8 history frames at 17.3 FPS on an A100. The purpose is to check that the design is not uncompetitive on real driving data; no sim-to-real claim is made there. We would run single-frame versions of both for a fair comparison. No reported result was found on Lacuna for a synthetic-to-real surround-camera occupancy benchmark in driving, so driving transfer is out of scope.
Why not just…?
Why not just keep OmniOcc's unshared decoder and add stronger domain randomization or pseudo-label adaptation? This is a real threat to the transfer claim. CTS shows large gains from mean-teacher adaptation in LiDAR detection, and Driving in the Matrix shows that data scale alone narrows the gap. These techniques are complementary rather than competing, so we would apply the same augmentation and, in a second round, the same pseudo-label loop to both the shared and unshared models. If the shared model's advantage disappears once both are adapted, then the transfer claim fails, and only the adaptive-compute claim remains to be tested.
Why not just shrink or prune the 3D decoder for edge speed? Pruning in the style of SAViT or Width & Depth Pruning produces one fixed-cost model. It cannot spend more compute on a hard real frame and less on an empty one. If a pruned OmniOcc with our mean compute matches our WildTrack accuracy, then adaptivity buys nothing and the proposal should drop the efficiency claim. The compute-matched comparison would show this directly.
Why not just add early exits to an unshared decoder? This is the closest alternative, and it is why the unshared ablation includes early-exit heads. An early-exit decoder can save compute, but it cannot exceed its trained depth. The proposal's distinctive prediction is that real-domain accuracy keeps rising for $T_{test} > T_{train}$. If that curve is flat, early-exit unshared decoders are the simpler and better choice, and we would say so.
Why not just skip training and use foundation models at test time? TT-Occ avoids synthetic training entirely. However, it runs several heavy foundation models per frame, reaches only 16.70 mIoU camera-only on Occ3D-nuScenes, and is not evaluated on WildTrack, so it serves as a latency and accuracy reference rather than a like-for-like baseline.
Decisive Checks
Three results would settle the proposal. The first is a curve of WildTrack MODA against $T_{test}$ that rises past $T_{train}$ for the shared model with re-sampling, and not for the variant without re-sampling. The second is a zero-shot MODA and F1 that match or exceed reproduced OmniOcc at no greater mean compute, with the unshared same-cost ablation performing worse. The third is a halting sweep in which mean iterations per query fall well below $T_{train}$ on WildTrack at under one point of MODA loss, while worst-case latency stays within the edge budget. The idea is also supported if real frames consume measurably more iterations than synthetic frames, concentrated at occlusions and view seams. If the first check fails, the idea is not supported: the remaining benefit would be parameter compression, which MiniViT already provides.
References
- Self-supervised surround-view depth estimation with volumetric feature fusion
- TesseTrack: End-to-End Learnable Multi-Person Articulated 3D Pose Tracking
- Multi-View Pedestrian Occupancy Prediction with a Novel Synthetic Dataset
- BEVFormer: Learning Bird's-Eye-View Representation from Multi-camera Images via Spatiotemporal Transformers
- SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving
- Occupancy as Set of Points
- A Simple Baseline for Multi-Camera 3D Object Detection
- Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation
- Deep Image Demosaicking Using a Cascade of Convolutional Residual Denoising Networks
- Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks
- Expressive Power of Implicit Models: Rich Equilibria and Test-Time Scaling
- On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encoding
- Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers
- CTS: Sim-to-Real Unsupervised Domain Adaptation on 3D Detection
- Driving in the Matrix: Can virtual worlds replace human-generated annotations for real world tasks?
- Amodal Instance Segmentation with IRAIS Dataset for Sim-to-Real Transfer
- MiniViT: Compressing Vision Transformers with Weight Multiplexing
- MiniALBERT: Model Distillation via Parameter-Efficient Recursive Transformers
- Fully Sparse 3D Occupancy Prediction
- Make Your ViT-Based Multi-view 3D Detectors Faster via Token Compression
- No Token Left Behind: Efficient Vision Transformer via Dynamic Token Idling
- SAViT: Structure-Aware Vision Transformer Pruning via Collaborative Optimization
- Width & Depth Pruning for Vision Transformers
- Test-Time 3D Occupancy Prediction