Cars and robots often carry several cameras bolted to one rigid body. Depth networks for these camera rigs gain a lot from that fact. If all cameras share one rigid body, the system needs to estimate one vehicle motion per frame instead of one per camera, and the depth maps from different cameras share a single metric scale. The same networks, however, learn depth by checking how well one frame predicts the next, and that check assumes the world stands still. Moving cars and pedestrians break it. The result is depth that flickers from frame to frame and tears apart on exactly the objects that matter most for safety.
This note rests on one observation. Two cameras on a rigid rig that capture images at the same instant see a moving car in the same place, because the car has had no time to move between the two shots. Two frames separated in time do not share that property. When the same-instant check across cameras agrees and the across-time check disagrees, the pattern points to independent object motion, and it requires no segmentation model. We propose an online surround-view depth network with a recurrent memory stored in the vehicle's coordinate frame. The rig's own geometry tells the network which parts of the memory can be carried forward by ego-motion alone and which parts need their own motion estimate. The goal is depth that is metric, stable over time, and correct on moving agents, produced at streaming rates.
Core Claim
A synchronized rigid rig supplies two kinds of geometric constraint. Spatial constraints link cameras at the same timestamp and hold for every point, moving or not, provided the object lies in the overlap region and any trigger-time offset between cameras is accounted for. Temporal constraints are tied to vehicle ego-motion and hold only for static points. For a pixel $p$ we compute two residuals. The first, $r_{\text{sp}}(p)$, is the depth disagreement with a neighboring camera at the same timestamp, found by reprojecting through the fixed camera-to-rig transforms. The second, $r_{\text{tm}}(p)$, is the disagreement with the previous frame after warping by the single rig ego-motion $T_{t-1\to t}$. Both are computed from the network's own depth and the known rig geometry. The motion evidence is
$$m(p) = \sigma\big(\alpha\, r_{\text{tm}}(p) - \beta\, r_{\text{sp}}(p)\big),$$
where $\sigma$ is the logistic function and $\alpha,\beta$ are learned temperatures. A high $m$ means "geometrically consistent now, but not explained by ego-motion," which is the signature of an independently moving object. Pixels where both residuals are high are treated as unreliable, for example occlusions or depth errors, rather than as motion. This separation is something a monocular stabilizer cannot make.
The claim is that this signal is enough to route a recurrent memory. Static memory is warped by rig ego-motion. Memory on dynamic objects is warped by a per-object rigid motion fitted to clustered high-$m$ points. The network should then remove flicker on both background and moving agents without an external segmenter or optical flow.
Relation to Prior Work
Three surround-view methods supply the rig machinery we reuse. All three are trained under a static-world loss. SurroundDepth predicts one vehicle ego-motion and fuses views with cross-view attention, but it processes each multi-camera snapshot independently and has no temporal memory. Volumetric feature fusion builds a shared metric voxel grid and estimates ego-motion from it, using the same static warping loss. Adjacent-view Transformers exchange features between neighboring camera cones, but it is trained with LiDAR supervision and does not model time. UC-NeRF refines rig and ego-poses jointly and names dynamic objects as unaddressed future work. These papers motivate the gap. We keep their single-ego-motion design and add motion-aware memory.
The closest technical relatives approach the problem from the two other directions. DeepVideoMVS supplies the core mechanism: it warps a ConvLSTM hidden state by the previous depth and the known camera motion. It does this for one camera and assumes a static scene, so moving objects contaminate its memory. STS shows that ego-motion carries one camera's view at time $t-1$ into a neighboring camera's view at time $t$, which creates useful baselines across cameras and across time. STS falls back to a monocular branch on fast movers. We reuse this cross-camera handoff as a memory path and try to keep geometry valid on movers instead of giving up on them. STATIC separates static from dynamic regions using the variance of surface normals across frames, works on single cameras, and requires surface-normal supervision. Our separation signal comes from rig geometry instead.
The nearest attempt at all three directions together is Dense Dynamic Scene Reconstruction and Camera Pose Estimation from Multi-View Videos. It is an offline, two-stage bundle adjustment for freely moving cameras, initialized with a feed-forward model and refined with dense flow. It discards rigid extrinsics because its cameras have none. We make the opposite bet: on a rig, the known extrinsics replace the pose graph, and the system can run online.
Two papers constrain the design. SC-DepthV3 reports that excluding dynamic regions from training leaves degenerate depth on those regions, so our method must supervise movers rather than mask them. From-Ground-To-Objects shows that self-supervised movers "float" or "sink" into the road. We borrow its ground-contact regularizer only as an optional coarse-stage term, and test whether rig agreement makes it unnecessary.
Research Program
The model starts from a SurroundDepth-style encoder with shared ego-motion. A ConvLSTM at the bottleneck holds a hidden state for each camera. At each step, the state for camera $i$ is assembled by warping from two sources: camera $i$ at $t-1$, and the neighboring cameras at $t-1$ through the cross-camera temporal overlap described in STS. Points with low $m$ are warped by $T_{t-1\to t}$. Points with high $m$ are grouped into connected clusters in 3D, and each cluster receives a single rigid motion fitted by least squares to its points, which are matched through the warped features. This per-object motion is regularized toward zero velocity unless the evidence is strong.
Training adds a temporal photometric loss to moving clusters that uses the per-object motion, so movers are supervised rather than dropped. Training also adds a same-instant cross-camera depth agreement loss in the overlap regions. When nuScenes camera trigger offsets are not negligible, the same-instant residual is computed after compensating each cluster's motion over the known offset.
The main risk is coverage. The same-instant residual exists only in narrow overlaps. We address this in two ways. First, the cross-camera temporal handoff sweeps objects through overlap regions over time. Second, a small head is distilled from the geometric $m$ to predict motion everywhere, with geometric $m$ acting as its teacher where overlaps exist. We will report how often moving objects ever pass through overlap regions. If that fraction is low, the claim weakens to "a geometric teacher for a learned motion head," and we will say so.
Baselines
The naive reference is per-frame SurroundDepth with no memory. On DDAD it reports Abs Rel 0.200 (0.208 without test-time scaling), and on nuScenes it reports 0.245 (SurroundDepth). We retrain it on our splits and our compute budget so that its temporal metrics, which have no reported values, are measured under identical conditions. Beating it means matching or improving Abs Rel while clearly reducing frame-to-frame jitter.
The strongest self-supervised accuracy figure in the file is SC-DepthV3, which reports Abs Rel 0.142 on DDAD (SC-DepthV3). That result is monocular and uses an external pseudo-depth teacher. We will run it per camera on our DDAD split with the same backbone budget and report it as the accuracy bar. We do not expect to beat it on whole-image Abs Rel without a teacher. The claim we test against it is better temporal stability and better accuracy inside moving-object boxes. A version of our model with its pseudo-depth term added gives a like-for-like comparison.
AVT-SSDepth's 0.067 on nuScenes is LiDAR-supervised and is cited only as an upper reference (Adjacent-view Transformers).
The temporal-consistency competitor is SurroundDepth followed by NVDS+ applied independently to each camera stream (NVDS+), using its released small model. NVDS+ reports an improvement of over 18% in its temporal-consistency metric on its own video benchmark. No surround-view number is reported for it on Lacuna. Beating this baseline means equal or lower jitter together with lower depth disagreement between cameras.
Three ablations isolate the key ingredient. The first is rig memory with ego-motion-only warping, which is DeepVideoMVS extended to the rig and should smear moving objects. The second replaces $m$ with masks from an off-the-shelf 2D instance segmenter. The third replaces $m$ with oracle masks derived from the annotated boxes of moving objects, which gives an upper bound. All variants share the same backbone, training schedule, and hyperparameter search budget.
Why not just…?
Why not just mask movers with a 2D segmenter and keep the static pipeline? This is the most serious alternative. If segmenter-routed memory matches our geometric routing, the geometry adds nothing, and we would reframe the work around the per-object memory warp alone. Two pieces of evidence suggest it will fall short. SC-DepthV3 found that dropping dynamic regions degrades depth on them. Split4D reports that drift in 2D labels corrupts 4D models, with mIoU of 0.893 for its native motion approach versus 0.592 for SA4D. A segmenter also cannot tell a parked car from a moving one, so it would discard valid static supervision on parked vehicles. The segmenter and oracle ablations will settle the question, with results reported separately for parked and moving vehicles.
Why not run a depth foundation model per frame and stabilize afterwards? Per-camera stabilizers give each camera its own scale. SurroundDepth reports that scale-ambiguous baselines exceed Abs Rel 0.96 without test-time scaling. DVD notes that its affine stitching fails under extreme ego-motion, and it runs a 1.3-billion-parameter model for each camera. We will still include DVD-style alignment on a short DDAD subset, scaled using the rig, as an offline reference for accuracy and latency.
Why not penalize depth changes along precomputed optical flow? Flow is least reliable on fast movers and in occluded regions. NVDS+ has to down-weight pixels with high motion, and STATIC avoids flow entirely for this reason (STATIC). We will test a flow-warping-loss variant of SurroundDepth, using the same flow network budget, as a further baseline.
Proposed Experiments
We train self-supervised on DDAD and nuScenes and report Abs Rel, Sq Rel, RMSE, and RMSlog against LiDAR. Temporal consistency is measured in two ways. For static structure, we take LiDAR points accumulated across frames, compensated for ego-motion, and measure the frame-to-frame variance of predicted depth at those fixed world points. For moving agents, we use the nuScenes 3D box tracks: the box gives the object's true trajectory, and we measure depth error and jitter inside boxes with nonzero annotated velocity. Cross-camera tearing is measured as the depth disagreement for boxes that straddle two views. Latency is reported on one GPU and compared against SurroundDepth's reported 0.088 seconds for six images.
The decisive checks are threefold. First, whether $m$ separates moving from parked vehicles: we report AUROC against box velocity, alongside the fraction of movers that ever enter an overlap region. Second, whether geometric routing approaches the oracle-mask upper bound and beats segmenter masks on jitter inside moving boxes. Third, whether ego-motion-only rig memory measurably smears movers, which is the failure DeepVideoMVS predicts. If geometric routing does no better than the segmenter, or if overlap coverage is negligible, the central claim fails.
References
- SurroundDepth: Entangling Surrounding Views for Self-Supervised Multi-Camera Depth Estimation
- Self-supervised surround-view depth estimation with volumetric feature fusion
- Adjacent-view Transformers for Supervised Surround-view Depth Estimation
- UC-NERF: Neural Radiance Field for Under-Calibrated Multi-View Cameras in Autonomous Driving
- DeepVideoMVS: Multi-View Stereo on Video with Recurrent Spatio-Temporal Fusion
- STS: Surround-view Temporal Stereo for Multi-view 3D Detection
- STATIC : Surface Temporal Affine for TIme Consistency in Video Monocular Depth Estimation
- Dense Dynamic Scene Reconstruction and Camera Pose Estimation from Multi-View Videos
- SC-DepthV3: Robust Self-supervised Monocular Depth Estimation for Dynamic Scenes
- From-Ground-To-Objects: Coarse-to-Fine Self-supervised Monocular Depth Estimation of Dynamic Objects with Ground Contact Prior
- Neural Video Depth Stabilizer
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- DVD: Deterministic Video Depth Estimation with Generative Priors