Systems that recover 3D depth from ordinary video rely on hand-written rules of thumb. Road surfaces and walls are flat, things that do not move should look the same from frame to frame, and depth jumps sharply at the edge of an object. Each rule helps where it is true and harms where it is not. A flatness rule smooths away a curb or a pedestrian. A "nothing moves" rule fills the road with ghost geometry wherever a car passes. Current methods decide where to apply these rules in one of two ways. Some apply them everywhere. Others decide with semantic labels such as "road" or "wall", which only stand in for the property that actually matters. We propose to learn the decision directly. In simulation, ground truth tells us exactly which pixels are locally planar, which are static, which belong to rigidly or non-rigidly moving objects, and which lie on a contact line where a moving object touches a supporting surface. We train a small network to predict this per-pixel geometric regime map, then use it on real video to switch each regularizer on or off.
The bet is that regime labels transfer from simulation to reality better than appearance does, provided the regime predictor sees geometric evidence and not raw texture. That evidence includes the depth network's current estimate, the surface normals derived from it, and the residuals from checking consistency between neighboring frames. Simulated renderings differ from real photos mostly in lighting, materials, and sensor noise. Whether a patch of the scene is flat and still is a fact about the 3D scene itself, so a network that sees only geometric cues should have far less room to pick up rendering quirks. If this holds, simulation's richest output is not synthetic training images. It is supervision for the meta-decision of which geometric prior to trust where.
Core Claim
Self-supervised monocular depth on dynamic driving scenes will improve over the strongest synthetic-to-real method in the file when its losses are gated by a regime map learned in simulation. The gains should concentrate in three places: large planar regions, dynamic objects, and object–ground contact bands. The regime map should also beat gating by open-vocabulary semantic masks, because category and geometric regime come apart often enough to matter. Parked cars belong to a "dynamic" class but are static. Roads bend and bank. Facades carry balconies. A wheel touching asphalt is neither a depth discontinuity nor part of the plane.
Relation to Prior Work
Two lines of work motivate the gap. On the dynamic side, SGDepth shares an encoder between depth and segmentation heads. It removes moving cars from the photometric loss but uses a cross-frame mask-overlap check to keep parked ones. That patch shows class membership alone is the wrong signal. S3Net combines translation of synthetic images by a generative adversarial network with forward/backward auto-masking. Back2Color reverses the usual direction: it trains a network to paint real-looking color onto synthetic depth, and it fuses video and stereo cues with learned per-pixel uncertainty. None of the three uses a surface prior. Back2Color's authors report residual texture artifacts on large surfaces, which is exactly where a planarity prior should help.
On the surface-prior side, P2Net enforces a piecewise-planar loss over color superpixels, which are clusters of similar-looking pixels. PlanarGS prompts GroundedSAM with words like "wall" and "floor" to pick out planes for 3D Gaussian splatting, a scene representation built from many small colored ellipsoids. DynaSurfGS applies normal and as-rigid-as-possible constraints uniformly in dynamic scenes. Its authors note that this over-smooths fine detail.
These papers constrain the design in two ways. P2Net and DynaSurfGS show that uniform priors oversmooth. PlanarGS shows that semantics-gated priors work but assume a static scene and inherit any errors the foundation model makes.
Two papers supply methods we reuse. MoDOT trained on synthetic occlusion-boundary data and transferred zero-shot to real indoor benchmarks. This is the closest evidence that geometric structure learned in simulation transfers. We reuse its boundary-depth constraint idea, but we split boundaries into two kinds: occlusion boundaries, where depth should jump, and contact boundaries, where depth should be continuous. MoDOT does not draw this distinction. Split4D names contact zones as the place where its static/dynamic decomposition bleeds, which motivates the contact class.
Research Program
We first derive regime labels from Virtual KITTI 2, which provides depth, scene flow, instance masks, and multiple weather conditions. A pixel is labeled static-planar when two conditions hold: its back-projected neighborhood fits a plane within a small residual, and its scene flow equals what camera ego-motion alone would produce. It is dynamic-rigid or dynamic-deforming according to whether its instance's scene flow fits a single rigid motion. It is contact when it lies within a few pixels of the boundary between a moving instance and a static-planar region, with continuous depth across that boundary. It is occlusion when depth jumps there. Everything else is static-free: static but curved or detailed.
The regime predictor $R$ outputs a probability per regime and pixel. Its input is deliberately restricted to the depth network's predicted depth, the normals computed from it, and the photometric and flow residuals between neighboring frames. It does not see the RGB image. Because those inputs come from the depth network, we train $R$ on simulated sequences while that network is also trained there, then freeze $R$ and carry it to real KITTI.
On real video, the depth network's loss becomes a regime-weighted sum:
$$ L = \sum_{p} \Big[ (1 - r_{\text{dyn}}(p))\, L_{\text{photo}}(p) + \lambda_{\text{pl}}\, r_{\text{pl}}(p)\, L_{\text{plane}}(p) + \lambda_{\text{c}}\, r_{\text{c}}(p)\, L_{\text{cont}}(p) + \lambda_{\text{o}}\, r_{\text{o}}(p)\, L_{\text{step}}(p) \Big] $$
The symbols are as follows. $r_{\text{dyn}}$, $r_{\text{pl}}$, $r_{\text{c}}$ and $r_{\text{o}}$ are $R$'s probabilities for dynamic (rigid plus deforming), static-planar, contact, and occlusion at pixel $p$. $L_{\text{photo}}$ is the standard frame-to-frame photometric reconstruction loss. $L_{\text{plane}}$ is the distance of back-projected points to a plane fit over the local high-$r_{\text{pl}}$ neighborhood. $L_{\text{cont}}$ penalizes depth gradients across contact pixels. $L_{\text{step}}$ is a MoDOT-style hinge that requires a depth jump at occlusion pixels.
Because the loss is a drop-in replacement, it can be added on top of Back2Color's pipeline without other changes. We also run a second, lighter study that applies the same gating to the regularizers of a dynamic Gaussian-splatting reconstruction. It is exploratory and is not the main claim.
Baselines
The naive reference point is the same depth network and backbone trained with the plain photometric loss and no masks or priors. The file has no number for this configuration, so we will report our own run.
The semantic-masking reference is SGDepth, which reports Abs Rel 0.107 on the KITTI Eigen split at 1280×384. Abs Rel is the absolute depth error divided by true depth, averaged over pixels.
The strongest existing method is Back2Color, which reports Abs Rel 0.087 on KITTI with its VAN2 backbone. Because Back2Color uses stereo pairs during training and several synthetic sources, we will retrain it under our exact data: vKITTI2 plus KITTI, the same stereo availability, the same backbone, and the same number of epochs. We then add our gated loss to that retrained model. Beating it means two things. The first is a consistent Abs Rel reduction across three seeds over the retrained Back2Color. The second is a larger reduction in the region-specific errors on planar, dynamic, and contact pixels defined below. We do not expect to match a mixed-source 0.087 if our retrained run lands higher, so the comparison is made within the matched setting.
The uniform-prior baseline applies P2Net's superpixel planar loss everywhere, with its weight tuned on the same validation split.
The decisive competitor is semantic gating. Here GroundedSAM masks for "road, building, wall" gate the planar loss, and SGDepth-style dynamic-class masks gate the photometric loss. This baseline gets the same tuning budget as our method.
Two ablations isolate the key ingredient. The first gives the regime predictor RGB input instead of geometry-only input. The second removes the contact/occlusion split by merging the two classes.
Why not just…?
Why not just gate the priors with an off-the-shelf segmenter, as PlanarGS does? This is the real threat, and the experiment is built to settle it. Split4D shows that relying on a 2D tracker costs a lot on dynamic data: its own method reaches 0.893 mIoU against 0.592 for the tracking-dependent SA4D on Multi-Human. SGDepth needed an extra check just to recover parked cars. Still, if semantic gating matches the regime map on planar and dynamic regions, the claim narrows to contact bands. If it matches there too, the idea fails, and we will say so.
Why not just apply a planar loss everywhere? DynaSurfGS reports over-smoothing under uniform regularization. P2Net's superpixels are color clusters that will merge cars with roads. The uniform-prior baseline measures this cost directly on curbs, vehicles, and pedestrians.
Why not just make the simulator more realistic, or fine-tune depth on synthetic ground truth? Virtual KITTI 2 halved the gap relative to its predecessor, yet a gap remains: stereo error is 0.1259% on the synthetic clone versus 0.1579% on real KITTI. Our approach never asks synthetic appearance to transfer. The RGB-input ablation tests whether keeping appearance out of the regime predictor is what makes transfer work.
Why not just rely on Back2Color's learned uncertainty to down-weight bad regions? Uncertainty tells the network where to trust a cue less. It does not supply a replacement constraint, which is why textureless planes still show artifacts. Stacking our loss on Back2Color tests whether the two are complementary.
Proposed Experiments
The main evaluation is KITTI Eigen, reporting Abs Rel, RMSE (root-mean-square depth error), and a1 (the fraction of pixels within 25% of true depth). We add region-specific errors computed against projected LiDAR. These cover road and facade pixels, measured both as Abs Rel and as plane-fit residual of back-projected points. They also cover dynamic-instance pixels, using motion masks from KITTI InstanceMotSeg, and contact bands, defined as pixels within a few rows of an instance's lower boundary on the road. Zero-shot results on Make3D test outdoor generalization.
To check whether the regime predictor itself transfers to real data, we measure its dynamic-regime IoU against InstanceMotSeg masks and its planar-regime precision against local plane fits to KITTI LiDAR. We compare geometry-input and RGB-input versions. The prediction is that the geometry-input version loses less accuracy going from vKITTI2 to KITTI.
The decisive checks are three. First, gating must beat semantic gating in contact bands. Second, the geometry-only regime predictor must transfer better than the RGB version. Third, the gains must survive being added to a retrained Back2Color. If any of the three fails, the corresponding piece of the claim is withdrawn.
References
- Self-supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidance
- S3Net: Semantic-Aware Self-supervised Depth Estimation with Monocular Videos and Synthetic Data
- Back2Color: Domain-Adaptive Synthetic-to-Real Monocular Depth Estimation for Dynamic Traffic Scenes
- P2Net: Patch-Match and Plane-Regularization for Unsupervised Indoor Depth Estimation
- PlanarGS: High-Fidelity Indoor 3D Gaussian Splatting Guided by Vision-Language Planar Priors
- DynaSurfGS: Dynamic Surface Reconstruction with Planar-based Gaussian Splatting
- Occlusion Boundary and Depth: Mutual Enhancement via Multi-Task Learning
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- Virtual KITTI 2
- Monocular Instance Motion Segmentation for Autonomous Driving: KITTI InstanceMotSeg Dataset and Multi-Task Baseline