Mitigating the Synthetic to Real Gap in Depth Estimation

22 concepts22 papers1 proposal

Predicting the 3D structure of a scene from 2D images—whether through a single camera (monocular) or a stereo pair—is a foundational challenge in computer vision. To train deep learning models for this task, researchers need massive amounts of ground-truth depth data. While capturing accurate depth in the real world is expensive, logistically challenging, and plagued by sensor noise, synthetic environments offer an elegant alternative: perfect, infinite, and pixel-precise depth labels for free.

However, a model trained purely on pristine, simulated environments often fails dramatically when deployed in the real world. As tracked in a comprehensive survey by Sohel et al. in Deep Learning-based Depth Estimation Methods from Monocular Image and Videos: A Comprehensive Survey, this performance drop stems from the "synthetic-to-real domain gap." Real-world scenes introduce complex lighting, weather variations, sensor noise, and entirely novel objects that do not exist in idealized simulations.

To bridge this gap, researchers have developed strategies that move beyond simple pixel-level adjustments. They focus on forcing networks to learn fundamental geometry rather than superficial visual patterns.


The Danger of Superficial Shortcuts

Why exactly do models trained on synthetic data fail in the real world? The core issue is that deep neural networks are highly opportunistic; they naturally seek the easiest path to minimize training loss. In a synthetic dataset, a specific texture, shading pattern, or rendering artifact might consistently correlate with a certain depth. The network latches onto these superficial "shortcuts" instead of learning true geometric principles like perspective or parallax. When the model encounters real-world imagery—or even AI-generated synthetic content with its own unique noise patterns, as explored by Zhai et al. in IDOL: Unified Dual-Modal Latent Diffusion for Human-Centric Joint Video-Depth Generation—these shortcuts break down entirely.

To combat this, Chuah et al. introduced an information-theoretic approach called ITSA in ITSA: An Information-Theoretic Approach to Automatic Shortcut Avoidance and Domain Generalization in Stereo Matching Networks. By using Fisher Information to quantify how much a network relies on specific image features, the authors actively penalize the model for exploiting fragile, simulator-specific shortcuts. This forces the network to ignore superficial visual cues and focus on robust, structural features that remain consistent across both simulated and physical environments.


Aligning the Domains

If the visual discrepancy between simulation and reality is the root cause of the domain gap, one intuitive solution is to explicitly align the two domains. This alignment can happen at multiple stages of the estimation pipeline: the input images, the internal features, or the final output predictions.

Translating Styles

One approach is to use Generative Adversarial Networks (GANs) to alter the visual style of the images. For instance, in StereoGAN: Bridging Synthetic-to-Real Domain Gap by Joint Optimization of Domain Translation and Stereo Matching, Liu et al. jointly optimize an image-translation network alongside a stereo-matching network. By enforcing geometric consistency during the translation process, they ensure that changing the "style" of a synthetic image to look real does not distort its underlying 3D structure.

Conversely, instead of making synthetic images look real, models can align domains across multiple levels simultaneously. In AdaStereo: An Efficient Domain-Adaptive Stereo Matching Approach, Song et al. introduce a multi-level alignment framework that normalizes differences at the input level, aligns feature distributions internally, and refines the final disparity outputs, ensuring robust transfer without any real-world labels.

Domain Decluttering

Standard style-transfer methods often fail to address high-level semantic discrepancies, such as real-world objects (e.g., complex trash cans, unique vehicles) that never appeared in the training simulation. When a model encounters these out-of-domain objects, its spatial interpretation collapses, as observed by Somers et al. in Cystoscopic depth estimation using gated adversarial domain adaptation.

To solve this, Kong et al. proposed a counter-intuitive but highly effective strategy called ARC (Attend, Remove, Complete) in Domain Decluttering: Simplifying Images to Mitigate Synthetic-Real Domain Shift and Improve Depth Estimation. Instead of trying to make synthetic data more complex, they simplify real-world images. The framework identifies complex, out-of-domain objects, removes them, and inpaints the missing regions to match the cleaner, more predictable structure of synthetic scenes.

[Real-World Image] ---> [Identify Complex Objects] ---> [Remove & Inpaint] ---> [Simplified Image] ---> [Depth Model]

Stripping Style to Uncover Pure Geometry

An alternative to translating images is to build networks that are inherently blind to visual style, forcing them to rely strictly on geometric invariants.

Training on Non-Realistic Data

In S2R-DepthNet: Learning a Generalizable Depth-Specific Structural Representation, Chen et al. trained a monocular depth model on a combination of synthetic data and a diverse collection of artistic paintings. Because paintings feature wild variations in texture, color, and brushstroke style while still maintaining basic spatial layout, the network learns that surface appearance is completely unreliable for predicting depth. As a result, it suppresses style features entirely and isolates a domain-invariant structural representation.

Similarly, Liu et al. developed GraftNet in GraftNet: Towards Domain Generalized Stereo Matching with a Broad-Spectrum and Task-Oriented Feature. They "graft" robust, broad-spectrum features from massive models pre-trained on ImageNet into the stereo network, providing a visual foundation that is highly resilient to the idiosyncratic style artifacts of simulated data.

Bypassing Pixels Entirely

For tasks like depth completion—where the goal is to fill in the blanks of a sparse depth map generated by a LiDAR sensor—the visual domain gap can be bypassed entirely. In Learning Topology from Synthetic Data for Unsupervised Depth Completion, Wong et al. developed a framework that learns scene topology solely from sparse point clouds, ignoring the RGB images altogether. By stripping away textures, lighting, and material properties, the model trains exclusively on pure geometric relationships, allowing it to generalize seamlessly to the real world.


Architectural Robustness and Uncertainty Calibration

Even with advanced adaptation techniques, some domain shift is inevitable. Robust architectures must be engineered to handle environmental variations gracefully, and they must know when to communicate uncertainty.

Domain-Invariant Layers

To mitigate variations in lighting, weather, and sensor characteristics, Zhang et al. designed DSMNet in Domain-Invariant Stereo Matching Networks. Instead of relying on standard normalization layers that can be sensitive to domain-specific styles, the authors introduce specialized normalization layers that filter out style variations directly within the network's feature extractors, ensuring robust zero-shot transfer.

Calibrating What the Model Doesn't Know

When a model encounters an environment that differs significantly from its training data, maintaining reliable depth accuracy is difficult. In these scenarios, knowing where the model is likely making mistakes is critical for downstream tasks like robotic navigation.

In Bayesian Deep Neural Networks for Supervised Learning of Single-View Depth, Javier Rodríguez-Puigvert demonstrates that placing Monte Carlo (MC) Dropout specifically within the encoder of a single-view depth network yields significantly better depth accuracy and highly calibrated uncertainty estimates. This allows autonomous systems to identify and filter out unreliable, high-uncertainty 3D points caused by the domain gap.

[Real Image (Domain Shift)] ---> [Encoder + MC Dropout] ---> [Calibrated Uncertainty Map] ---> [Filter Unreliable Points]

This focus on filtering out errors is mirrored in classical stereo analysis. Haeusler et al., in Analysis of KITTI Data for Stereo Analysis with Stereo Confidence Measures, showed that while individual confidence metrics struggle in complex outdoor environments, combining multiple confidence measures multiplicatively provides a robust mechanism for isolating and filtering out incorrect depth estimates.

Finally, to handle variations in sensor hardware itself, Tosi et al. developed a continuous-coordinate neural refiner in Neural Disparity Refinement for Arbitrary Resolution Stereo. By decoupling the refinement process from fixed pixel grids, their system achieves zero-shot generalization across arbitrary resolutions, bridging the gap between different physical sensor specifications.


Conclusion

The synthetic-to-real domain gap is not merely a difference in pixel values; it is a fundamental mismatch in how networks interpret visual context. By preventing models from taking superficial shortcuts (Chuah et al.), explicitly translating or simplifying scene semantics (Kong et al.), and forcing internal representations to prioritize pure geometry over surface style (Chen et al.), modern depth estimation frameworks can successfully leverage the power of simulation to achieve robust, zero-shot generalization in the real world.

Key Foundational Work

Domain Decluttering: Simplifying Images to Mitigate Synthetic-Real Domain Shift and Improve Depth Estimation (2020) — contributed 1 concept elements to this direction.
StereoGAN: Bridging Synthetic-to-Real Domain Gap by Joint Optimization of Domain Translation and Stereo Matching (2020) — contributed 1 concept elements to this direction.
Domain-Invariant Stereo Matching Networks (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_synthetic_to_real_domain_gap_synthetic_t_81168062,
  title = {synthetic-to-real domain gap (synthetic-to-real domain shift)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/synthetic-to-real-domain-gap-synthetic-to-real-domain-shift/rdr_81168062ecd14132942a7d7713c47641},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_synthetic_to_real_domain_gap_synthetic_t_81168062_v1,
  title = {synthetic-to-real domain gap (synthetic-to-real domain shift)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/synthetic-to-real-domain-gap-synthetic-to-real-domain-shift/rdr_81168062ecd14132942a7d7713c47641/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake