Abstract
SyntStereo2Real is a lightweight, edge-aware Generative Adversarial Network (GAN) designed to bridge the domain gap between synthetic and real-world remote sensing imagery. By explicitly incorporating edge maps and a geometric warping loss, the model ensures that translated stereo pairs remain structurally consistent. This approach allows for more accurate 3D reconstruction and disparity estimation using only synthetic training data, overcoming the limitations of standard image-to-image translation methods that often compromise geometric integrity.
Introduction
In remote sensing, training deep neural networks for tasks like 3D reconstruction requires massive amounts of ground truth data—specifically, perfectly aligned stereo pairs with accurate disparity maps. Obtaining this data in the real world is challenging due to variable weather conditions, moving objects, and the high cost of LiDAR equipment.
Synthetic data from simulators provides a viable alternative with perfect labels; however, a significant "domain gap" exists. Models trained on clean synthetic images often fail when applied to complex real-world imagery. While Image-to-Image (I2I) translation tools like CycleGAN can improve realism, they frequently "hallucinate" textures or shift pixels. In stereo pairs, even minor pixel shifts break the geometric relationship (the stereo constraint), rendering the data useless for training depth-sensing models.
Methodology
The authors propose a GAN-based framework that translates images while strictly maintaining their geometric structure through several key innovations:
- Edge-Aware Encoding: The model utilizes a Sobel operator to generate edge maps of the input imagery. These maps are fed into the encoder alongside the RGB image, forcing the generator to respect the boundaries of buildings and roads and preventing the structural distortion common in standard GANs.
- Disentangled Representation: Following the MUNIT architecture, the model separates "content" (scene geometry) from "style" (lighting, texture, and sensor noise), allowing for more precise control over the translation process.
- Warping Loss: To enforce the stereo constraint, the authors leverage ground-truth disparity maps from synthetic data to mathematically "warp" the translated left image. The model is penalized if the warped left image does not match the translated right image, ensuring geometric consistency across views.
- Lightweight Architecture: SyntStereo2Real performs these tasks in a single, efficient framework, requiring significantly fewer parameters than previous state-of-the-art models like StereoGAN.
Experimental Results
The model was evaluated on two distinct domains: remote sensing (SyntCities to US3D) and autonomous driving (Driving to KITTI 2015).
Quantitative Performance
On the SyntCities-to-US3D task, SyntStereo2Real achieved a 3px-accuracy of 69.9%, outperforming StereoGAN (66.7%) and the non-translated baseline (63.1%). The Median Absolute Deviation (MAD) also saw a notable reduction to 1.319.
Efficiency and Quality
SyntStereo2Real achieved these results with only 11 million parameters, a nearly 5x reduction compared to the 54 million parameters required by StereoGAN. Qualitatively, the model successfully reproduces architectural details such as rooftops and bridges while handling shadows more effectively, avoiding the hallucinated vegetation common in other methods.
Conclusion
SyntStereo2Real demonstrates that making a GAN "edge-aware" and using geometry as a regularizer can significantly reduce hallucinations and maintain structural integrity in I2I translation. This lightweight approach provides a robust pathway for using synthetic data to train real-world remote sensing models. Future work may extend this framework to multi-view networks and address challenges posed by extreme occlusions in large-baseline remote sensing tasks.
Summary updated