Abstract
This paper presents a new GAN-based deep learning framework for estimating absolute scale aware depth and ego motion from monocular images using a completely unsupervised mode of learning. The proposed architecture uses two separate generators to learn the distribution of depth and pose data for a given input image sequence. The depth and pose data, thus generated, are then evaluated by a patch-based discriminator using the reconstructed image and its corresponding actual image. The patch-based GAN (or PatchGAN) is shown to detect high frequency local structural defects in the reconstructed image, thereby improving the accuracy of overall depth and pose estimation. Unlike conventional GANs, the proposed architecture uses a conditioned version of input and output of the generator for training the whole network. The resulting framework is shown to outperform all existing deep networks in this field and beating the current state-of-the-art method by 8.7% in absolute error and 5.2% in RMSE metric. To the best of our knowledge, this is first deep network based model to estimate both depth and pose simultaneously using a conditional patch-based GAN paradigm. The efficacy of the proposed approach is demonstrated through rigorous ablation studies and exhaustive performance comparison on the popular KITTI outdoor driving dataset.
Introduction
Estimating 3D depth and camera movement (ego-motion) from 2D images is a foundational task for autonomous driving, augmented reality, and 3D reconstruction. While supervised deep learning models perform well, they require massive amounts of ground-truth depth data (e.g., from expensive LIDAR sensors), which is difficult to obtain in diverse real-world environments.
Unsupervised methods bypass this bottleneck by using geometric constraints between consecutive video frames or stereo image pairs to train networks via "image reconstruction"—warping one image to look like another based on predicted depth. However, these methods often produce blurry results and struggle with absolute scale. Recent attempts to use Generative Adversarial Networks (GANs) to improve these results often rely on standard discriminators that output a single "real/fake" score for the whole image. This global evaluation tends to ignore fine, high-frequency local structural defects, leaving room for significant improvement in depth map sharpness and accuracy.
Methodology
The authors propose a conditional GAN framework consisting of two generators and one shared discriminator.
Generators
The system uses two distinct networks:
- Disp-Net: A modified, lightweight network (trimmed to just 5 convolutional layers and ~8M parameters) that predicts left-to-right and right-to-left disparities from an input image.
- Pose-Net: A network that predicts the relative 6-DOF camera motion (translation and rotation) from a short sequence of frames.
View Reconstruction and PatchGAN Discriminator
Using the predicted disparity and pose, a View Reconstruction Module employs inverse-image warping to reconstruct the target image from source images. This is performed both spatially (using stereo pairs) and temporally (using consecutive video frames).
Instead of a standard discriminator, the authors use a fully convolutional PatchGAN. Rather than outputting a single scalar for the whole image, it outputs an $m \times n$ grid of probabilities, evaluating the authenticity of local patches of the reconstructed image against the real image. The network is trained entirely unsupervised, combining standard content losses (spatial/temporal appearance, disparity smoothness, left-right consistency) with the adversarial loss from the PatchGAN.
Key Insights
- Local vs. Global Discrimination: Standard GANs often produce globally plausible but locally blurry images. By switching to a PatchGAN discriminator, the network is heavily penalized for local structural defects, forcing the generators to respect high-frequency local structures.
- Extreme Efficiency: The proposed architecture is remarkably lightweight. The trimmed Disp-Net uses only about one-fourth of the parameters (8 million) of standard depth estimation networks. The entire model is around 12 million parameters.
- Scale Awareness: Utilizing stereo image pairs during the training phase allows the network to predict absolute scale-aware depth during monocular testing, even though it is trained unsupervised.
Results
The proposed method sets a new state-of-the-art for unsupervised monocular depth estimation on the KITTI dataset. On the KITTI Eigen split (50m range), it achieved an Absolute Relative Error (Abs Rel) of 0.1102 and an RMSE of 5.535, outperforming previous state-of-the-art methods by 8.7% and 5.2%, respectively.
Visual comparisons demonstrate that the PatchGAN approach yields noticeably sharper depth maps with better-defined object boundaries (such as cars, poles, and trees) compared to baseline models trained without adversarial loss. In pose estimation, the model outperformed other monocular methods like SfMLearner and achieved results comparable to methods using stereo ground truth.
Conclusion and Future Work
While the model achieves significant gains in accuracy and sharpness, it currently assumes a static world geometry and struggles with dynamic objects or occlusions. Additionally, the pose estimation network only considers short sequences, limiting its ability to handle long-term temporal dependencies like loop closure in full SLAM systems. Future research will focus on addressing moving objects and incorporating long-term dependencies.
Summary updated