TL;DR
This paper introduces LiftFormer, a novel framework for monocular depth estimation that leverages lifting and frame theory to create intermediate "subspaces" that better bridge image color information with geometric depth. By transforming depth prediction into a more robust representation problem, LiftFormer achieves state-of-the-art results, particularly in accurately capturing sharp depth changes around object edges.
What Problem Does This Paper Address?
Estimating the 3D structure of a scene from a single 2D image (monocular depth estimation or MDE) is a fundamentally ill-posed problem. Traditional methods struggle with the inherent ambiguity and the need for fine-grained, pixel-level accuracy. While deep learning has significantly advanced MDE, current approaches often face challenges:
- Bridging Modalities: Directly mapping image color features to geometric depth values is difficult.
- Depth Bin Mismatch: Classification-regression methods, which discretize depth into bins, can suffer from a misalignment between learned depth features and the bin centers, leading to suboptimal predictions.
- Edge Accuracy: Object edges, characterized by sharp depth changes and high-frequency information, are notoriously difficult to predict accurately, often resulting in blurred or inaccurate boundaries in depth maps.
- Transformer Sensitivity: While powerful for global feature extraction, transformers can be less sensitive to local inductive biases, potentially exacerbating edge prediction issues.
Methods
LiftFormer tackles these challenges by introducing two novel "lifting" modules based on lifting and frame theory:
- Depth-Oriented Geometric Representation (DGR) Subspace (SF-DGR Lifting): This module transforms image spatial features into a DGR subspace. This subspace is designed to represent depth-level-related features that correspond to depth bin centers. Instead of directly predicting depth, the network learns to map image features into this intermediate space, which is theoretically shown to be a more robust representation for depth prediction. Frame theory is used to construct an overcomplete frame-based DGR subspace with linearly dependent vectors, allowing for a sparser and more robust representation of depth features.
- Edge-Aware Representation (ER) Subspace (DF-ER Lifting): To specifically address the issue of inaccurate edge prediction, a second lifting module transforms depth features into an ER subspace. This subspace is constructed to capture local structure information and enhance edge awareness. By explicitly modeling edge features, this module helps to preserve sharp depth changes and improve the accuracy of depth prediction around object boundaries.
The overall architecture is a U-Net with a Swin Transformer encoder. The SF-DGR lifting is applied at multiple scales, transforming encoder features to the decoder. The DF-ER lifting then enhances these depth features with local edge information. The final depth map is predicted using an AdaBin-style approach.
Datasets
The paper evaluates LiftFormer on two standard datasets for monocular depth estimation:
- KITTI Dataset: This dataset is widely used for depth estimation, featuring real-world urban scenes. The Eigen split is used, with depth values ranging up to 80 meters.
- NYU Depth V2 Dataset: This is an indoor dataset with RGB-D images. The evaluation uses a subset with depth values up to 10 meters.
Key Insights and Ideas
- Lifting Theory for MDE: The paper theoretically validates the use of intermediate, lifted representations (subspaces) for improving MDE, moving beyond direct end-to-end mapping.
- Frame Theory for Robustness: Utilizing frame theory to construct the DGR subspace with linearly dependent vectors provides a more redundant and robust representation compared to traditional orthogonal bases, especially for capturing continuous depth variations.
- Decoupling Appearance from Geometry: The DGR subspace helps to isolate depth-unrelated appearance information from image spatial features, making the transformation to depth features more effective.
- Edge Enhancement as a Separate Module: Treating edge enhancement as a distinct lifting module (DF-ER) allows for targeted improvement of a critical aspect of depth estimation that is often overlooked.
- Generalization of Lifting Modules: The ablation studies suggest that the SF-DGR and DF-ER lifting modules can be used in a plug-and-play manner and can generalize to different decoders, indicating their broad applicability.
Key Results
LiftFormer achieves state-of-the-art performance on both the KITTI and NYU Depth V2 datasets.
- KITTI: LiftFormer achieves an RMSE of 2.038, outperforming existing methods like PixelFormer (2.081), BinsFormer (2.098), and iDisc (2.067). Qualitative results show smoother and clearer depth maps, especially around object boundaries.
- NYU Depth V2: LiftFormer achieves an RMSE of 0.313, surpassing methods like PixelFormer (0.322) and DepthFormer (0.339). Qualitative comparisons highlight smoother and less noisy depth predictions with clearer local structures.
- Error Reduction: Error maps demonstrate that LiftFormer significantly reduces errors, particularly around object edges, validating the effectiveness of the DF-ER module.
- Ablation Studies: The paper shows that both SF-DGR and DF-ER lifting modules individually contribute to performance improvements. The study also investigates the impact of redundancy in the DGR subspace, finding an optimal balance.
Limitations and Future Work
- Redundancy in DGR: While redundancy in the DGR subspace is beneficial, excessive redundancy (e.g., 256 basis vectors) can complicate the network and lead to marginal performance degradation. The paper settles on 128 basis vectors.
- Future Work: The paper suggests that the proposed lifting modules can be generalized to various decoders, implying potential for integration with other advanced architectures. Further exploration of different configurations and applications of these lifting techniques could be a future direction. The paper also mentions that the ER subspace is learned without direct supervision, hinting at potential for incorporating explicit edge supervision in future work.
Summary updated