LiftFormer: Lifting and Frame Theory Based Monocular Depth Estimation Using Depth and Edge Oriented Subspace Representation

Shuai Li, Huibin Bai, Yanbo Gao, Chong Lv, Hui Yuan, Chuankun Li, Wei Hua, Tian Xie

TL;DR

This paper introduces LiftFormer, a novel framework for monocular depth estimation that leverages lifting and frame theory to create intermediate "subspaces" that better bridge image color information with geometric depth. By transforming depth prediction into a more robust representation problem, LiftFormer achieves state-of-the-art results, particularly in accurately capturing sharp depth changes around object edges.

What Problem Does This Paper Address?

Estimating the 3D structure of a scene from a single 2D image (monocular depth estimation or MDE) is a fundamentally ill-posed problem. Traditional methods struggle with the inherent ambiguity and the need for fine-grained, pixel-level accuracy. While deep learning has significantly advanced MDE, current approaches often face challenges:

  • Bridging Modalities: Directly mapping image color features to geometric depth values is difficult.
  • Depth Bin Mismatch: Classification-regression methods, which discretize depth into bins, can suffer from a misalignment between learned depth features and the bin centers, leading to suboptimal predictions.
  • Edge Accuracy: Object edges, characterized by sharp depth changes and high-frequency information, are notoriously difficult to predict accurately, often resulting in blurred or inaccurate boundaries in depth maps.
  • Transformer Sensitivity: While powerful for global feature extraction, transformers can be less sensitive to local inductive biases, potentially exacerbating edge prediction issues.

Methods

LiftFormer tackles these challenges by introducing two novel "lifting" modules based on lifting and frame theory:

  • Depth-Oriented Geometric Representation (DGR) Subspace (SF-DGR Lifting): This module transforms image spatial features into a DGR subspace. This subspace is designed to represent depth-level-related features that correspond to depth bin centers. Instead of directly predicting depth, the network learns to map image features into this intermediate space, which is theoretically shown to be a more robust representation for depth prediction. Frame theory is used to construct an overcomplete frame-based DGR subspace with linearly dependent vectors, allowing for a sparser and more robust representation of depth features.
  • Edge-Aware Representation (ER) Subspace (DF-ER Lifting): To specifically address the issue of inaccurate edge prediction, a second lifting module transforms depth features into an ER subspace. This subspace is constructed to capture local structure information and enhance edge awareness. By explicitly modeling edge features, this module helps to preserve sharp depth changes and improve the accuracy of depth prediction around object boundaries.

The overall architecture is a U-Net with a Swin Transformer encoder. The SF-DGR lifting is applied at multiple scales, transforming encoder features to the decoder. The DF-ER lifting then enhances these depth features with local edge information. The final depth map is predicted using an AdaBin-style approach.

Datasets

The paper evaluates LiftFormer on two standard datasets for monocular depth estimation:

  • KITTI Dataset: This dataset is widely used for depth estimation, featuring real-world urban scenes. The Eigen split is used, with depth values ranging up to 80 meters.
  • NYU Depth V2 Dataset: This is an indoor dataset with RGB-D images. The evaluation uses a subset with depth values up to 10 meters.

Key Insights and Ideas

  • Lifting Theory for MDE: The paper theoretically validates the use of intermediate, lifted representations (subspaces) for improving MDE, moving beyond direct end-to-end mapping.
  • Frame Theory for Robustness: Utilizing frame theory to construct the DGR subspace with linearly dependent vectors provides a more redundant and robust representation compared to traditional orthogonal bases, especially for capturing continuous depth variations.
  • Decoupling Appearance from Geometry: The DGR subspace helps to isolate depth-unrelated appearance information from image spatial features, making the transformation to depth features more effective.
  • Edge Enhancement as a Separate Module: Treating edge enhancement as a distinct lifting module (DF-ER) allows for targeted improvement of a critical aspect of depth estimation that is often overlooked.
  • Generalization of Lifting Modules: The ablation studies suggest that the SF-DGR and DF-ER lifting modules can be used in a plug-and-play manner and can generalize to different decoders, indicating their broad applicability.

Key Results

LiftFormer achieves state-of-the-art performance on both the KITTI and NYU Depth V2 datasets.

  • KITTI: LiftFormer achieves an RMSE of 2.038, outperforming existing methods like PixelFormer (2.081), BinsFormer (2.098), and iDisc (2.067). Qualitative results show smoother and clearer depth maps, especially around object boundaries.
  • NYU Depth V2: LiftFormer achieves an RMSE of 0.313, surpassing methods like PixelFormer (0.322) and DepthFormer (0.339). Qualitative comparisons highlight smoother and less noisy depth predictions with clearer local structures.
  • Error Reduction: Error maps demonstrate that LiftFormer significantly reduces errors, particularly around object edges, validating the effectiveness of the DF-ER module.
  • Ablation Studies: The paper shows that both SF-DGR and DF-ER lifting modules individually contribute to performance improvements. The study also investigates the impact of redundancy in the DGR subspace, finding an optimal balance.

Limitations and Future Work

  • Redundancy in DGR: While redundancy in the DGR subspace is beneficial, excessive redundancy (e.g., 256 basis vectors) can complicate the network and lead to marginal performance degradation. The paper settles on 128 basis vectors.
  • Future Work: The paper suggests that the proposed lifting modules can be generalized to various decoders, implying potential for integration with other advanced architectures. Further exploration of different configurations and applications of these lifting techniques could be a future direction. The paper also mentions that the ER subspace is learned without direct supervision, hinting at potential for incorporating explicit edge supervision in future work.

Summary updated

Abstract

Monocular depth estimation (MDE) has attracted increasing interest in the past few years, owing to its important role in 3D vision. MDE is the estimation of a depth map from a monocular image/video to represent the 3D structure of a scene, which is a highly ill-posed problem. To solve this problem, in this paper, we propose a LiftFormer based on lifting theory topology, for constructing an intermediate subspace that bridges the image color features and depth values, and a subspace that enhances the depth prediction around edges. MDE is formulated by transforming the depth value prediction problem into depth-oriented geometric representation (DGR) subspace feature representation, thus bridging the learning from color values to geometric depth values. A DGR subspace is constructed based on frame theory by using linearly dependent vectors in accordance with depth bins to provide a redundant and robust representation. The image spatial features are transformed into the DGR subspace, where these features correspond directly to the depth values. Moreover, considering that edges usually present sharp changes in a depth map and tend to be erroneously predicted, an edge-aware representation (ER) subspace is constructed, where depth features are transformed and further used to enhance the local features around edges. The experimental results demonstrate that our LiftFormer achieves state-of-the-art performance on widely used datasets, and an ablation study validates the effectiveness of both proposed lifting modules in our LiftFormer.
BibTeX
@article{li2025liftformer,
  author = {Shuai Li and Huibin Bai and Yanbo Gao and Chong Lv and Hui Yuan and Chuankun Li and Wei Hua and Tian Xie},
  title = {{LiftFormer: Lifting and Frame Theory Based Monocular Depth Estimation Using Depth and Edge Oriented Subspace Representation}},
  journal = {IEEE Trans. Multimedia},
  year = {2025},
  url = {https://openreview.net/forum?id=ahK7XyJ36n},
}
Figure