TL;DR
This paper introduces PixelFormer, a novel approach to monocular depth estimation that leverages a Skip Attention Module (SAM) to effectively fuse coarse global context from the decoder with fine-grained local features from the encoder. By reframing depth prediction as a pixel query refinement problem and introducing an adaptive bin center predictor, PixelFormer achieves state-of-the-art results on several benchmark datasets.
What Problem Does This Paper Address?
Monocular Depth Estimation (MDE) aims to predict the depth of each pixel in an image using only a single RGB image. This is a challenging task because it requires both understanding the overall scene context (global information) and preserving fine-grained details (local resolution) to accurately estimate depth, especially at object boundaries. Traditional encoder-decoder architectures with simple skip connections (like concatenation followed by convolution) struggle to effectively bridge the semantic gap between encoder and decoder features, limiting the fusion of global and local information. While attention mechanisms have shown promise in computer vision, applying them to high-resolution feature maps is computationally expensive. Existing attention-based methods often resort to windowed attention or other approximations to manage this cost.
Methods
PixelFormer tackles MDE by treating it as a pixel query refinement problem. The core innovations are:
- Skip Attention Module (SAM): This module replaces traditional skip connections. Instead of simple concatenation, SAM uses a window-based cross-attention mechanism. It allows pixel queries from the decoder to attend to encoder features within a defined window, effectively aggregating long-range contextual information from the encoder while maintaining efficiency. This helps to fuse semantically rich decoder features with high-resolution encoder features.
- Pixel Query Initializer (PQI): At the coarsest encoder resolution, PQI aggregates global scene information using multi-scale pooling to initialize the first set of pixel queries.
- Bin Center Predictor (BCP): This lightweight module predicts adaptive bin centers for depth discretization. Unlike previous methods that might use a separate transformer for this, BCP directly uses the initial pixel queries from PQI, making it more efficient and allowing for direct ground truth supervision to embed depth information early in the process.
- Overall Architecture: PixelFormer uses a Swin Transformer backbone as the encoder. The initial pixel queries from PQI are refined through multiple stages by SAM modules, which query the corresponding encoder features at different resolutions. Finally, a convolution and softmax operation predict the probability distribution over the adaptive bin centers.
Datasets
The paper evaluates PixelFormer on the following standard datasets:
- NYU Depth V2: An indoor dataset with RGB-D images from 464 scenes.
- KITTI: An outdoor dataset containing stereo images and LiDAR scans from 61 scenes.
- SUNRGB-D: Another indoor dataset collected using various sensors.
The authors use standard training/testing splits for NYUV2 and KITTI. For SUNRGB-D, they evaluate the model's generalization performance without fine-tuning on that specific dataset.
Key Insights and Ideas
- Attention for Fusion, Not Just Representation: The paper demonstrates that attention can be a powerful tool not just for learning feature representations but also for effectively fusing features from different network stages, specifically addressing the encoder-decoder semantic gap.
- Efficiency Through Windowed Cross-Attention: By using window-based cross-attention in SAM, the authors achieve the benefits of attention (long-range dependencies) without the prohibitive computational cost of full self-attention on high-resolution maps.
- Early Depth Supervision for Better Queries: Embedding depth information directly into the initial pixel queries via the BCP module and ground truth supervision leads to improved depth estimation, suggesting that guiding the initial representations is crucial.
- Pixel Query Refinement as a Paradigm: Reframing MDE as a process of refining pixel queries, starting from coarse global information and progressively adding finer details, provides a flexible and effective framework.
Key Results
PixelFormer achieves state-of-the-art performance on multiple benchmarks:
- NYUV2: Improves Abs Rel by 5.3% and RMSE by 9.6% over recent methods.
- KITTI: Improves Sq Rel by 3.9% and RMSE by 2.3% over SOTA.
- SUNRGB-D (Generalization): Shows a 9.4% improvement in Abs Rel over SOTA, demonstrating strong out-of-distribution performance.
- Qualitative results show PixelFormer producing sharper depth boundaries and more accurate depth labels, particularly in challenging scenarios where other methods exhibit confusion.
Limitations and Future Work
- Computational Cost: While SAM is more efficient than full self-attention, it still involves attention mechanisms, which can be more computationally intensive than pure convolutional approaches.
- Future Work: The authors suggest applying their content-adaptive fusion strategy (SAM) to other dense estimation tasks like semantic segmentation. They also mention that further exploration of the bin center prediction mechanism could yield additional improvements.
Summary updated