Patch-Wise Attention Network for Monocular Depth Estimation

Sihaeng Lee, Janghyeon Lee, Byungju Kim, Eojindl Yi, Junmo Kim

Abstract

Monocular depth estimation is a challenging task due to its inherently ill-posed nature. While modern convolutional neural networks (CNNs) effectively capture global scene structures, they often struggle with local pixel relationships, leading to blurry object boundaries. This paper introduces the Patch-Wise Attention (PWA) network, which applies dedicated attention modules to localized image patches. By focusing on regional geometric nuances, PWA achieves sharper boundaries and state-of-the-art performance on the KITTI and NYU Depth V2 benchmarks.

Introduction

Predicting 3D distance from a single 2D image is mathematically ill-posed because a single 2D projection can correspond to infinite 3D configurations. Consequently, AI models must rely heavily on contextual cues and learned patterns.

Current CNN-based methods are proficient at understanding global scene layout but frequently fail to model local pixel relationships accurately. Intuitively, neighboring pixels on the same object should share similar depth values, while pixels at an edge should exhibit sharp transitions. Global attention mechanisms often overlook these critical local details, resulting in "depth bleeding" between foreground and background objects.

Proposed Method: Patch-Wise Attention

The authors propose the Patch-Wise Attention (PWA) module, a plug-and-play component designed for standard encoder-decoder architectures. The PWA module processes feature maps in four distinct stages:

  1. Patch Extraction: Decoder feature maps are divided into localized patches (e.g., $s_p = 64$).
  2. Patch-Wise Channel Attention: Each patch utilizes a dedicated Multi-Layer Perceptron (MLP) to determine the importance of specific feature channels relative to its local context.
  3. Patch-Wise Spatial Attention: A convolutional layer learns pixel-wise relevance within each patch, enhancing the model's ability to resolve local shapes and edges.
  4. Re-assembly: Refined patches are re-stitched and integrated with global features via skip connections.
Detailed architecture of the Patch-Wise Attention (PWA) module, showing channel and spatial refinement stages.

By treating each patch as a semi-independent sub-problem, the network avoids the distractions of global context when refining local geometric details like table corners or vehicle silhouettes.

Experimental Results

The PWA network was evaluated on two industry-standard datasets: KITTI (outdoor driving) and NYU Depth V2 (complex indoor scenes).

Quantitative Performance

At the time of its release, the PWA model ranked #1 on the official KITTI online leaderboard, achieving a Scale-Invariant Logarithmic error (SILog) of 11.45. On the NYU Depth V2 dataset, the model reduced Absolute Relative Error (AbsRel) and Root Mean Square Error (RMSE) by approximately 4% compared to previous state-of-the-art methods using identical backbones.

Qualitative Analysis

Visual comparisons demonstrate that PWA produces significantly "cleaner" depth maps. While baseline models often generate blurry "blobs" for distant or thin objects, PWA maintains distinct silhouettes for cars, poles, and furniture.

Qualitative comparison on the KITTI dataset. PWA produces sharper object boundaries and more accurate depth maps compared to BTS and DORN.

Key Insights

  • Locality is Key: Depth is a highly local property. A pixel's depth is most strongly correlated with its immediate neighbors, making local attention more effective for boundary preservation than global attention.
  • Position-Specific Learning: By assigning unique attention parameters to different patches, the model learns that patches in different image regions (e.g., road vs. sky) require different feature processing.
  • Noise Robustness: The PWA module acts as an intelligent filter, leveraging local context to smooth out noise in LiDAR training data while preserving sharp geometric transitions.

Limitations and Future Work

The primary hyperparameter introduced is the patch size ($s_p$). The authors identified $s_p = 64$ as optimal; smaller patches lose necessary context, while larger patches suffer from the same blurring issues as global attention. Future research may explore more sophisticated local relationship layers to handle challenging scenarios like transparent surfaces or high reflectivity.

Summary updated

BibTeX
@inproceedings{lee2021patch,
  author = {Sihaeng Lee and Janghyeon Lee and Byungju Kim and Eojindl Yi and Junmo Kim},
  title = {{Patch-Wise Attention Network for Monocular Depth Estimation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i3.16282},
}
Figure