PCDepth: Pattern-based Complementary Learning for Monocular Depth Estimation by Best of Both Worlds

Haotian Liu, Sanqing Qu, Fan Lu, Zongtao Bu, Florian Röhrbein, Alois Knoll, Guang Chen

Abstract

Monocular Depth Estimation (MDE) is critical for autonomous systems but faces significant challenges in low-light or high-dynamic-range environments. Standard RGB cameras often fail in these conditions, while event cameras provide high-speed, sparse contour information. This paper presents PCDepth, a novel framework that shifts from traditional pixel-level fusion to a "pattern-level" approach. By discretizing scenes into abstract visual tokens, PCDepth effectively integrates complementary information from both sensors. Experimental results on the MVSEC and DSEC benchmarks demonstrate that PCDepth achieves a 37.9% accuracy improvement in nighttime scenarios over previous state-of-the-art methods.

Introduction

Predicting 3D distance from a single 2D viewpoint is a fundamental challenge for autonomous navigation. While RGB cameras provide rich color and intensity, their performance degrades significantly in poor lighting or high-contrast scenes. Event cameras, which record brightness changes at microsecond resolution, excel in these conditions by capturing sharp object contours without the motion blur or saturation issues typical of standard sensors.

The prevailing approach to multi-modal depth estimation is pixel-level fusion. However, this is often suboptimal because RGB pixels can be noisy in the dark, and event data is inherently sparse. The true complementarity between these sensors lies in high-level structural patterns—such as the shape of a car or the edge of a building—which may only occupy a few pixels. Traditional pixel-level fusion often fails to capture these structural relationships, leading to blurred depth boundaries.

Conceptual comparison between traditional pixel-level fusion (a) and the proposed pattern-level fusion.

PCDepth addresses this by representing the scene as a set of learnable patterns (tokens), making the model robust to individual pixel noise and data sparsity.

Methods

The PCDepth architecture consists of two primary stages designed to maximize the synergy between RGB and event data by focusing on structural patterns rather than raw pixels.

Complementary Visual Representation Learning

Instead of maintaining data in a rigid grid, PCDepth uses transposed attention to discretize the scene into a finite set of "visual tokens." These tokens encapsulate high-level patterns. A score-based fusion module then adaptively integrates these tokens from both the image and event branches. This allows the model to dynamically "decide" which sensor provides the most reliable information for a specific pattern—for instance, prioritizing event data for sharp contours in low-light environments while relying on RGB for global intensity in daylight.

Refined Depth Estimator

To reconstruct a dense depth map from these abstract tokens, the model employs cross-attention at a single spatial resolution. This is followed by Gated Recurrent Unit (GRU) blocks that iteratively refine the depth estimates. This approach balances computational efficiency by using a token-based bottleneck (e.g., 32 or 128 tokens) while maintaining high reconstruction accuracy through iterative refinement.

The complete PCDepth architecture, illustrating the encoder-decoder structure, pattern-level fusion, and the iterative refinement process.

Results and Discussion

PCDepth was evaluated on two major benchmarks: the Multi-Vehicle Stereo Event Camera (MVSEC) dataset and the DSEC dataset.

  • Nighttime Performance: On the MVSEC nighttime dataset, PCDepth achieved a 37.9% improvement in accuracy compared to existing state-of-the-art methods like HMNet.
  • Object Definition: Qualitative analysis shows that PCDepth is significantly better at distinguishing objects at varying depths, such as individual trees or distant vehicles, where pixel-level methods often produce blurred or merged estimates.
  • Robustness: A version of the model trained on synthetic data (PCDepth[S]) outperformed many models trained directly on real-world data when tested on nighttime sequences, highlighting its strong generalization capabilities.

Conclusion

PCDepth demonstrates that pattern-level fusion is a superior paradigm for multi-modal monocular depth estimation. By moving beyond pixels to structural tokens, the model achieves unprecedented robustness in challenging lighting conditions. Future work will focus on improving the scene discretization process to better handle highly cluttered environments and exploring more efficient projection methods from tokens back to the pixel plane.

Summary updated

BibTeX
@inproceedings{liu2024pcdepth,
  author = {Haotian Liu and Sanqing Qu and Fan Lu and Zongtao Bu and Florian Röhrbein and Alois Knoll and Guang Chen},
  title = {{PCDepth: Pattern-based Complementary Learning for Monocular Depth Estimation by Best of Both Worlds}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=YUOt1IQKeo},
}

Loading related content...

Figure