Abstract
Monocular Depth Estimation (MDE) is critical for autonomous systems but faces significant challenges in low-light or high-dynamic-range environments. Standard RGB cameras often fail in these conditions, while event cameras provide high-speed, sparse contour information. This paper presents PCDepth, a novel framework that shifts from traditional pixel-level fusion to a "pattern-level" approach. By discretizing scenes into abstract visual tokens, PCDepth effectively integrates complementary information from both sensors. Experimental results on the MVSEC and DSEC benchmarks demonstrate that PCDepth achieves a 37.9% accuracy improvement in nighttime scenarios over previous state-of-the-art methods.
Introduction
Predicting 3D distance from a single 2D viewpoint is a fundamental challenge for autonomous navigation. While RGB cameras provide rich color and intensity, their performance degrades significantly in poor lighting or high-contrast scenes. Event cameras, which record brightness changes at microsecond resolution, excel in these conditions by capturing sharp object contours without the motion blur or saturation issues typical of standard sensors.
The prevailing approach to multi-modal depth estimation is pixel-level fusion. However, this is often suboptimal because RGB pixels can be noisy in the dark, and event data is inherently sparse. The true complementarity between these sensors lies in high-level structural patterns—such as the shape of a car or the edge of a building—which may only occupy a few pixels. Traditional pixel-level fusion often fails to capture these structural relationships, leading to blurred depth boundaries.
PCDepth addresses this by representing the scene as a set of learnable patterns (tokens), making the model robust to individual pixel noise and data sparsity.
Methods
The PCDepth architecture consists of two primary stages designed to maximize the synergy between RGB and event data by focusing on structural patterns rather than raw pixels.
Complementary Visual Representation Learning
Instead of maintaining data in a rigid grid, PCDepth uses transposed attention to discretize the scene into a finite set of "visual tokens." These tokens encapsulate high-level patterns. A score-based fusion module then adaptively integrates these tokens from both the image and event branches. This allows the model to dynamically "decide" which sensor provides the most reliable information for a specific pattern—for instance, prioritizing event data for sharp contours in low-light environments while relying on RGB for global intensity in daylight.
Refined Depth Estimator
To reconstruct a dense depth map from these abstract tokens, the model employs cross-attention at a single spatial resolution. This is followed by Gated Recurrent Unit (GRU) blocks that iteratively refine the depth estimates. This approach balances computational efficiency by using a token-based bottleneck (e.g., 32 or 128 tokens) while maintaining high reconstruction accuracy through iterative refinement.
Results and Discussion
PCDepth was evaluated on two major benchmarks: the Multi-Vehicle Stereo Event Camera (MVSEC) dataset and the DSEC dataset.
- Nighttime Performance: On the MVSEC nighttime dataset, PCDepth achieved a 37.9% improvement in accuracy compared to existing state-of-the-art methods like HMNet.
- Object Definition: Qualitative analysis shows that PCDepth is significantly better at distinguishing objects at varying depths, such as individual trees or distant vehicles, where pixel-level methods often produce blurred or merged estimates.
- Robustness: A version of the model trained on synthetic data (PCDepth[S]) outperformed many models trained directly on real-world data when tested on nighttime sequences, highlighting its strong generalization capabilities.
Conclusion
PCDepth demonstrates that pattern-level fusion is a superior paradigm for multi-modal monocular depth estimation. By moving beyond pixels to structural tokens, the model achieves unprecedented robustness in challenging lighting conditions. Future work will focus on improving the scene discretization process to better handle highly cluttered environments and exploring more efficient projection methods from tokens back to the pixel plane.
Summary updated