UniCT Depth: Event-Image Fusion Based Monocular Depth Estimation with Convolution-Compensated ViT Dual SA Block

Luoxi Jing, Dianxi Shi, Zhe Liu, Songchang Jin, Chunping Qiu, Ziteng Qiao, Yuxian Li, Jianqiang Xia

Abstract

Depth estimation plays a crucial role in 3D scene understanding and is extensively used in a wide range of vision tasks. Image-based methods struggle in challenging scenarios, while event cameras offer high dynamic range and temporal resolution but face difficulties with sparse data. Combining event and image data provides significant advantages, yet effective integration remains challenging. Existing CNN-based fusion methods struggle with occlusions and depth disparities due to limited receptive fields, while Transformer-based fusion methods often lack deep modality interaction. To address these issues, we propose UniCT Depth, an event-image fusion method that unifies CNNs and Transformers to model local and global features. We propose the Convolution-compensated ViT Dual SA (CcViT-DA) Block, designed for the encoder, which integrates Context Modeling Self-Attention (CMSA) to capture spatial dependencies and Modal Fusion Self-Attention (MFSA) for effective cross-modal fusion. Furthermore, we design the tailored Detail Compensation Convolution (DCC) Block to improve texture details and enhances edge representations. Experiments show that UniCT Depth outperforms existing image, event, and fusion-based monocular depth estimation methods across key metrics.

Introduction

Monocular depth estimation—predicting 3D distance from a single 2D viewpoint—is a cornerstone of autonomous driving and robotics. However, traditional image-based methods face significant challenges in extreme lighting conditions, such as night driving or high-contrast tunnel exits, and under complex occlusions where limited receptive fields hinder spatial understanding.

Event cameras offer a solution by recording pixel-level brightness changes with microsecond latency and massive dynamic range. The challenge lies in effective fusion: combining dense, texture-rich images with sparse, asynchronous event data. UniCT Depth addresses this by introducing a hybrid architecture that leverages the global modeling capabilities of Transformers alongside the local detail preservation of CNNs.

Proposed Method

UniCT Depth utilizes a U-Net-style architecture featuring a specialized encoder designed for deep modality interaction from the early stages of feature extraction.

The CcViT-DA Block

The core innovation is the Convolution-compensated ViT Dual SA Block. It splits the attention mechanism into two parallel branches to handle the multidimensional nature of sensor fusion:

  • Context Modeling Self-Attention (CMSA): This branch focuses on spatial dependencies using window-based attention. It helps the model understand local depth variations and handle occlusions by analyzing surrounding context.
  • Modal Fusion Self-Attention (MFSA): This branch operates on the channel dimension to adaptively weight features from the image and event streams. It dynamically "decides" which sensor is more reliable for a given part of the scene, such as prioritizing event data in low-light regions.
The UniCT Depth architecture featuring the CcViT-DA block, which integrates spatial and modal attention branches alongside detail compensation.

Detail Compensation Convolution (DCC)

While Transformers excel at global context, they can lose fine-grained edge information. The DCC Block acts as a corrective measure, using specialized convolutional layers to recover high-frequency texture details and sharpen object boundaries that might be blurred during the attention process.

Experimental Results

The model was evaluated on the MVSEC (real-world driving) and DENSE (simulated) benchmarks, outperforming state-of-the-art (SOTA) methods like SRFNet and RAM Net.

Performance in Challenging Scenarios

UniCT Depth demonstrated a significant performance leap in night scenes. When intensity images become noisy or underexposed, the MFSA branch successfully pivots to rely on high-dynamic-range event data. On the MVSEC dataset, the model improved the Absolute Relative Error (Abs Rel) by 5.56% in daytime and 7.16% at night compared to previous best fusion methods.

Qualitative comparison on MVSEC night sequences. UniCT Depth (f) provides sharper edges and better foreground-background separation than DPT, E2Depth, and RAM Net.

Real-Time Efficiency

Despite the dual-branch attention complexity, the model maintains a real-time processing speed of approximately 25 FPS on an NVIDIA RTX 3090, balancing high accuracy with practical deployment requirements.

Conclusion

UniCT Depth effectively bridges the gap between CNNs and Transformers for event-image fusion. By balancing global context with local detail compensation, it sets a new standard for monocular depth estimation in adverse conditions. Future work will focus on self-supervised learning to reduce dependency on ground-truth labels and further optimization for low-power edge devices.

Summary updated

Abstract

Depth estimation plays a crucial role in 3D scene understanding and is extensively used in a wide range of vision tasks. Image-based methods struggle in challenging scenarios, while event cameras offer high dynamic range and temporal resolution but face difficulties with sparse data. Combining event and image data provides significant advantages, yet effective integration remains challenging. Existing CNN-based fusion methods struggle with occlusions and depth disparities due to limited receptive fields, while Transformer-based fusion methods often lack deep modality interaction. To address these issues, we propose UniCT Depth, an event-image fusion method that unifies CNNs and Transformers to model local and global features. We propose the Convolution-compensated ViT Dual SA (CcViT-DA) Block, designed for the encoder, which integrates Context Modeling Self-Attention (CMSA) to capture spatial dependencies and Modal Fusion Self-Attention (MFSA) for effective cross-modal fusion. Furthermore, we design the tailored Detail Compensation Convolution (DCC) Block to improve texture details and enhances edge representations. Experiments show that UniCT Depth outperforms existing image, event, and fusion-based monocular depth estimation methods across key metrics.
BibTeX
@inproceedings{jing2025unict,
  author = {Luoxi Jing and Dianxi Shi and Zhe Liu and Songchang Jin and Chunping Qiu and Ziteng Qiao and Yuxian Li and Jianqiang Xia},
  title = {{UniCT Depth: Event-Image Fusion Based Monocular Depth Estimation with Convolution-Compensated ViT Dual SA Block}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/144},
}
Figure