Dual Transfer Learning for Event-based End-task Prediction via Pluggable Event to Image Translation

Lin Wang, Yujeong Chae, Kuk-Jin Yoon

Introduction

Event cameras, or neuromorphic sensors, represent a significant shift in computer vision technology. Unlike traditional frame-based cameras that capture full images at fixed intervals, event cameras asynchronously report per-pixel intensity changes. This unique mechanism offers advantages such as high temporal resolution, high dynamic range (HDR), and minimal motion blur. However, the resulting data is inherently sparse, consisting only of "dots and lines" representing edges and movement. Standard deep learning architectures often struggle to recover fine-grained structural details from this sparse input, leading to suboptimal performance in downstream tasks like semantic segmentation and depth estimation.

The Dual Transfer Learning (DTL) Framework

To address the sparsity of event data, this paper proposes Dual Transfer Learning (DTL). The core philosophy is that learning to reconstruct a standard image (Event to Image Translation) serves as a powerful regularizer for understanding the scene's semantics (Event to End-task Learning).

The architecture consists of two primary branches sharing a common encoder:

  1. Event to End-task Learning (EEL) Branch: The main pipeline responsible for predicting the final output, such as a semantic segmentation map.
  2. Event to Image Translation (EIT) Branch: A "pluggable" auxiliary branch designed to reconstruct a standard grayscale image from the event stream. By forcing the shared encoder to retain enough information for image reconstruction, the model learns richer structural representations.
Detailed architecture of the Dual Transfer Learning (DTL) framework, illustrating the shared encoder and the dual transfer mechanisms (feature-level and prediction-level) between the EEL and EIT branches.

Knowledge Transfer Mechanisms

The framework employs a Transfer Learning (TL) module to bridge the two branches during training:

  • Feature-level Transfer: Utilizes affinity graphs to ensure spatial relationships in the reconstructed image (e.g., pixels belonging to the same object) are preserved in the end-task features.
  • Prediction-level Transfer: Employs a pre-trained "Teacher Network" (trained on standard images) to provide soft labels, guiding the event-based branch to align its logic with models that have access to full-frame visual context.

Crucially, the EIT branch and the Teacher Network are only used during training. At inference, these components are discarded, allowing the EEL branch to benefit from the learned structural knowledge with zero additional computational overhead.

Experimental Evaluation

The DTL framework was evaluated on two major benchmarks: DDD17 for semantic segmentation and MVSEC for monocular dense depth estimation.

Key Results

  • Semantic Segmentation: On the DDD17 dataset, DTL achieved an 8% to 10% improvement in Mean Intersection over Union (MIoU) over the baseline EEL model. It also outperformed specialized architectures like EvSegNet by approximately 4%.
  • Depth Estimation: On the MVSEC dataset, the framework reduced Absolute Relative Error by 10% to 17% compared to existing state-of-the-art methods.
  • Robustness in Extreme Lighting: The model demonstrated superior performance in HDR scenarios. In night sequences where standard Active Pixel Sensor (APS) frames were obscured by low light or glare, the DTL-enhanced model successfully identified object shapes and scene structures.
Qualitative comparison on the DDD17 dataset. From left to right: (a) Input event stream, (b) Ground truth segmentation, (c) Corresponding APS frame, (d) Baseline prediction, (e) Reconstructed image from the EIT branch, and (f) Final DTL segmentation prediction.

Conclusion

Dual Transfer Learning provides a robust solution to the sparsity problem in event-based vision. By treating image reconstruction as an auxiliary task, the model captures structural details that are typically lost in direct end-task learning. The "pluggable" nature of the EIT branch ensures that these performance gains do not come at the cost of inference speed or model size. While currently dependent on synchronized APS frames for training, DTL establishes a new standard for high-performance, efficient neuromorphic vision.

Summary updated

Abstract

Event cameras are novel sensors that perceive the per-pixel intensity changes and output asynchronous event streams with high dynamic range and less motion blur. It has been shown that events alone can be used for end-task learning, e.g., semantic segmentation, based on encoder-decoder-like networks. However, as events are sparse and mostly reflect edge information, it is difficult to recover original details merely relying on the decoder. Moreover, most methods resort to pixel-wise loss alone for supervision, which might be insufficient to fully exploit the visual details from sparse events, thus leading to less optimal performance. In this paper, we propose a simple yet flexible two-stream framework named Dual Transfer Learning (DTL) to effectively enhance the performance on the end-tasks without adding extra inference cost. The proposed approach consists of three parts: event to end-task learning (EEL) branch, event to image translation (EIT) branch, and transfer learning (TL) module that simultaneously explores the feature-level affinity information and pixel-level knowledge from the EIT branch to improve the EEL branch. This simple yet novel method leads to strong representation learning from events and is evidenced by the significant performance boost on the end-tasks such as semantic segmentation and depth estimation.
BibTeX
@inproceedings{wang2021dual,
  author = {Lin Wang and Yujeong Chae and Kuk-Jin Yoon},
  title = {{Dual Transfer Learning for Event-based End-task Prediction via Pluggable Event to Image Translation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00214},
}
Figure