TL;DR
This paper introduces EVSNet, a novel framework that leverages event cameras to improve low-light video semantic segmentation. By fusing image and event data, EVSNet effectively captures temporal motion, leading to more robust and accurate segmentation in challenging lighting conditions, outperforming existing methods with significantly higher parameter efficiency.
What Problem Does This Paper Address?
Video semantic segmentation (VSS) is crucial for many computer vision tasks, but its performance drastically degrades in low-light environments. This is due to limited visibility, reduced contextual details, and increased noise, which hinder accurate object boundary discrimination and temporal consistency. Traditional cameras struggle to capture the wide range of brightness levels in such scenarios, leading to flickering effects and inaccurate predictions. While event cameras offer a promising alternative due to their high temporal resolution, dynamic range, and robustness to lighting, effectively integrating their sparse, motion-centric data with traditional image data for VSS remains a challenge, particularly in low-light conditions.
Methods
The proposed EVSNet framework tackles this problem by introducing a novel approach to fuse image and event modalities. It consists of four main components:
- Image Encoder: A lightweight backbone (e.g., Afformer, MiT) extracts multi-scale features from input RGB frames. A Multi-scale Mixer then aggregates these features.
- Motion Extraction Module (MEM): This module is designed to extract both short-term and long-term temporal motion features from event data. It first uses an Event Encoder to process event images (generated from event streams) and then employs a Temporal Convolutional Block to learn long-term temporal relations. This is inspired by the Atkinson-Shiffrin memory model, suggesting the importance of both short and long-term memory.
- Motion Fusion Module (MFM): This module adaptively integrates the image features from the Image Encoder and the motion features from the MEM. It comprises a Channel Attention Layer and a Spatial Attention Layer. The Channel Attention Layer uses event data to guide the attention across image feature channels, while the Spatial Attention Layer captures local and global spatial contexts. The paper explores different arrangements of these layers, finding a sequential placement to be most effective.
- Temporal Decoder: This component utilizes the fused features from the MFM to model temporal consistency across frames and generate the final segmentation predictions.
A key innovation is the use of event data specifically for motion extraction, avoiding direct fusion of raw event and image features early on to mitigate misalignment issues.
Datasets
The authors evaluate EVSNet on three large-scale datasets:
- Low-light VSPW: A synthetic dataset with 2806 videos, providing per-frame pixel-level annotations at 15 FPS.
- Low-light Cityscapes: A synthetic dataset derived from the original Cityscapes dataset, focusing on urban driving scenes.
- NightCity: A real-world dataset featuring urban driving scenes at nighttime with ground truth pixel-level semantic annotations.
The paper also mentions synthesizing low-light videos from normal light frames using linear scaling and gamma correction, and using a video-to-event simulator (v2e) to generate event images.
Key Insights and Ideas
- Event cameras as a powerful tool for low-light VSS: The paper convincingly demonstrates that event cameras, by capturing motion dynamics, can significantly enhance segmentation performance in low-light scenarios where traditional cameras struggle.
- Separating motion extraction from image features: Instead of directly fusing raw event and image features, EVSNet first extracts motion features from events and then fuses them with image features. This strategy effectively addresses the domain gap and potential misalignment issues between modalities.
- Leveraging short-term and long-term temporal motions: Inspired by memory models, the MEM module explicitly learns both short-term (from event encoder) and long-term (from temporal convolutional block) motion patterns, which are crucial for temporal consistency in video.
- Adaptive fusion is key: The MFM, with its channel and spatial attention mechanisms, adaptively combines image and motion features, allowing the model to leverage the strengths of each modality effectively.
Key Results
EVSNet achieves state-of-the-art (SOTA) performance on all three tested datasets.
- On the low-light VSPW dataset, EVSNet with different backbones achieved mIoU scores ranging from 23.6% to 34.1%, outperforming other SOTA methods by a significant margin. Notably, it achieved up to 11x higher parameter efficiency compared to some SOTA methods.
- On low-light Cityscapes, EVSNet achieved mIoU scores from 57.9% to 63.2%, outperforming other SOTA models.
- On NightCity, EVSNet achieved mIoU of 53.9% and 55.2% with a significantly smaller model size.
- Qualitative results show EVSNet generating more accurate boundaries and resolving temporal inconsistencies compared to CFFM [49], a strong baseline.
Limitations and Future Work
- Dataset limitations: While the paper uses large datasets, the real-world NightCity dataset is still relatively small compared to synthetic ones.
- Computational cost of event simulation: The paper relies on event simulators for training. While the proposed EVSNet is lightweight, the simulation process itself can be computationally intensive.
- Future work could explore: Further investigation into more efficient event simulation techniques, exploring other event-based sensors, and extending the framework to other low-light video understanding tasks. The authors also suggest that the sequential arrangement of attention layers in the MFM provides better results, hinting at further exploration of attention mechanisms.
Summary updated