ISSAFE: Improving Semantic Segmentation in Accidents by Fusing Event-based Data

Jiaming Zhang, Kailun Yang, Rainer Stiefelhagen

Abstract

Ensuring the safety of all traffic participants is a prerequisite for bringing intelligent vehicles closer to practical applications. While assistance systems achieve high accuracy under normal conditions, they often fail in extreme situations like traffic accidents involving object collisions, deformations, and overturns. These "edge cases" are rarely seen in training sets, largely harming the performance of existing semantic segmentation models. To tackle this, we present a rarely addressed task regarding semantic segmentation in accidental scenarios, along with a new accident dataset, DADA-seg. We propose ISSAFE, a novel event-based multi-modal segmentation architecture. Our experiments indicate that event-based data provides complementary information to stabilize segmentation under adverse conditions by preserving fine-grain motion of fast-moving foreground objects. ISSAFE achieves an +8.2% mIoU performance gain on DADA-seg, exceeding more than 10 state-of-the-art methods.

Introduction: The Reality Gap in Accident Scenarios

Current semantic segmentation models—the "eyes" of autonomous vehicles—are remarkably effective in identifying roads, cars, and pedestrians under normal conditions. However, they suffer from a significant "reality gap" when faced with traffic accidents. Accidents introduce visual abnormalities that standard models struggle to interpret:

  • Deformation & Orientation: Overturned trucks or crumpled cars do not match the canonical "vehicle" shapes the models were trained on.
  • Motion Blur: Fast-moving objects during a crash become a smear of pixels for standard frame-based cameras.
  • Adverse Lighting: Back-lighting, windshield reflections, or extreme darkness can blind traditional sensors.
Figure 1: Examples of challenging visual conditions in traffic accidents, including motion blur, vehicle overturns, backlighting, and occlusions.

When these abnormalities occur, standard models experience a sharp drop in performance, precisely when a safety system needs to be most reliable to avoid secondary collisions.

Methodology: Fusing Event-based Data

The core innovation of ISSAFE is the integration of Event Cameras (Dynamic Vision Sensors). Unlike traditional cameras that capture frames at fixed intervals, event cameras only record changes in intensity at each pixel asynchronously. This provides a massive dynamic range (>120dB) and virtually eliminates motion blur, making them ideal for high-speed accident scenarios.

Fusion Strategies

The researchers explored two primary multi-modal fusion architectures to combine RGB and event data:

  1. Sparse-to-Dense (S2D): Fuses sparse event features into the RGB branch using a channel-wise attention module to emphasize motion-related features.
  2. Dense-to-Sparse (D2S): Uses high-level semantic features from the RGB branch to "guide" the event branch, deactivating non-event features and focusing on the most relevant motion data. This approach proved more effective by filtering out noise and focusing on semantic structure.
Figure 5: Comparison of the two proposed fusion strategies: Sparse-to-Dense (S2D) on the left and Dense-to-Sparse (D2S) on the right.

Event-aware Domain Adaptation (EDA)

Because event data is monochromatic and focuses on structure and motion, it acts as a "bridge" to help models trained on normal driving data (like Cityscapes) generalize to the chaotic style of accident scenes. To overcome the scarcity of real-world accident event data, the team used EventGAN to synthesize high-fidelity event streams from existing RGB video sequences.

Datasets and Benchmarking

The study introduces DADA-seg, a specialized benchmark for Semantic Segmentation in Accidents (SSA). It contains 313 accident sequences (12,520 frames) with manual pixel-wise annotations for 19 classes. The model was also validated across multiple source domains, including Cityscapes, KITTI-360, BDD, and ApolloScape, demonstrating consistent effectiveness.

Key Results and Insights

  • Performance Boost: The ISSAFE-D2S model achieved a +8.2% mIoU gain on the DADA-seg dataset compared to the RGB-only baseline.
  • Robustness: The model successfully segmented overturned vehicles and pedestrians in low-light conditions where traditional models failed completely. Event data acts as a stabilizer, preserving fine-grained edges of moving objects even when RGB frames are blurred.
  • Efficiency: By utilizing a ResNet-18 backbone, the architecture maintains the potential for real-time deployment on mobile platforms.

Conclusion and Future Work

ISSAFE demonstrates that multi-modal fusion with event-based data significantly enhances the reliability of autonomous perception in critical safety scenarios. Future research will focus on bridging the gap between synthetic and real event data, improving performance in extremely dark night-time scenes, and leveraging unlabeled video data through self-supervised learning techniques.

Summary updated

Abstract

Ensuring the safety of all traffic participants is a prerequisite for bringing intelligent vehicles closer to practical applications. The assistance system should not only achieve high accuracy under normal conditions, but obtain robust perception against extreme situations. However, traffic accidents that involve object collisions, deformations, overturns, etc., yet unseen in most training sets, will largely harm the performance of existing semantic segmentation models. To tackle this issue, we present a rarely addressed task regarding semantic segmentation in accidental scenarios, along with an accident dataset DADA-seg. It contains 313 various accident sequences with 40 frames each, of which the time windows are located before and during a traffic accident. Every 11th frame is manually annotated for benchmarking the segmentation performance. Furthermore, we propose a novel event-based multi-modal segmentation architecture ISSAFE. Our experiments indicate that event-based data can provide complementary information to stabilize semantic segmentation under adverse conditions by preserving fine-grain motion of fast-moving foreground (crash objects) in accidents. Our approach achieves +8.2% mIoU performance gain on the proposed evaluation set, exceeding more than 10 state-of-the-art segmentation methods. The proposed ISSAFE architecture is demonstrated to be consistently effective for models learned on multiple source databases including Cityscapes, KITTI-360, BDD and ApolloScape.
BibTeX
@inproceedings{zhang2021issafe,
  author = {Jiaming Zhang and Kailun Yang and Rainer Stiefelhagen},
  title = {{ISSAFE: Improving Semantic Segmentation in Accidents by Fusing Event-based Data}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2021},
  doi = {10.1109/iros51168.2021.9636109},
}
Figure