ISSAFE: Improving Semantic Segmentation in Accidents by Fusing Event-based Data

Jiaming Zhang, Kailun Yang, Rainer Stiefelhagen

TL;DR

Researchers have developed ISSAFE, a multi-modal framework that significantly improves how autonomous vehicles perceive traffic accidents. By fusing standard RGB images with "event-based" data (which captures pixel-level motion rather than static frames), the system achieves an 8.2% boost in segmentation accuracy during high-stakes scenarios like collisions, rollovers, and extreme motion blur.

What Problem Does This Paper Address?

Current semantic segmentation models—the "eyes" of autonomous vehicles—are remarkably good at identifying roads, cars, and pedestrians in normal conditions. However, they suffer from a "reality gap" when faced with traffic accidents.

Accidents introduce "edge cases" rarely seen in training datasets (like Cityscapes):

  • Deformation & Orientation: Overturned trucks or crumpled cars don't look like the "vehicles" the model was trained on.
  • Motion Blur: Fast-moving objects during a crash become a smear of pixels for standard cameras.
  • Adverse Lighting: Back-lighting or windshield reflections can blind traditional sensors.

When these abnormalities occur, standard models experience a sharp drop in performance, which is exactly when a safety system needs to be most reliable to avoid pileups or secondary collisions.

Methods

The authors propose a multi-modal architecture called ISSAFE (Improving Semantic Segmentation in Accidents by Fusing Event-based data). The core innovation is the integration of Event Cameras (Dynamic Vision Sensors). Unlike traditional cameras that capture frames at fixed intervals, event cameras only record changes in intensity at each pixel asynchronously. This gives them a massive dynamic range (>120dB) and virtually eliminates motion blur.

The paper explores two fusion strategies:

  1. S2D (Sparse-to-Dense): Fuses sparse event features into the RGB branch using a channel-wise attention module to emphasize motion-related features.
  2. D2S (Dense-to-Sparse): Uses high-level semantic features from the RGB branch to "guide" the event branch, deactivating non-event features and focusing on the most relevant motion data.
  3. Event-aware Domain Adaptation (EDA): Since event data is monochromatic and focuses on structure/motion, it acts as a "bridge" to help models trained on normal driving data generalize to the chaotic "style" of accident scenes.

Because real-world event data for accidents is scarce, the team used EventGAN to synthesize high-fidelity event streams from existing RGB video sequences.

Datasets

The paper introduces and utilizes several key datasets:

  • DADA-seg (New): A specialized benchmark for Semantic Segmentation in Accidents (SSA). It contains 313 accident sequences (12,520 frames) with manual pixel-wise annotations for 19 classes. It includes hazards like motion blur, overturns, and occlusions.
  • Source Datasets: The models were pre-trained on standard benchmarks including Cityscapes, KITTI-360, BDD3K, and ApolloScape.
  • Merge3: A combined dataset used to test the model's ability to learn from diverse source domains simultaneously.

Key Insights and Ideas

  • Events as a Stabilizer: Event data provides "complementary information." While an RGB frame might be a blurry mess during a collision, the event stream preserves the fine-grained edges of the moving objects.
  • The D2S Advantage: The Dense-to-Sparse (D2S) approach proved more effective than S2D. By using the RGB branch to filter the event branch, the model effectively ignores "noise" and focuses on the semantic structure of the crash.
  • Monochromatic Consistency: One "aha moment" is that event data is semantically more consistent across different environments than RGB data. This makes it an ideal candidate for Domain Adaptation, as the "look" of an event stream doesn't change as drastically as RGB colors do when moving from a sunny city to a dark accident site.

Key Results

  • Performance Boost: The ISSAFE-D2S model achieved a +8.2% mIoU gain on the DADA-seg dataset compared to the RGB-only baseline.
  • State-of-the-Art: ISSAFE outperformed more than 10 established segmentation models (including DeepLabV3+ and OCRNet) when tested on accidental edge cases.
  • Robustness: The model successfully segmented overturned vehicles and pedestrians in low-light conditions where traditional models failed completely.
  • Efficiency: By using a ResNet-18 backbone, the architecture maintains the potential for real-time deployment on mobile platforms.

Limitations and Future Work

  • Synthetic vs. Real Events: A primary limitation is the reliance on synthesized event data. While EventGAN is sophisticated, real event cameras may introduce different types of noise not present in the simulations.
  • Night-time Challenges: Although the model performs better than baselines at night, segmenting extremely dark scenes remains a significant hurdle for the field.
  • Unlabeled Data: The DADA-seg dataset contains a large amount of unlabeled video. The authors suggest that future work could involve self-supervised or contrastive learning to extract even more value from these sequences without requiring manual annotation.

Summary updated

Abstract

Ensuring the safety of all traffic participants is a prerequisite for bringing intelligent vehicles closer to practical applications. The assistance system should not only achieve high accuracy under normal conditions, but obtain robust perception against extreme situations. However, traffic accidents that involve object collisions, deformations, overturns, etc., yet unseen in most training sets, will largely harm the performance of existing semantic segmentation models. To tackle this issue, we present a rarely addressed task regarding semantic segmentation in accidental scenarios, along with an accident dataset DADA-seg. It contains 313 various accident sequences with 40 frames each, of which the time windows are located before and during a traffic accident. Every 11th frame is manually annotated for benchmarking the segmentation performance. Furthermore, we propose a novel event-based multi-modal segmentation architecture ISSAFE. Our experiments indicate that event-based data can provide complementary information to stabilize semantic segmentation under adverse conditions by preserving fine-grain motion of fast-moving foreground (crash objects) in accidents. Our approach achieves +8.2% mIoU performance gain on the proposed evaluation set, exceeding more than 10 state-of-the-art segmentation methods. The proposed ISSAFE architecture is demonstrated to be consistently effective for models learned on multiple source databases including Cityscapes, KITTI-360, BDD and ApolloScape.
BibTeX
@inproceedings{zhang2021issafe,
  author = {Jiaming Zhang and Kailun Yang and Rainer Stiefelhagen},
  title = {{ISSAFE: Improving Semantic Segmentation in Accidents by Fusing Event-based Data}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2021},
  doi = {10.1109/iros51168.2021.9636109},
}

Loading related content...

Figure