Abstract
Ensuring the safety of all traffic participants is a prerequisite for bringing intelligent vehicles closer to practical applications. While assistance systems achieve high accuracy under normal conditions, they often fail in extreme situations like traffic accidents involving object collisions, deformations, and overturns. These "edge cases" are rarely seen in training sets, largely harming the performance of existing semantic segmentation models. To tackle this, we present a rarely addressed task regarding semantic segmentation in accidental scenarios, along with a new accident dataset, DADA-seg. We propose ISSAFE, a novel event-based multi-modal segmentation architecture. Our experiments indicate that event-based data provides complementary information to stabilize segmentation under adverse conditions by preserving fine-grain motion of fast-moving foreground objects. ISSAFE achieves an +8.2% mIoU performance gain on DADA-seg, exceeding more than 10 state-of-the-art methods.
Introduction: The Reality Gap in Accident Scenarios
Current semantic segmentation models—the "eyes" of autonomous vehicles—are remarkably effective in identifying roads, cars, and pedestrians under normal conditions. However, they suffer from a significant "reality gap" when faced with traffic accidents. Accidents introduce visual abnormalities that standard models struggle to interpret:
- Deformation & Orientation: Overturned trucks or crumpled cars do not match the canonical "vehicle" shapes the models were trained on.
- Motion Blur: Fast-moving objects during a crash become a smear of pixels for standard frame-based cameras.
- Adverse Lighting: Back-lighting, windshield reflections, or extreme darkness can blind traditional sensors.
When these abnormalities occur, standard models experience a sharp drop in performance, precisely when a safety system needs to be most reliable to avoid secondary collisions.
Methodology: Fusing Event-based Data
The core innovation of ISSAFE is the integration of Event Cameras (Dynamic Vision Sensors). Unlike traditional cameras that capture frames at fixed intervals, event cameras only record changes in intensity at each pixel asynchronously. This provides a massive dynamic range (>120dB) and virtually eliminates motion blur, making them ideal for high-speed accident scenarios.
Fusion Strategies
The researchers explored two primary multi-modal fusion architectures to combine RGB and event data:
- Sparse-to-Dense (S2D): Fuses sparse event features into the RGB branch using a channel-wise attention module to emphasize motion-related features.
- Dense-to-Sparse (D2S): Uses high-level semantic features from the RGB branch to "guide" the event branch, deactivating non-event features and focusing on the most relevant motion data. This approach proved more effective by filtering out noise and focusing on semantic structure.
Event-aware Domain Adaptation (EDA)
Because event data is monochromatic and focuses on structure and motion, it acts as a "bridge" to help models trained on normal driving data (like Cityscapes) generalize to the chaotic style of accident scenes. To overcome the scarcity of real-world accident event data, the team used EventGAN to synthesize high-fidelity event streams from existing RGB video sequences.
Datasets and Benchmarking
The study introduces DADA-seg, a specialized benchmark for Semantic Segmentation in Accidents (SSA). It contains 313 accident sequences (12,520 frames) with manual pixel-wise annotations for 19 classes. The model was also validated across multiple source domains, including Cityscapes, KITTI-360, BDD, and ApolloScape, demonstrating consistent effectiveness.
Key Results and Insights
- Performance Boost: The ISSAFE-D2S model achieved a +8.2% mIoU gain on the DADA-seg dataset compared to the RGB-only baseline.
- Robustness: The model successfully segmented overturned vehicles and pedestrians in low-light conditions where traditional models failed completely. Event data acts as a stabilizer, preserving fine-grained edges of moving objects even when RGB frames are blurred.
- Efficiency: By utilizing a ResNet-18 backbone, the architecture maintains the potential for real-time deployment on mobile platforms.
Conclusion and Future Work
ISSAFE demonstrates that multi-modal fusion with event-based data significantly enhances the reliability of autonomous perception in critical safety scenarios. Future research will focus on bridging the gap between synthetic and real event data, improving performance in extremely dark night-time scenes, and leveraging unlabeled video data through self-supervised learning techniques.
Summary updated