ESEG: Event-Based Segmentation Boosted by Explicit Edge-Semantic Guidance

Yucheng Zhao, Gengyu Lyu, Ke Li, Zihao Wang, Hao Chen, Zhen Yang, Yongjian DengAAAI 2025 · OpenReview

TL;DR

This paper introduces ESEG, a novel framework for event-based semantic segmentation that explicitly guides the model with semantic edge information. By generating and leveraging these edge-semantic labels, ESEG significantly improves the model's ability to accurately segment objects in challenging visual conditions where traditional cameras struggle.

What Problem Does This Paper Address?

Semantic segmentation, a crucial task in computer vision for understanding image content, faces significant challenges with event-based cameras. While event cameras excel in low-light and high-speed scenarios due to their asynchronous, event-driven nature, the resulting data is often sparse and noisy. This sparsity makes it difficult for standard segmentation models to reliably locate and extract object boundaries. Existing approaches often rely on complex architectures or transfer learning to implicitly learn these cues, but this paper questions whether explicitly informing the model about important regions could be more effective. The core problem is how to effectively leverage the unique properties of event data for robust semantic segmentation, particularly in identifying reliable object edges amidst noise and sparsity.

Methods

The ESEG framework tackles this problem through a multi-pronged approach:

  • Explicit Edge-Semantic Supervision: Instead of expecting the model to infer edge information from sparse event data alone, ESEG introduces explicit semantic edge labels as a supervisory signal. This guides the model to focus on reliable edge regions.
  • SELSAM (Semantic Edge Label with SAM): To address the lack of existing semantic edge labels for event-based datasets, the authors propose SELSAM. This algorithm leverages the Segment Anything Model (SAM) to generate high-quality semantic edge labels by filtering out internal object textures and focusing on clear object boundaries.
  • D²CAF (Density-Aware Dynamic-Window Cross-Attention Fusion): This novel fusion module is designed to effectively integrate the sparse edge-semantic features with the dense semantic features. D²CAF incorporates:
  • Density Indicator Matrix (DIM): This component helps the model understand the information density differences between edge and dense features, allowing for adaptive attention.
  • Dynamic Window Masking: This mechanism adjusts the receptive field based on whether a region is an edge or not, considering more context for edges and less for smooth areas to avoid noise.
  • Cross-Attention Fusion: This allows the edge-semantic features to guide the fusion process, refining the dense semantic features.

The ESEG architecture consists of three branches: an edge-semantic branch (trained with generated edge-semantic labels), a dense-semantic branch (trained with standard semantic labels), and a fusion branch (D²CAF) that combines the outputs of the first two.

Datasets

The paper evaluates ESEG on two commonly used datasets for event-based semantic segmentation:

  • DSEC-Semantic: This dataset focuses on driving scenarios and provides both event and RGB data with semantic labels.
  • DDD17: This dataset also includes event-based vision data, RGB images, and IMU data, with semantic labels for driving scenarios and objects.

The authors generated new semantic edge labels for these datasets using their SELSAM algorithm.

Key Insights and Ideas

  • Explicit Guidance is Powerful: The core insight is that directly providing semantic edge information as supervision is more effective than relying on models to implicitly learn it from sparse event data. This is particularly relevant for event cameras where edges are a primary signal.
  • SAM as a Label Generator: Leveraging large foundation models like SAM for generating specialized labels (semantic edges) is a practical and effective strategy, especially when manual annotation is scarce or difficult.
  • Density and Dynamics Matter for Fusion: The D²CAF module's success highlights the importance of considering not just the features themselves, but also their density and the spatial context (dynamic window) when fusing information from different sources (sparse edges vs. dense semantics).
  • Event Data's Unique Strengths: The paper reinforces the idea that event cameras offer a unique advantage in challenging visual conditions, and specialized methods are needed to fully exploit this potential.

Key Results

  • State-of-the-Art Performance: ESEG achieves state-of-the-art performance on both the DSEC-Semantic and DDD17 datasets, outperforming previous methods.
  • Significant Gains over Baselines: ESEG-B (using MiT-b0) outperforms the baseline SegFormer by 1.7% mIoU on DSEC, demonstrating the effectiveness of the explicit edge-semantic supervision.
  • Competitive with Multi-Modal Approaches: ESEG shows comparable accuracy to methods that leverage both event and RGB data, even when trained solely on event data, indicating its strong ability to extract semantic information.
  • Ablation Studies Confirm Design Choices: Ablation studies show that each component of ESEG (edge-semantic supervision, DIM, dynamic window masking) contributes to the overall performance improvement. The full D²CAF module with dynamic window masking yields the highest accuracy.

Limitations and Future Work

  • Label Generation Dependency: The performance of ESEG relies on the quality of the generated semantic edge labels. While SELSAM is effective, future work could explore more robust or automated label generation methods.
  • Dataset Specificity: While ESEG performs well on driving datasets, its generalizability to other domains with event cameras could be further explored.
  • Potential for Other Tasks: The authors suggest that the ESEG framework, particularly its edge-semantic guidance and fusion mechanisms, has the potential to be adapted for other event-based vision tasks beyond semantic segmentation.

Summary updated

BibTeX
@inproceedings{zhao2025eseg,
  author = {Yucheng Zhao and Gengyu Lyu and Ke Li and Zihao Wang and Hao Chen and Zhen Yang and Yongjian Deng},
  title = {{ESEG: Event-Based Segmentation Boosted by Explicit Edge-Semantic Guidance}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=d39Q9tKBB3},
}
Figure