Segment Any Event Streams via Weighted Adaptation of Pivotal Tokens

Zhiwen Chen, Zhiyu Zhu, Yifan Zhang, Junhui Hou, Guangming Shi, Jinjian WuCVPR 2024 · DOI

Abstract

The Segment Anything Model (SAM) has set a new benchmark for universal object segmentation in the RGB domain. However, its performance degrades significantly in challenging visual conditions such as extreme low light or high-speed motion. Event cameras, which capture per-pixel brightness changes with microsecond latency, offer a solution to these hardware limitations but present a significant domain gap for standard vision models. This paper introduces EventSAM, a framework that adapts SAM to event streams through a novel weighted distillation approach. By identifying and prioritizing "pivotal tokens" within the transformer architecture, EventSAM achieves state-of-the-art zero-shot segmentation performance on event data while preserving the foundational knowledge of the original SAM.

Introduction

The emergence of foundation models like SAM has revolutionized computer vision by enabling high-quality segmentation across diverse scenes without task-specific training. Despite this success, SAM remains tethered to the limitations of traditional frame-based RGB sensors. In scenarios involving high dynamic range (HDR) or rapid motion, RGB cameras suffer from overexposure or motion blur, leading to a total failure of segmentation masks.

Event cameras (neuromorphic sensors) operate differently, asynchronously recording only local intensity changes. This allows them to excel where RGB sensors fail. However, the sparse and asynchronous nature of event data makes it incompatible with models trained on dense RGB grids. Previous attempts to bridge this gap often involved training specialized models from scratch, which lack the broad generalization capabilities of large-scale foundation models. EventSAM addresses this by transferring SAM’s spatial reasoning into the event domain through parameter-efficient fine-tuning and intelligent knowledge distillation.

Methodology: The EventSAM Framework

The core of the proposed approach is a teacher-student distillation strategy where a frozen RGB-based SAM (the teacher) guides an event-based encoder (the student).

1. Cross-modal Distillation with Token Mixing

To stabilize the initial learning phase, the authors employ a "Token Mixing" strategy. During training, a small fraction of event tokens are randomly replaced with their corresponding RGB tokens. This provides a "bridge" of familiar information, helping the student model align event features with the high-level semantic space of the teacher.

2. Correlation-aware Weighted Token Distillation

Not all tokens in a transformer block are equally critical for defining an object. The authors introduce the concept of pivotal tokens—those that contribute most significantly to the final segmentation output.

The EventSAM architecture featuring the teacher-student distillation process and the identification of pivotal tokens.

As shown in Figure 2, the framework calculates a transition matrix $H^{(s)}$ by tracking the flow of attention from early layers to the final layer. This matrix allows the model to assign higher weights to the distillation loss for tokens that represent critical structural information, such as object boundaries. By focusing the student's learning on these pivotal tokens, the model achieves higher fidelity in its mask predictions.

3. Parameter-Efficient Fine-tuning

To maintain the "zero-shot" capabilities of the original model, the authors avoid retraining the entire SAM backbone. Instead, they fine-tune only the Multi-Layer Perceptron (MLP) layers within the transformer blocks. This strategy preserves the pre-trained spatial priors while adapting the input processing to the event modality.

Experimental Results

The model was evaluated on the RGBE-SEG dataset, a large-scale collection of 65,957 image-event pairs categorized by difficulty, and the MVSEC dataset for zero-shot testing.

Quantitative Performance

EventSAM demonstrates a significant performance leap over existing event-based segmentation methods and the original SAM.

Quantitative comparison of EventSAM against state-of-the-art methods across different difficulty levels.

As detailed in Table 1, EventSAM achieves a mean Intersection over Union (mIoU) of 0.41 across all categories, representing a 9% improvement over previous state-of-the-art methods. Notably, in the "Hard" category—where lighting and motion are most extreme—EventSAM maintains a robust mIoU of 0.34, whereas the original SAM drops to 0.15.

Qualitative Robustness and Generalization

Qualitative analysis confirms that EventSAM produces clean, accurate masks in environments where RGB-based SAM fails completely. In zero-shot evaluations on the MVSEC dataset, EventSAM outperformed models specifically trained on that data, proving that the "Segment Anything" philosophy successfully generalizes to the event domain.

Conclusion and Future Work

EventSAM successfully bridges the gap between high-performance foundation models and the robust sensing capabilities of event cameras. By prioritizing pivotal tokens during distillation, the framework enables precise segmentation in conditions that were previously inaccessible to computer vision. Future research will focus on optimizing the model's computational efficiency for real-time deployment on edge devices and improving performance in texture-less regions where event generation is minimal.

Summary updated

BibTeX
@inproceedings{chen2024segment,
  author = {Zhiwen Chen and Zhiyu Zhu and Yifan Zhang and Junhui Hou and Guangming Shi and Jinjian Wu},
  title = {{Segment Any Event Streams via Weighted Adaptation of Pivotal Tokens}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.00373},
}
Figure