Retain, Blend, and Exchange: A Quality-aware Spatial-Stereo Fusion Approach for Event Stream Recognition

Lan Chen, Dong Li, Xiao Wang, Pengpeng Shao, Wei Zhang, Yaowei Wang, Yonghong Tian, Jin Tang

Introduction

Event cameras, such as Dynamic Vision Sensors (DVS), represent a paradigm shift in computer vision. Unlike traditional cameras that capture full frames at fixed intervals, event cameras only record asynchronous brightness changes at each pixel. This allows for extremely high temporal resolution, high dynamic range, and low power consumption, making them ideal for high-speed motion and challenging lighting conditions. However, the sparse and non-uniform nature of event data poses significant challenges for standard recognition algorithms.

The Challenge of Event Stream Recognition

Conventional models often struggle with event data because they rely on a single representation—typically either 2D images, 3D voxels, or point clouds. This "monolithic" approach fails to capture the full richness of the event stream. Furthermore, when multiple representations are combined, standard fusion techniques like simple concatenation often fail to account for varying feature quality. Noisy or redundant features from one modality can inadvertently degrade the overall performance of the model.

Comparison of conventional frame-based imaging and asynchronous event-based streams.

The EFV++ Framework

To address these limitations, researchers have introduced EFV++, a novel dual-stream framework designed for robust event-based object recognition. The core philosophy of EFV++ is to process event data through two complementary branches simultaneously: spatial (Event Images) and volumetric (Event Voxels).

Dual Representations

  1. Event Images: The event stream is stacked into 2D images and processed by a Spatio-Temporal Transformer (ST-Transformer) to capture long-range spatial dependencies.
  2. Event Voxels: The stream is voxelized and converted into a graph structure, which is then processed by a Graph Neural Network (GNN) to preserve the sparse, non-uniform 3D structure of the events.

Quality-aware Fusion: The RBE Module

The defining innovation of EFV++ is the Retain, Blend, and Exchange (RBE) module. Rather than treating all features equally, the RBE module uses attention weights as a proxy for feature quality, categorizing them into three distinct levels:

  • Retain: High-quality features are kept to preserve unique, modality-specific information.
  • Blend: Medium-quality features are merged with the corresponding branch to enhance the overall representation.
  • Exchange: Low-quality (noisy) features are replaced with averaged features from the stronger branch, effectively filtering out noise.
The EFV++ architecture featuring dual-stream representation and the RBE module.

Hybrid Interaction Readout

Following feature refinement, a Bottleneck Transformer facilitates deep interaction between the two streams. The final classification is handled by a Hybrid Interaction Readout head, which utilizes a Gated Recurrent Unit (GRU) to process various permutations of the refined features, ensuring a diverse and robust final representation.

Experimental Results

EFV++ was evaluated across five major event-based benchmarks, achieving state-of-the-art (SOTA) results in every category:

  • Bullying10k: Achieved 90.51% accuracy, a +2.21% improvement over previous baselines.
  • N-Caltech101: Reached 89.7% accuracy, significantly outperforming the second-best model (77.8%).
  • ASL-DVS: Achieved a near-perfect 99.9% accuracy.
  • PokerEvent: Reached 55.40%, proving competitive even against multi-modal models that utilize both RGB and event data.

Conclusion

EFV++ demonstrates that intelligent, quality-aware fusion is key to unlocking the potential of event-based sensors. By discriminatively processing features through the RBE module, the framework prevents noisy data from degrading performance while leveraging the complementary strengths of image and voxel representations. Future work will focus on reducing the computational complexity of the dual-branch architecture and incorporating higher-level semantic reasoning to further enhance recognition capabilities in real-world scenarios.

Summary updated

BibTeX
@article{chen2025retain,
  author = {Lan Chen and Dong Li and Xiao Wang and Pengpeng Shao and Wei Zhang and Yaowei Wang and Yonghong Tian and Jin Tang},
  title = {{Retain, Blend, and Exchange: A Quality-aware Spatial-Stereo Fusion Approach for Event Stream Recognition}},
  journal = {IEEE Trans. Multimedia},
  year = {2025},
  doi = {10.1109/tmm.2025.3607771},
}
Figure