Retain, Blend, and Exchange: A Quality-Aware Spatial-Stereo Fusion Approach for Event Stream Recognition

Lan Chen, Dong Li, Xiao Wang, Pengpeng Shao, Wei Zhang, Yaowei Wang, Yonghong Tian, Jin Tang

TL;DR

Researchers have developed EFV++, a novel dual-stream framework for event-based object recognition that simultaneously processes event data as both images and voxels. By introducing a "Retain, Blend, and Exchange" (RBE) module, the system intelligently filters and fuses features based on their quality, achieving state-of-the-art results on major benchmarks, including a significant +2.21% accuracy boost on the large-scale Bullying10k dataset.

What Problem Does This Paper Address?

Standard cameras record light intensity for every pixel at a fixed frame rate. In contrast, event cameras (like Dynamic Vision Sensors or DVS) only record changes in brightness, producing a sparse, asynchronous stream of "events." While event cameras excel in high-speed motion and low-light conditions, recognizing objects from this data is difficult.

Most existing models rely on a single way of representing these events—either as 2D images, 3D voxels, or point clouds. The authors argue that this "monolithic" approach limits the model's expressiveness. Furthermore, when researchers do try to combine different representations, they often use "brute-force" fusion (like simple concatenation), which ignores the fact that some features are high-quality (informative) while others are redundant or noisy.

Methods

The authors propose EFV++, which utilizes a dual-branch architecture to capture both spatial and volumetric information:

  1. Dual Representations: The event stream is transformed into Event Images (processed by a Spatio-Temporal Transformer) and Event Voxels (processed by a Graph Neural Network or GNN).
  2. The RBE Module (Retain, Blend, and Exchange): This is the core innovation. Using attention weights as a proxy for feature quality, the module categorizes features into three levels:
  3. High-quality: Retained to preserve unique modality-specific info.
  4. Medium-quality: Blended with the other branch to enhance representation.
  5. Low-quality: Exchanged (replaced) with average features from the stronger branch to eliminate noise.
  6. Bottleneck Transformer: A specialized Transformer layer that facilitates deep interaction between the two refined feature sets.
  7. Hybrid Interaction Readout: Instead of a simple final layer, the model uses a Gated Recurrent Unit (GRU) to fuse different permutations of the image, voxel, and bottleneck features, ensuring the final representation is as diverse and robust as possible.

Datasets

The model was evaluated on five widely used event-based classification benchmarks:

  • ASL-DVS: American Sign Language finger-spelling.
  • N-Caltech101 & N-MNIST: Neuromorphic versions of classic computer vision datasets.
  • Bullying10k: A large-scale, privacy-preserving dataset for detecting bullying behavior.
  • PokerEvent: Recognizing character patterns on poker cards.

Key Insights and Ideas

  • Quality Matters in Fusion: The "aha moment" here is that directly fusing two streams can be counterproductive if one stream contains "garbage" features. By treating features discriminatively based on their attention scores, the model prevents noisy data from one modality from degrading the other.
  • Permutation Diversity: The authors found that the order in which features are presented to the final readout layer matters. By using a GRU to process multiple permutations of feature sequences, the model captures complex combinatorial relationships that standard linear layers miss.
  • Complementary Strengths: Transformers are excellent at capturing long-range spatial dependencies in images, while GNNs are better suited for the sparse, non-uniform structure of voxels. Combining them covers the weaknesses of each.

Key Results

EFV++ achieved State-of-the-Art (SOTA) performance across the board:

  • Bullying10k: Achieved 90.51% accuracy, outperforming the previous best (SimpleBaseline) by 2.21%.
  • ASL-DVS: Reached a near-perfect 99.9% accuracy.
  • N-Caltech101: Achieved 89.7%, a significant jump over the second-place VMV-GCN (77.8%).
  • PokerEvent: Reached 55.40%, proving competitive even against models that use both RGB frames and event data, despite EFV++ using events only.

Limitations and Future Work

The authors acknowledge a few hurdles for real-world deployment:

  • Complexity: Transforming event streams into two different representations and running both a Transformer and a GNN is computationally expensive.
  • Lack of Semantics: The current model focuses on raw signal patterns but ignores higher-level semantic categories that could help in reasoning.
  • Future Direction: The authors suggest using EFV++ as a "teacher" network to distill knowledge into a more lightweight "student" network that could run on edge devices or mobile hardware. They also plan to incorporate semantic cues into future architectures.

Summary updated

Abstract

Existing event stream-based pattern recognition models usually represent the event stream as the point cloud, voxel, image, etc., and design various deep neural networks to learn their features. Although considerable results can be achieved in simple cases, however, the model performance may be limited by monotonous modality expressions, sub-optimal fusion, and readout mechanisms. In this paper, we propose a novel dual-stream framework for event stream-based pattern recognition via differentiated fusion, termed EFV++. It models two common event representations simultaneously, i.e., event images and event voxels. The spatial and three-dimensional stereo information can be learned separately by utilizing Transformer and Graph Neural Network (GNN). We believe the features of each representation still contain both efficient and redundant features and a sub-optimal solution may be obtained if we directly fuse them without differentiation. Thus, we divide each feature into three levels and retain high-quality features, blend medium-quality features, and exchange low-quality features. The enhanced dual features will be fed into the fusion Transformer together with bottleneck features. In addition, we introduce a novel hybrid interaction readout mechanism to enhance the diversity of features as final representations. Extensive experiments demonstrate that our proposed framework achieves state-of-the-art performance on multiple widely used event stream-based classification datasets. Specifically, we achieve new state-of-the-art performance on the Bullying10k dataset, i.e., $90.51\%$, which exceeds the second place by $+2.21\%$. The source code of this paper has been released on \url{https://github.com/Event-AHU/EFV_event_classification/tree/EFVpp}.
BibTeX
@article{chen2025retain,
  author = {Lan Chen and Dong Li and Xiao Wang and Pengpeng Shao and Wei Zhang and Yaowei Wang and Yonghong Tian and Jin Tang},
  title = {{Retain, Blend, and Exchange: A Quality-Aware Spatial-Stereo Fusion Approach for Event Stream Recognition}},
  journal = {IEEE Trans. Multimedia},
  year = {2025},
  doi = {10.1109/tmm.2025.3607771},
}
Figure