Uncertainty-aware Bridge based Mobile-Former Network for Event-based Pattern Recognition

Haoxiang Yang, Chengguo Yuan, Yabin Zhu, Lan Chen, Xiao Wang, Jin TangarXiv 2024 · DOI · arXiv

Abstract

This paper introduces a lightweight, hybrid neural network architecture specifically optimized for event-based cameras. By integrating the local feature extraction of MobileNet with the global contextual modeling of Transformers through a novel "uncertainty-aware bridge" (UA-Bridge), the model achieves state-of-the-art accuracy in pattern and gesture recognition. The design prioritizes efficiency for mobile applications while robustly managing the noise and sparsity inherent in asynchronous event streams.

Introduction

Traditional Human Activity Recognition (HAR) typically relies on standard RGB cameras, which face significant challenges in low-light environments, suffer from motion blur during rapid movements, and pose privacy risks. Event cameras, or Dynamic Vision Sensors (DVS), offer a compelling alternative by recording only pixel-level intensity changes. This results in high dynamic range, minimal motion blur, and inherently anonymized data.

However, processing sparse, asynchronous event data efficiently remains a challenge. Current models often struggle to balance local spatial features (typically handled by CNNs) with global relational context (handled by Transformers). Furthermore, event streams are often noisy, requiring a robust mechanism to handle uncertainty in the data.

Methodology

The authors propose a dual-branch architecture: the Uncertainty-aware Bridge based Mobile-Former Network.

Dual-Branch Architecture

The model operates two parallel paths:

  1. MobileNet Branch: Focuses on local feature mining to capture fine spatial details.
  2. Transformer Branch: Initialized with a small set of random tokens, this branch captures long-range global relations.
Overall architecture of the proposed Mobile-Former network with the UA-Bridge module.

The Uncertainty-Aware Bridge (UA-Bridge)

The core innovation is the UA-Bridge, which regulates information flow between the two branches. Unlike simple concatenation, the bridge uses Multi-Layer Perceptrons (MLPs) to predict a Gaussian distribution (mean $\mu$ and variance $\sigma$) for the features. By employing the re-parameterization trick, the model samples new features from this distribution, making the process differentiable. This allows the network to dynamically weight the reliability of information passing between the CNN and Transformer branches.

Information Fusion and Adaptation

Following sampling, a cross-attention mechanism merges the features. This ensures that Transformer tokens are updated with local context while CNN features are refined by global information. Additionally, the MobileNet branch utilizes Dynamic ReLU (DY-ReLU), which adapts based on global tokens from the Transformer, allowing the "big picture" to influence local feature filtering.

Experimental Evaluation

The model was evaluated on three benchmark event-based datasets: ASL-DVS (sign language), N-Caltech101 (object recognition), and DVS128-Gait-Day (human gait).

Key Results

The proposed model consistently outperformed existing state-of-the-art (SOTA) methods:

  • ASL-DVS: Achieved 99.9% accuracy, outperforming graph-based and LSTM models.
  • N-Caltech101: Reached 79.8% accuracy, a significant improvement over ResNet50 (+16.1%) and VMV-GCN (+2%).
  • DVS128-Gait-Day: Achieved 95.9% accuracy, proving effectiveness for complex temporal patterns.

Ablation Studies

Ablation studies confirmed that the UA-Bridge improved accuracy by nearly 2% over standard cross-attention. The researchers also optimized hyperparameters, such as the number of global tokens and embedding dimensions.

Ablation studies showing the impact of embedding dimensions (left) and the number of global tokens (right) on accuracy.

As shown in the ablation results, the model achieves peak performance with a specific configuration of embedding dimensions and a small number of tokens (6), balancing representational power with computational efficiency.

Conclusion

The Uncertainty-aware Bridge based Mobile-Former Network successfully bridges the gap between local and global feature extraction for event-based vision. By modeling information transfer as a probabilistic process, it handles the inherent uncertainty of sparse event data. While the model is highly efficient, future work may explore knowledge distillation to further enhance performance or extend the architecture to real-time object detection tasks.

Summary updated

Abstract

The mainstream human activity recognition (HAR) algorithms are developed based on RGB cameras, which are easily influenced by low-quality images (e.g., low illumination, motion blur). Meanwhile, the privacy protection issue caused by ultra-high definition (HD) RGB cameras aroused more and more people's attention. Inspired by the success of event cameras which perform better on high dynamic range, no motion blur, and low energy consumption, we propose to recognize human actions based on the event stream. We propose a lightweight uncertainty-aware information propagation based Mobile-Former network for efficient pattern recognition, which aggregates the MobileNet and Transformer network effectively. Specifically, we first embed the event images using a stem network into feature representations, then, feed them into uncertainty-aware Mobile-Former blocks for local and global feature learning and fusion. Finally, the features from MobileNet and Transformer branches are concatenated for pattern recognition. Extensive experiments on multiple event-based recognition datasets fully validated the effectiveness of our model. The source code of this work will be released at https://github.com/Event-AHU/Uncertainty_aware_MobileFormer.
BibTeX
@misc{yang2024uncertainty,
  author = {Haoxiang Yang and Chengguo Yuan and Yabin Zhu and Lan Chen and Xiao Wang and Jin Tang},
  title = {{Uncertainty-aware Bridge based Mobile-Former Network for Event-based Pattern Recognition}},
  year = {2024},
  eprint = {2401.11123},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.11123},
}
Figure