Abstract
This paper introduces a lightweight, hybrid neural network architecture specifically optimized for event-based cameras. By integrating the local feature extraction of MobileNet with the global contextual modeling of Transformers through a novel "uncertainty-aware bridge" (UA-Bridge), the model achieves state-of-the-art accuracy in pattern and gesture recognition. The design prioritizes efficiency for mobile applications while robustly managing the noise and sparsity inherent in asynchronous event streams.
Introduction
Traditional Human Activity Recognition (HAR) typically relies on standard RGB cameras, which face significant challenges in low-light environments, suffer from motion blur during rapid movements, and pose privacy risks. Event cameras, or Dynamic Vision Sensors (DVS), offer a compelling alternative by recording only pixel-level intensity changes. This results in high dynamic range, minimal motion blur, and inherently anonymized data.
However, processing sparse, asynchronous event data efficiently remains a challenge. Current models often struggle to balance local spatial features (typically handled by CNNs) with global relational context (handled by Transformers). Furthermore, event streams are often noisy, requiring a robust mechanism to handle uncertainty in the data.
Methodology
The authors propose a dual-branch architecture: the Uncertainty-aware Bridge based Mobile-Former Network.
Dual-Branch Architecture
The model operates two parallel paths:
- MobileNet Branch: Focuses on local feature mining to capture fine spatial details.
- Transformer Branch: Initialized with a small set of random tokens, this branch captures long-range global relations.
The Uncertainty-Aware Bridge (UA-Bridge)
The core innovation is the UA-Bridge, which regulates information flow between the two branches. Unlike simple concatenation, the bridge uses Multi-Layer Perceptrons (MLPs) to predict a Gaussian distribution (mean $\mu$ and variance $\sigma$) for the features. By employing the re-parameterization trick, the model samples new features from this distribution, making the process differentiable. This allows the network to dynamically weight the reliability of information passing between the CNN and Transformer branches.
Information Fusion and Adaptation
Following sampling, a cross-attention mechanism merges the features. This ensures that Transformer tokens are updated with local context while CNN features are refined by global information. Additionally, the MobileNet branch utilizes Dynamic ReLU (DY-ReLU), which adapts based on global tokens from the Transformer, allowing the "big picture" to influence local feature filtering.
Experimental Evaluation
The model was evaluated on three benchmark event-based datasets: ASL-DVS (sign language), N-Caltech101 (object recognition), and DVS128-Gait-Day (human gait).
Key Results
The proposed model consistently outperformed existing state-of-the-art (SOTA) methods:
- ASL-DVS: Achieved 99.9% accuracy, outperforming graph-based and LSTM models.
- N-Caltech101: Reached 79.8% accuracy, a significant improvement over ResNet50 (+16.1%) and VMV-GCN (+2%).
- DVS128-Gait-Day: Achieved 95.9% accuracy, proving effectiveness for complex temporal patterns.
Ablation Studies
Ablation studies confirmed that the UA-Bridge improved accuracy by nearly 2% over standard cross-attention. The researchers also optimized hyperparameters, such as the number of global tokens and embedding dimensions.
As shown in the ablation results, the model achieves peak performance with a specific configuration of embedding dimensions and a small number of tokens (6), balancing representational power with computational efficiency.
Conclusion
The Uncertainty-aware Bridge based Mobile-Former Network successfully bridges the gap between local and global feature extraction for event-based vision. By modeling information transfer as a probabilistic process, it handles the inherent uncertainty of sparse event data. While the model is highly efficient, future work may explore knowledge distillation to further enhance performance or extend the architecture to real-time object detection tasks.
Summary updated