Introduction
Human-robot collaboration requires systems that can anticipate actions in real-time to ensure safety and efficiency. Traditional Human Activity Recognition (HAR) relies on standard video cameras, which face significant challenges in robotics: high latency due to frame processing, data redundancy in static scenes, and motion blur during fast movements. This research addresses these issues by leveraging neuromorphic "event cameras" to achieve early prediction of fine-grained manipulation actions.
Methodology
The system utilizes event cameras (silicon retinas) that only transmit signals when a pixel detects a change in brightness. This provides microsecond-level temporal resolution and naturally filters out static backgrounds, focusing purely on dynamic motion.
To process this asynchronous data, the researchers developed a hybrid architecture. Sparse events are converted into Time Surfaces, 2D representations where pixel values reflect the timing of recent activity through exponential decay. A MobileNet v2 backbone extracts spatial features from these surfaces, which are then fed into a Transformer Encoder. By employing causal self-attention, the model ensures predictions are based solely on past information, enabling continuous "online" inference. A stateful queue maintains efficiency by storing previous features, allowing the model to update predictions using only new data chunks.
Experimental Setup and Datasets
The study introduces the Event-based Manipulation Action Dataset (E-MAD), the first of its kind focused specifically on manipulation. It comprises 750 sequences of 30 distinct actions (e.g., pouring, carving, wiping) performed with six different objects. The model was also validated on standard benchmarks like the DVS 128 Gesture and SL-Animals-DVS datasets to demonstrate its generalizability across different types of motion.
Key Results and Early Prediction
The MobileNet Transformer architecture achieved state-of-the-art results, reaching 86.8% accuracy on the E-MAD task, outperforming 3D CNNs (77.3%) and LSTMs (81.2%). Notably, event-based models significantly outperformed video-based counterparts in direct comparisons, as video models often struggled to distinguish between different actions performed on the same object due to background interference.
A critical advantage of this approach is early prediction. Accuracy typically spikes the moment a hand touches an object (the "touching point"), but the model often begins building confidence during the reaching phase by identifying subtle kinematic cues. For many actions, the system reaches high confidence (>90%) well before the action is halfway completed. Furthermore, the system is highly efficient, processing over 4,700 Time Surfaces per second on standard hardware, far exceeding real-time requirements.
Conclusion and Future Work
While the system shows robust performance, future work will focus on increasing subject diversity and moving toward a purely event-by-event update mechanism to further minimize latency. Additionally, enhancing noise filtering for complex environments with non-human motion will be essential for deployment in industrial settings.
Summary updated