First-Person Action Recognition (FPAR)—understanding human activities from the perspective of a head-mounted camera—is a cornerstone of future human-robot collaboration. However, a significant gap exists between high-accuracy laboratory models and the practical requirements of wearable robotics. Most state-of-the-art models are "lab-bound," requiring high-end GPUs and operating on pre-trimmed video clips where action boundaries are already known. This paper introduces a framework to bridge this gap, enabling real-time, online action recognition on low-power edge devices.
The Challenge of the "Wild"
Deploying FPAR in real-world scenarios presents three major hurdles. First, systems must operate under strict hardware constraints, running on low-power "edge" chips to ensure portability and privacy. Second, they must handle continuous streams rather than discrete clips, identifying actions within a never-ending feed where activities overlap or are separated by background noise. Finally, they must overcome domain shift, where models trained in one environment (e.g., a specific kitchen) struggle to generalize to new layouts or lighting conditions.
Methodology: Streaming Inference and Temporal Aggregation
The authors propose a model-agnostic framework to repurpose existing architectures for real-world deployment. A key component is the move toward Streaming Inference. Instead of processing entire videos, the system uses a sliding window approach. The authors highlight MoViNet as an ideal architecture; it utilizes a "stream buffer" to maintain temporal context without the redundant computation required by standard 3D-CNNs like I3D.
To manage continuous video, the framework employs Dynamic Boundary Localization (DBL). By treating action changes as anomalies in the feature space, the system can detect when an action has likely finished. This allows for the automatic "cleaning" of the model's memory, preventing data from previous actions from polluting new predictions.
The "secret sauce" of the framework is the Two-Fold Aggregator ($A^2$). In fine-grained tasks like cooking, actions often overlap (e.g., stirring while reaching for a spice). The $A^2$ method uses two asynchronous aggregators. When a potential boundary is detected, the second aggregator begins "listening" for the next action while the first concludes, ensuring a smooth transition and improved accuracy during overlaps.
Key Results and Performance
The study evaluated these methods using the EPIC-Kitchens 55 dataset and a new EgoWild benchmark designed to test models under increasingly realistic constraints.
The results demonstrate a significant leap in edge efficiency. While a standard I3D model manages only 3 FPS on an NVIDIA Jetson Nano, the MoViNet-A0 configuration achieved 56 FPS. Crucially, this setup consumed an average of only 2.4W, making it viable for battery-powered wearable devices. The $A^2$ aggregator also outperformed previous state-of-the-art unsupervised boundary detection methods, particularly in "unseen" environments.
Conclusion and Future Outlook
While this work successfully enables high-speed online recognition on the edge, challenges remain. Domain shift continues to impact accuracy significantly, often resulting in a 20-30% drop in performance in new environments. Future work points toward continual learning, allowing models to adapt to new users or locations on the fly. Additionally, incorporating multi-modal inputs like audio or eye-tracking could further refine boundary detection with minimal additional computational cost.
Summary updated