Abstract
The lack of large-scale real datasets with annotations makes transfer learning a necessity for video activity understanding. We aim to develop an effective method for few-shot transfer learning for first-person action classification. We leverage independently trained local visual cues to learn representations that can be transferred from a source domain, which provides primitive action labels, to a different target domain using only a handful of examples. Visual cues we employ include object-object interactions, hand grasps and motion within regions that are a function of hand locations. We employ a framework based on meta-learning to extract the distinctive and domain invariant components of the deployed visual cues. This enables transfer of action classification models across public datasets captured with diverse scene and action configurations. We present comparative results of our transfer learning methodology and report superior results over state-of-the-art action classification approaches for both inter-class and inter-dataset transfer.
Introduction
Action recognition in first-person (egocentric) video, such as footage from head-mounted cameras or AR headsets, presents unique challenges. Unlike traditional third-person video, the camera is constantly in motion, and actions are often characterized by fine-grained interactions between hands and small objects. A primary hurdle in this field is data scarcity; training deep neural networks typically requires thousands of annotated examples, yet real-world applications—like assembly line monitoring or personalized assistants—often provide only a handful of samples. Furthermore, models frequently overfit to environmental backgrounds rather than the actions themselves, leading to poor generalization across different scenes.
Methodology
The proposed approach addresses these challenges through a two-pronged strategy: leveraging domain-specific visual priors and employing a robust meta-learning framework.
Domain-Specific Visual Cues
Rather than processing the entire video frame, the model is guided by "hard attention" mechanisms to focus on action-relevant regions. Several specialized cues are extracted using models trained on independent datasets:
- Hand Detection & Trajectory: Localizing the hands to define the primary region of interest and tracking their movement.
- Object Cells: A spatial grid centered on the hand to capture local object-object and hand-object interactions.
- Grasp Features: Classifying the type of hand grip (e.g., power vs. precision), which serves as a strong indicator of the underlying action.
- Motion Cues: Utilizing optical flow to capture the dynamics of the movement within hand-centric regions.
Attentive Meta-Learning (A-MAML)
To enable rapid adaptation to new tasks with minimal data, the authors utilize meta-learning. They introduce Attentive MAML (A-MAML), an enhancement of the Model-Agnostic Meta-Learning algorithm. Standard MAML often exhibits instability when applied to Recurrent Neural Networks (RNNs) due to gradient issues. A-MAML mitigates this by focusing the "inner loop" of optimization specifically on a temporal attention layer. This creates a structured memory that effectively handles long-term dependencies in video sequences.
Datasets and Evaluation
The framework was evaluated on two major benchmarks: EPIC Kitchens, a large-scale, long-tailed dataset of daily activities, and Extended GTEA Gaze+ (EGTEA), which was used to evaluate inter-dataset transfer performance.
Results and Discussion
The researchers tested the model in 1-shot and 5-shot scenarios, yielding several key insights:
- Superior Transferability: The full model significantly outperformed standard 3D convolutional networks (e.g., I3D) and temporal networks (TRN) in few-shot settings.
- A-MAML Performance: In 5-way, 5-shot classification, A-MAML achieved 41.4% accuracy on EPIC and 51.4% on EGTEA, consistently beating other popular few-shot methods like ProtoNet and TARN.
- Scene Invariance: By decoupling actions from the background through hand-centric cues, the model successfully transferred knowledge across entirely different environments.
- The Importance of Grasps: Grasp classification emerged as a critical component, often providing a more reliable signal for action recognition than the visual appearance of the objects themselves.
- Synergy of Cues: The combination of Hand, Grasp, and Object Cell features provided the most robust representation for transferring knowledge to unseen action classes.
Conclusion
This work demonstrates that combining domain-specific priors with attentive meta-learning can overcome the data scarcity and scene-dependency issues inherent in egocentric action recognition. While the pipeline currently relies heavily on robust hand detection, future work could integrate gaze information or language-based priors to further enhance intent understanding in complex, multi-step activities.
Summary updated