Domain-Specific Priors and Meta Learning for Low-shot First-Person Action Recognition

Huseyin Coskun, Zeeshan Zia, Bugra Tekin, Federica Bogo, Nassir Navab, Federico Tombari, Harpreet S. Sawhney

Abstract

The lack of large-scale real datasets with annotations makes transfer learning a necessity for video activity understanding. We aim to develop an effective method for few-shot transfer learning for first-person action classification. We leverage independently trained local visual cues to learn representations that can be transferred from a source domain, which provides primitive action labels, to a different target domain using only a handful of examples. Visual cues we employ include object-object interactions, hand grasps and motion within regions that are a function of hand locations. We employ a framework based on meta-learning to extract the distinctive and domain invariant components of the deployed visual cues. This enables transfer of action classification models across public datasets captured with diverse scene and action configurations. We present comparative results of our transfer learning methodology and report superior results over state-of-the-art action classification approaches for both inter-class and inter-dataset transfer.

Introduction

Action recognition in first-person (egocentric) video, such as footage from head-mounted cameras or AR headsets, presents unique challenges. Unlike traditional third-person video, the camera is constantly in motion, and actions are often characterized by fine-grained interactions between hands and small objects. A primary hurdle in this field is data scarcity; training deep neural networks typically requires thousands of annotated examples, yet real-world applications—like assembly line monitoring or personalized assistants—often provide only a handful of samples. Furthermore, models frequently overfit to environmental backgrounds rather than the actions themselves, leading to poor generalization across different scenes.

Methodology

The proposed approach addresses these challenges through a two-pronged strategy: leveraging domain-specific visual priors and employing a robust meta-learning framework.

Domain-Specific Visual Cues

Rather than processing the entire video frame, the model is guided by "hard attention" mechanisms to focus on action-relevant regions. Several specialized cues are extracted using models trained on independent datasets:

  • Hand Detection & Trajectory: Localizing the hands to define the primary region of interest and tracking their movement.
  • Object Cells: A spatial grid centered on the hand to capture local object-object and hand-object interactions.
  • Grasp Features: Classifying the type of hand grip (e.g., power vs. precision), which serves as a strong indicator of the underlying action.
  • Motion Cues: Utilizing optical flow to capture the dynamics of the movement within hand-centric regions.
Overview of the proposed framework, illustrating the integration of visual cues like hand trajectory, grasps, and object cells into the representation and meta-learning stages.

Attentive Meta-Learning (A-MAML)

To enable rapid adaptation to new tasks with minimal data, the authors utilize meta-learning. They introduce Attentive MAML (A-MAML), an enhancement of the Model-Agnostic Meta-Learning algorithm. Standard MAML often exhibits instability when applied to Recurrent Neural Networks (RNNs) due to gradient issues. A-MAML mitigates this by focusing the "inner loop" of optimization specifically on a temporal attention layer. This creates a structured memory that effectively handles long-term dependencies in video sequences.

The temporal model architecture, featuring Bi-Directional GRUs and a self-attention mechanism, which is the focus of the A-MAML optimization.

Datasets and Evaluation

The framework was evaluated on two major benchmarks: EPIC Kitchens, a large-scale, long-tailed dataset of daily activities, and Extended GTEA Gaze+ (EGTEA), which was used to evaluate inter-dataset transfer performance.

Results and Discussion

The researchers tested the model in 1-shot and 5-shot scenarios, yielding several key insights:

  • Superior Transferability: The full model significantly outperformed standard 3D convolutional networks (e.g., I3D) and temporal networks (TRN) in few-shot settings.
  • A-MAML Performance: In 5-way, 5-shot classification, A-MAML achieved 41.4% accuracy on EPIC and 51.4% on EGTEA, consistently beating other popular few-shot methods like ProtoNet and TARN.
  • Scene Invariance: By decoupling actions from the background through hand-centric cues, the model successfully transferred knowledge across entirely different environments.
  • The Importance of Grasps: Grasp classification emerged as a critical component, often providing a more reliable signal for action recognition than the visual appearance of the objects themselves.
  • Synergy of Cues: The combination of Hand, Grasp, and Object Cell features provided the most robust representation for transferring knowledge to unseen action classes.

Conclusion

This work demonstrates that combining domain-specific priors with attentive meta-learning can overcome the data scarcity and scene-dependency issues inherent in egocentric action recognition. While the pipeline currently relies heavily on robust hand detection, future work could integrate gaze information or language-based priors to further enhance intent understanding in complex, multi-step activities.

Summary updated

Abstract

The lack of large-scale real datasets with annotations makes transfer learning a necessity for video activity understanding. We aim to develop an effective method for few-shot transfer learning for first-person action classification. We leverage independently trained local visual cues to learn representations that can be transferred from a source domain, which provides primitive action labels, to a different target domain using only a handful of examples. Visual cues we employ include object-object interactions, hand grasps and motion within regions that are a function of hand locations. We employ a framework based on meta-learning to extract the distinctive and domain invariant components of the deployed visual cues. This enables transfer of action classification models across public datasets captured with diverse scene and action configurations. We present comparative results of our transfer learning methodology and report superior results over state-of-the-art action classification approaches for both inter-class and inter-dataset transfer.
BibTeX
@article{coskun2022domain,
  author = {Huseyin Coskun and Zeeshan Zia and Bugra Tekin and Federica Bogo and Nassir Navab and Federico Tombari and Harpreet S. Sawhney},
  title = {{Domain-Specific Priors and Meta Learning for Low-shot First-Person Action Recognition}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2022},
  doi = {10.1109/tpami.2021.3058606},
}
Figure