Domain-specific priors and meta learning for few-shot first-person action recognition

Harpreet S. Sawhney, Huseyin Coskun, Zeeshan Zia

TL;DR

This paper tackles the challenge of few-shot first-person action recognition, where models need to learn new actions from very few examples. They achieve this by leveraging domain-specific visual cues (like hand-object interactions and motion) and a meta-learning approach (Attentive MAML) to enable robust transfer learning across different datasets and action categories.

What Problem Does This Paper Address?

Understanding human actions from a first-person perspective (like what you see through a GoPro camera) is crucial for applications ranging from augmented reality training to assistive technologies. However, creating large, annotated datasets for this task is incredibly difficult and expensive. This scarcity of data makes it hard to train models that can generalize well, especially when we want them to recognize new actions with only a handful of examples (few-shot learning). Existing methods often struggle to transfer knowledge effectively between different datasets or even between different action categories within the same dataset due to the inherent variability in environments and object appearances.

Methods

The core of this work lies in two main components:

  • Domain-Specific Visual Cues: The authors propose using "visual cues" as strong priors to guide action recognition. These cues are designed to be informative for first-person actions and include:
  • Hand Detection: Using a hand detector to focus on the regions where hands are present, acting as a "hard attention" mechanism.
  • Local Hand Context: Extracting features from regions around the hands, assuming that actions are strongly implied by what's near the hands.
  • Object-Interaction Modeling: Representing interactions between objects and between hands and objects, even for unknown object classes.
  • Grasp Features: Using fine-grained grasp classification as a proxy for hand poses.
  • Optical Flow Features: Capturing motion information within hand regions.
  • Hand Trajectory Features: Encoding the past movement of hands. These cues are learned independently from various datasets and then combined.

  • Attentive Meta-Learning (A-MAML): To enable few-shot learning and transfer, the authors adapt the Model-Agnostic Meta-Learning (MAML) algorithm. Their proposed "Attentive MAML" (A-MAML) integrates an attention mechanism into the meta-learning framework. This allows the model to learn how to quickly adapt to new tasks (recognizing new actions with few examples) by focusing on relevant temporal information within the visual cues. They also found that optimizing only the attention layer parameters during meta-learning improved stability and performance.

Datasets

The paper utilizes several datasets for training and evaluation:

  • For Visual Cue Training:
    • COCO: Used for object detection to extract global and object cell features.
  • EPIC (12 Classes): Used for training optical flow and grasp estimation networks.
  • EgoHands: Used for training the hand detector.
  • GUN71: Used for training the grasp classification network.
  • For Action Recognition Evaluation:
    • EPIC-Kitchens (EPIC): A large-scale first-person action dataset with 55 hours of recordings and 125 primitive action labels. Used for both inter-class and intra-dataset transfer experiments.
  • Extended GTEA Gaze+ (EGTEA): Another first-person action dataset with 28 hours of recordings and 28 activity labels. Used for inter-dataset transfer experiments.

Key Insights and Ideas

  • Decoupling Action from Background is Key: The authors hypothesize that the strong coupling between foreground actions and background appearance in first-person videos is a major hurdle for transfer learning. By focusing on hand-centric visual cues, they effectively decouple the action from the background.
  • Leveraging Diverse, Unrelated Datasets: The ability to learn visual cues from datasets not specifically designed for action recognition (e.g., using object detection datasets for hand-object interactions) is crucial for overcoming data limitations.
  • Grasp Information is Highly Discriminative: The results consistently show that grasp features significantly contribute to action recognition, especially in few-shot scenarios, suggesting that how a hand interacts with an object is a strong indicator of the action.
  • Attention Enhances Meta-Learning: Integrating an attention mechanism into MAML (A-MAML) not only improves performance but also makes the meta-learning process more stable, a known challenge with MAML.
  • Hand as a "Hard Attention" Mechanism: Using hand detection to define regions of interest acts as a powerful way to focus the model's attention on the most relevant parts of the scene for action recognition.

Key Results

  • Superior Transfer Learning: The proposed A-MAML method, combined with the visual cues, significantly outperforms state-of-the-art baselines (I3D, TRN, Two-Stream TRN) on both inter-class and inter-dataset transfer tasks.
  • Inter-Class Transfer: The A-MAML approach achieved a substantial improvement (e.g., 7.5 percentage points) over baselines on the EPIC dataset for inter-class transfer.
  • Inter-Dataset Transfer: On the EGTEA dataset, the proposed method also showed strong performance, outperforming baselines in various few-shot settings. For instance, with 10 training samples, their method achieved around 57% accuracy, compared to around 45% for Two-Stream TRN.
  • Grasp Accuracy: Their grasp classification network achieved 42.32% accuracy on the GUN71 dataset, significantly outperforming a previous state-of-the-art method (22.67%).
  • Qualitative Success: The paper includes qualitative results showing correct predictions for actions like "Peel," "Scoop," and "Squeeze," and even in failure cases, the predictions still offer a plausible explanation of the scene.

Limitations and Future Work

  • Dataset Imbalance: The paper acknowledges the long-tail distribution of actions in real-world datasets, which poses a challenge for few-shot learning.
  • Model Stability: While A-MAML improves stability, meta-learning in general can be sensitive to hyperparameter choices and optimization difficulties.
  • Future Work: The authors suggest that their approach can be combined with other techniques, such as reducing intra-class variations (as in Baseline++), to further enhance performance. They also highlight the potential for their framework to be applied to more complex scenarios and a wider range of actions.

Summary updated

Abstract

The lack of large-scale real datasets with annotations makes transfer learning a necessity for video activity understanding. We aim to develop an effective method for few-shot transfer learning for first-person action classification. We leverage independently trained local visual cues to learn representations that can be transferred from a source domain, which provides primitive action labels, to a different target domain - using only a handful of examples. Visual cues we employ include object-object interactions, hand grasps and motion within regions that are a function of hand locations. We employ a framework based on meta-learning to extract the distinctive and domain invariant components of the deployed visual cues. This enables transfer of action classification models across public datasets captured with diverse scene and action configurations. We present comparative results of our transfer learning methodology …
BibTeX
@article{sawhney2022domain,
  author = {Harpreet S. Sawhney and Huseyin Coskun and Zeeshan Zia},
  title = {{Domain-specific priors and meta learning for few-shot first-person action recognition}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2022},
  doi = {10.1109/tpami.2021.3058606},
}
Figure