TL;DR
This paper introduces two novel learning methods, Hand-Object Contact Learning (HOCL) and Object State Learning (OSL), to improve egocentric action recognition by explicitly modeling hand-object interactions and object state changes. These methods enhance performance across various datasets and model architectures, demonstrating the importance of fine-grained interaction details for understanding human actions from a first-person perspective.
What Problem Does This Paper Address?
Egocentric action recognition (EAR) aims to understand human actions from a first-person viewpoint, which has applications in areas like worker safety and healthcare. While deep learning models have advanced EAR, they often struggle to capture the crucial nuances of human-object interactions. Standard approaches that train models using only video-action label pairs can lead to models that focus on irrelevant visual cues, resulting in suboptimal performance. The core challenge is to enable EAR models to learn actions by focusing on how hands interact with objects and how those objects change state as a result of the action.
Methods
The paper proposes two complementary learning methods, HOCL and OSL, integrated into the training phase of an end-to-end EAR model.
- Hand-Object Contact Learning (HOCL): This method uses a "deep mutual learning" approach with two models. One model (VRM1) learns from raw videos, while the other (VRM2) learns from masked videos where only hand-object regions are visible (extracted by a hand-object detector). By minimizing the KL divergence between their predictions, VRM1 is encouraged to focus on hand-object contact.
- Object State Learning (OSL): This method introduces a frame-level object state prediction task. The model learns to predict the state of an object (e.g., "open" or "closed" for a fridge) in each frame, alongside the action classification. This is achieved by defining pseudo-object state distributions based on manually annotated initial and final object states for actions. A loss function encourages the model to align its predictions with these ground truth states.
During inference, only VRM1 is used, which has learned to integrate information about actions, hand-object contact, and object state changes.
Datasets
The proposed methods were evaluated on three datasets:
- EGTEA [31]: Contains kitchen-related actions. The authors manually annotated initial and final object states for 106 actions.
- MECCANO [43]: Features industrial-like domain actions. Initial and final states for 61 actions were annotated.
- EPIC-KITCHENS 100 (EPIC-100) [9]: A large-scale dataset with verb-noun action labels. Initial and final states were annotated for verb-noun pairs.
Key Insights and Ideas
- Beyond Simple Video-Label Pairs: The paper strongly argues that standard EAR training, relying solely on video-action labels, is insufficient. It highlights that models need to explicitly learn how interactions occur, not just what action is performed.
- The Power of Mutual Learning for Contact: Using two models, one with full visual input and another with masked input focusing on hands and objects, is an effective way to force the primary model to learn fine-grained hand-object contact information.
- Object State as a Crucial Cue: Recognizing that actions often involve changing an object's state (e.g., opening a door) provides a powerful signal for action recognition. Explicitly modeling this state change improves understanding.
- Architectural Affinity: The authors observe that HOCL seems to benefit CNN-based models like SlowFast more, while OSL shows stronger gains with transformer-based models like Swin-B. This suggests a potential synergy between the learning constraints and the inherent strengths of different architectures (local vs. global feature learning).
Key Results
- Overall Accuracy Gains: The proposed methods improved overall accuracy by up to 2.24% on EGTEA, 3.97% on MECCANO, and 1.49% on EPIC-KITCHENS 100 when applied to SlowFast and Swin-B models.
- Robustness: HOCL and OSL demonstrated improved performance on data with small training samples and on unfamiliar scenes, indicating increased robustness.
- Qualitative Evidence: GradCAM visualizations showed that models trained with HOCL and OSL focused on relevant hand-object contact regions and object state changes, unlike baseline models that focused on irrelevant areas or made incorrect predictions. For instance, a "Close cabinet" action was correctly identified as "Open cabinet" when the proposed methods were used.
Limitations and Future Work
- Annotation Requirement: The primary limitation is the need for manual annotation of object states (initial and final) for OSL. While the cost is proportional to the number of action labels, not videos, it's still an additional burden. The authors suggest that future work should explore unsupervised or self-supervised methods to overcome this.
- Salient Action Identification: Current models struggle to determine the most salient action when multiple actions occur simultaneously in a video (e.g., picking up bread and a knife at the same time). Future research could address this by developing methods to identify and prioritize salient actions.
Summary updated