Embedding Task Structure for Action Detection

Michael Peven, Gregory D. HagerWACV 2024

TL;DR

This paper introduces a novel method to improve action detection in videos by embedding prior knowledge about the relationships between actions into the model's loss function. By leveraging hierarchical structures (like "what's a type of what") and temporal sequences (like "what usually happens after what"), their approach leads to more accurate and robust action detection, achieving state-of-the-art results on several benchmarks.

What Problem Does This Paper Address?

Action detection, the task of identifying and localizing specific actions within a video, is crucial for understanding human behavior in videos. Traditional methods often rely on deep neural networks trained with a simple cross-entropy loss function. This loss treats all actions as independent, failing to capture inherent relationships. For instance, "peeling a carrot" and "cutting a carrot" are visually and temporally related, but a standard loss function would treat them as equally distinct from "washing hands." As videos become longer and more complex (moving from image classification to activity recognition and then to action detection), abstracting these high-level relationships directly from data becomes increasingly challenging. This paper aims to bridge this gap by explicitly incorporating known structural information into the training process.

Methods

The core of this paper's contribution lies in modifying the loss function to embed structural information. Instead of a simple one-hot encoding for ground truth labels, they propose using a "distance" metric that reflects the relationships between actions.

  • Hierarchical Structure (Tree-based): For datasets with hierarchical labels (e.g., a taxonomy of actions), they define a distance based on the path length in the hierarchy. This means actions that share a parent or grandparent in the hierarchy are considered "closer" than unrelated actions, reducing the penalty for slightly incorrect predictions.
  • Temporal Structure (Sequence-based): They model the temporal ordering of actions by estimating transition probabilities from the dataset. This is done using a Poisson point process to capture how often one action follows another, with fewer intervening actions indicating a closer temporal relationship.
  • Combined Loss: These structural distances are then integrated into the cross-entropy loss. The modified loss function penalizes incorrect predictions less severely if the predicted action is structurally or temporally close to the ground truth action. This encourages the model to learn more nuanced relationships beyond simple classification.

Datasets

The authors evaluate their method on four distinct datasets, chosen for their varied structures:

  • Activity Recognition Datasets:
    • FineGym [58]: Contains gymnastics events with a three-layer hierarchy.
  • Berkeley MHAD [50]: Features actors performing actions with a three-layer hierarchy based on motion.
  • Action Detection Datasets:
    • IKEA ASM [4]: Videos of furniture assembly, with structure defined by objects and verbs.
  • EPIC-Kitchens-100 [10]: Egocentric videos of kitchen activities, with two three-layer hierarchies for {activity, verb, verb-category} and {activity, noun, noun-category}.

Key Insights and Ideas

  • Structure is Implicitly Learned, but Explicitly Embedded is Better: While deep networks can learn latent relationships from large datasets, explicitly embedding known structures (hierarchical, temporal) into the loss function provides a significant boost, especially when data might be limited or relationships are complex.
  • Loss Function as a Knowledge Injector: The paper demonstrates that the loss function is a powerful mechanism to inject prior knowledge into neural networks, guiding their learning process more effectively than relying solely on data.
  • Data Dependence of Structure Benefits: The effectiveness of embedding different types of structure (e.g., object vs. verb hierarchies) can vary depending on the dataset. For instance, if object distinctions are more salient than verb distinctions in a dataset, embedding object structure might yield better results.
  • Beyond Accuracy: Edit Distance Matters: For action detection, not only is frame-wise accuracy important, but the temporal ordering and smoothness of predictions are also critical. The proposed method significantly improves edit distance, indicating more temporally coherent and accurate action sequences.

Key Results

  • State-of-the-Art Performance: The proposed method achieves state-of-the-art results on multiple benchmarks for both activity recognition and action detection.
  • Improved Accuracy: For activity recognition, embedding hierarchical structure led to improvements in accuracy, with the best results achieved by considering child-parent relationships.
  • Significant Edit Distance Reduction: In action detection, the method led to substantial improvements in edit distance (e.g., a reduction of 2.0% to 10.3% on IKEA-ASM), indicating more temporally accurate predictions.
  • Efficiency: The authors emphasize that their approach is computationally efficient, allowing for improvements without requiring significantly larger models or more training data.

Limitations and Future Work

  • Parameter Sensitivity: The effectiveness of the distance weighting parameters can be dataset-dependent, requiring careful selection (as shown in Figure 3).
  • Temporal Structure Effectiveness: While visual structure consistently improved performance, the impact of temporal structure on the sequence model was less pronounced when the backbone was already well-trained with visual structure. The authors hypothesize this is because the TSM backbone already captures short-term temporal information.
  • Future Work: The authors suggest that their framework is generalizable and can be applied to various backbone architectures and sequence models. Future work could explore more complex graph structures beyond simple trees and investigate how to automatically discover optimal structure representations.

Summary updated

BibTeX
@inproceedings{peven2024embedding,
  author = {Michael Peven and Gregory D. Hager},
  title = {{Embedding Task Structure for Action Detection}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure