Hierarchical Modeling for Task Recognition and Action Segmentation in Weakly-Labeled Instructional Videos

Reza Ghoddoosian, Saif Sayed, Vassilis AthitsosWACV 2022

TL;DR

This paper introduces a novel two-stream framework for understanding instructional videos, simultaneously recognizing the overall task and segmenting fine-grained actions. By leveraging both semantic and temporal hierarchies, their method significantly outperforms existing approaches in task recognition and achieves state-of-the-art action segmentation with a dramatic reduction in inference time.

What Problem Does This Paper Address?

Instructional videos, like cooking tutorials or DIY guides, are a rich source of information for learning new skills. However, automatically understanding these videos is challenging. Specifically, this paper tackles two key problems:

  1. Task Recognition: Identifying the overall goal of the video (e.g., "making coffee").
  2. Action Segmentation: Pinpointing the start and end times of individual sub-actions that constitute the task (e.g., "pouring milk," "adding sugar").

A major hurdle is the annotation effort required for training. Fully supervised methods need precise frame-level labels for every action, which is time-consuming and expensive to obtain for large datasets. This paper focuses on weakly-labeled instructional videos, where only the ordered sequence of video-level actions (tasks) is provided during training, without specific start/end times for sub-actions. This "weak supervision" makes the problem significantly harder but more practical for real-world applications.

Methods

The core of the proposed solution is a two-stream hierarchical modeling framework for task recognition. This framework leverages two distinct hierarchical perspectives:

  • Semantic Hierarchy Stream (SHS): This stream models the relationship between the top-level task and lower-level attributes. These attributes can be the actual actions themselves or components like verb-object pairs (e.g., "take" and "cup"). The system learns to predict attribute scores from video features and then uses these scores to predict the task. This is beneficial because attributes are often shared across different tasks, allowing for more robust learning with limited task-specific data.
  • Temporal Hierarchy Stream (THS): This stream recognizes that tasks unfold in a sequence of stages. The video is divided into equal temporal segments (stages), and a classifier is trained for each stage. The system then learns an aggregation function to combine these stage-wise predictions into a final video-level task prediction. This helps distinguish tasks that might have visually similar actions but differ in their temporal progression.

These two streams are fused to produce a final task prediction. Crucially, the paper also proposes a top-down weakly-supervised action segmentation approach. Here, the predicted top-level task from the recognition framework is used to constrain the search space for action segmentation. This means the segmentation module only considers action sequences that are plausible for the predicted task, significantly improving accuracy and efficiency.

Datasets

The proposed methods were evaluated on two widely-used instructional video datasets:

  • Breakfast Dataset (BD): Contains around 1.7k untrimmed cooking videos with 48 action labels, demonstrating 10 breakfast dishes.
  • MPII Cooking 2 (C2): Features 201 training and 42 testing videos, offering longer durations and more fine-grained annotations (155 objects, 67 actions), presenting a greater challenge due to data imbalance and complexity.

Key Insights and Ideas

  • Hierarchy is Key: Exploiting both semantic (attributes) and temporal (stages) hierarchies is crucial for robust task recognition in instructional videos, especially under weak supervision.
  • Attributes Aid Generalization: Using shared attributes across tasks helps overcome the problem of limited data per task, leading to better generalization.
  • Temporal Stages Provide Complementary Information: Even if individual stage predictions are weak, their aggregation can lead to accurate task recognition, highlighting the importance of temporal structure.
  • Task Guides Segmentation: Using the predicted task to guide action segmentation is a powerful "top-down" approach that significantly improves accuracy and efficiency compared to purely "bottom-up" methods.
  • TF-IDF for Attribute Importance: Adapting TF-IDF weighting from text retrieval to model the discriminative power of attributes for specific tasks is an effective way to refine the semantic hierarchy.

Key Results

  • Task Recognition: The proposed two-stream hierarchical model significantly outperforms state-of-the-art methods on both the Breakfast and MPII Cooking 2 datasets, achieving t-acc scores of 80.04% and 86.36% respectively on the Breakfast dataset, and 45.24% and 54.49% on the MPII Cooking 2 dataset.
  • Action Segmentation: When combined with existing weakly-supervised segmentation methods (NNViterbi and CDFL), the proposed top-down approach achieves new state-of-the-art results on both datasets.
  • Efficiency: The top-down segmentation approach dramatically reduces inference time by 80-90% compared to previous methods.
  • Ablation Studies: The paper demonstrates that both the semantic and temporal streams contribute significantly to performance, and the task-wise switching gates for stream fusion yield the best results.

Limitations and Future Work

  • Attribute Selection: The paper uses verb/object components as attributes. Future work could explore more sophisticated methods for attribute discovery and selection.
  • Fusion Mechanisms: While task-wise switching gates perform best, other fusion mechanisms could be explored.
  • Generalizability: While tested on cooking and breakfast preparation, extending the framework to a wider variety of instructional tasks could be beneficial.
  • Fine-grained Temporal Modeling: The paper uses fixed-duration stages for the temporal hierarchy. More adaptive temporal modeling could further improve performance.

Summary updated

BibTeX
@inproceedings{ghoddoosian2022hierarchical,
  author = {Reza Ghoddoosian and Saif Sayed and Vassilis Athitsos},
  title = {{Hierarchical Modeling for Task Recognition and Action Segmentation in Weakly-Labeled Instructional Videos}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure