Permutation-Aware Activity Segmentation via Unsupervised Frame-To-Segment Alignment

Quoc-Huy Tran, Ahmed Mehmood, Muhammad Ahmed, Muhammad Naufil, Anas Zafar, Andrey Konin, Zeeshan ZiaWACV 2024

TL;DR

This paper introduces UFSA, an unsupervised framework for temporal activity segmentation that leverages both frame-level and segment-level cues. By introducing a novel frame-to-segment alignment module and using permutation-aware pseudo-labels derived from estimated transcripts, UFSA achieves state-of-the-art performance on several challenging datasets, outperforming previous unsupervised methods.

What Problem Does This Paper Address?

Temporal activity segmentation aims to label each frame in a video with a corresponding action class. This is crucial for applications like video analysis, robotics, and healthcare. While supervised methods achieve high accuracy, they require extensive manual annotation (dense frame-level labels or precise timestamps), which is costly and time-consuming. Existing unsupervised methods often struggle with handling permutations of actions within a sequence, missing actions, or repetitive actions, as they primarily rely on frame-level information and fixed action order assumptions. This paper addresses the need for an unsupervised approach that can effectively segment activities while being robust to variations in action order.

Methods

The proposed framework, UFSA (Unsupervised Frame-to-Segment Alignment), builds upon transformer architectures. It consists of three main modules:

  • Frame-Level Prediction Module: This module uses a transformer encoder to extract frame-level features and predict framewise action classes. It's trained using temporal optimal transport (TOT) with a fixed-order prior, similar to prior work.
  • Segment-Level Prediction Module: This module introduces a transformer decoder to process estimated video transcripts and extract segment-level features. This allows the model to learn dependencies between video segments and the sequence of actions.
  • Frame-to-Segment Alignment Module: This novel module aligns frame-level features with segment-level features using cross-attention. This is key to refining framewise predictions and achieving permutation-aware segmentation.

A core innovation is the use of permutation-aware pseudo-labels for unsupervised training. Instead of relying on fixed action orders, the framework first estimates a transcript of actions from the frame-level predictions. This estimated transcript is then used to generate pseudo-labels for both the segment-level prediction and the frame-to-segment alignment modules, allowing them to learn from segment-level cues and handle action permutations. The training involves a combination of losses from these modules, optimized via temporal optimal transport.

Datasets

The authors evaluate their approach on four widely used datasets for temporal activity segmentation:

  • 50 Salads: Contains videos of salad preparation with two granularity levels (Eval and Mid).
  • YouTube Instructions (YTI): Features instructional videos with many background frames.
  • Breakfast: A large dataset of cooking activities.
  • Desktop Assembly: Includes videos with both fixed and permuted/missing action sequences.

Key Insights and Ideas

  • Segment-level cues are crucial for robust unsupervised segmentation: Relying solely on frame-level information and fixed action orders limits performance, especially when dealing with variations in action sequences.
  • Estimated transcripts enable permutation-awareness: By inferring a likely sequence of actions (transcript) from frame-level predictions, the model can learn to handle actions that appear out of a predefined order.
  • Frame-to-segment alignment refines predictions: Explicitly matching frame-level features with segment-level representations helps to correct over-segmentation and improve the accuracy of the final segmentation.
  • Unsupervised learning with pseudo-labels is effective: The proposed method demonstrates that generating pseudo-labels from intermediate predictions can effectively train complex modules without requiring ground truth annotations.

Key Results

UFSA achieves state-of-the-art performance on the evaluated datasets.

  • On 50 Salads (Eval granularity), UFSA achieved 55.8% MOF and 50.3% F1-Score, significantly outperforming previous methods like TOT [31] and UDE [54].
  • On YouTube Instructions (YTI), UFSA achieved 49.6% MOF and 32.4% F1-Score, leading in MOF and being second best in F1-Score.
  • On Breakfast, UFSA achieved 52.1% MOF and 38.0% F1-Score, performing competitively with methods that use action shuffling.
  • On Desktop Assembly, UFSA showed substantial improvements, particularly on the "Extra" set with permuted and missing steps, demonstrating its ability to handle complex action sequences.

Limitations and Future Work

  • Repetitive Actions: The paper acknowledges that handling repetitive actions is an interesting topic and remains a future work direction.
  • Transcript Estimation: While the transcript estimation is effective, the method tends to assign a small number of frames to missing actions, which can lead to minor impacts on overall accuracy.
  • Generalization: The authors evaluated generalization by splitting datasets into training and testing sets, showing UFSA's robustness. However, the exact splits used by other methods were not always available, leading to potential differences in reported results.

Future work could explore incorporating explicit mechanisms for handling repetitive actions and further refining the transcript estimation process.

Summary updated

BibTeX
@inproceedings{tran2024permutation,
  author = {Quoc-Huy Tran and Ahmed Mehmood and Muhammad Ahmed and Muhammad Naufil and Anas Zafar and Andrey Konin and Zeeshan Zia},
  title = {{Permutation-Aware Activity Segmentation via Unsupervised Frame-To-Segment Alignment}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure