Unsupervised Action Segmentation via Fast Learning of Semantically Consistent Actoms

Zheng Xing, Weibing ZhaoAAAI 2024 · DOI

TL;DR

This paper introduces "SaM," a novel unsupervised framework for action segmentation in videos that learns by splitting videos into "actoms" (semantically consistent action units) and then merging them. It achieves state-of-the-art performance on several benchmarks without requiring any training data, making it significantly faster than existing methods.

What Problem Does This Paper Address?

Understanding the sequence of actions within a video is crucial for many applications like video retrieval and surveillance. This task, known as action segmentation, involves assigning labels to individual frames to delineate distinct actions. While supervised methods have made strides, they rely heavily on meticulously annotated data, which is time-consuming and expensive to create. Unsupervised methods aim to overcome this by learning without explicit frame-level labels, but they often struggle with accuracy or require extensive training. This paper tackles the challenge of achieving accurate and fast unsupervised action segmentation by proposing a method that avoids laborious annotation and lengthy training processes.

Methods

The core of the proposed "SaM" (Split-and-Merge) algorithm lies in its two-phase approach:

  1. Splitting: The algorithm first divides a video into distinct "actoms." It assumes that frames belonging to the same action reside in similar subspaces. To achieve this, it calculates the similarity between adjacent frames using a cosine-based measurement derived from subspace assumptions. Local minima in this similarity curve are then identified to pinpoint actom boundaries. A key innovation here is the use of a local minimum search algorithm with a sliding window to mitigate errors around actual action boundaries.
  2. Merging: Since the same action can occur multiple times, the initial actoms need to be grouped. The algorithm introduces a novel spatio-temporal similarity measure that considers both the feature-space similarity and the temporal proximity between actoms. This measure helps to merge actoms that represent the same action, even if they are separated in time. The merging process continues until the desired number of action segments (K) is achieved.

A significant advantage of SaM is its complete lack of a training phase, relying solely on the splitting and merging procedures.

Datasets

The proposed method was evaluated on four widely-used benchmark datasets for action segmentation:

  • Breakfast (BF): A dataset featuring diverse cooking activities.
  • YouTube Instructional Videos (YTI): Videos demonstrating various instructional tasks.
  • Hollywood Extended (HE): A dataset containing actions from movie clips.
  • 50Salads (FS): Another dataset focused on cooking activities, with varying granularities.

The paper does not introduce any new datasets.

Key Insights and Ideas

  • Actoms as Building Blocks: The concept of "actoms" – semantically consistent units of action – provides a powerful abstraction for segmenting videos.
  • Unsupervised Speed-Up: By eliminating the need for training, SaM achieves remarkable speed improvements, making it practical for real-world applications where rapid analysis is essential.
  • Spatio-Temporal Similarity is Key: The novel spatio-temporal similarity measure effectively captures both visual and temporal relationships between actoms, crucial for accurate merging.
  • Robustness to Parameters: The method demonstrates stable performance across a reasonable range of parameter values (δ and λ), indicating its robustness.

Key Results

SaM achieves state-of-the-art performance among unsupervised action segmentation methods on the evaluated datasets.

  • Performance: It outperforms existing unsupervised methods by significant margins on metrics like F1 and MoF. For instance, on the BF dataset, it achieves 44.4% IoU, 55.9% F1, and 64.0% MoF. On YTI, it reaches 49.6% F1 and 68.1% MoF. On HE, it shows a 5.7% improvement in MoF over the best unsupervised method. On FS, it achieves 78.2% F1 and 71.9% MoF (mid granularity).
  • Speed: The method is exceptionally fast, with a learning time of only 0.20 hours per video on the BF dataset, compared to hundreds of hours for some supervised methods and significantly faster than other unsupervised approaches like CTE.
  • Comparison to Supervised Methods: While not always surpassing fully supervised methods, SaM achieves comparable or near-comparable performance, highlighting its effectiveness without extensive annotation.
  • Ablation Studies: Both the splitting and merging steps are shown to be critical for the method's performance. Removing either step leads to a substantial drop in accuracy.

Limitations and Future Work

  • Parameter Sensitivity: While generally robust, the performance is sensitive to the choice of parameters δ (window size for splitting) and λ (trade-off parameter for merging). The paper suggests setting δ = 0.3 and λ = 0.001 based on experimental findings.
  • Potential for Merging Errors: A high value of λ, while increasing temporal consistency, could inadvertently merge actoms that represent distinct actions. The authors advise a balanced approach.
  • Future Work: Not explicitly detailed in the paper, but the success of this unsupervised, fast learning approach suggests potential for further exploration in more complex video understanding tasks and larger-scale datasets. The authors also mention the availability of their code for further research.

Summary updated

BibTeX
@inproceedings{xing2024unsupervised,
  author = {Zheng Xing and Weibing Zhao},
  title = {{Unsupervised Action Segmentation via Fast Learning of Semantically Consistent Actoms}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i6.28445},
}
Figure