Learning To Segment Actions From Visual and Language Instructions via Differentiable Weak Sequence Alignment

Yuhan Shen, Lu Wang, Ehsan ElhamifarCVPR 2021 · OpenReview

Abstract

We address the problem of unsupervised localization of key-steps and feature learning in instructional videos using both visual and language instructions. Our key observation is that the sequences of visual and linguistic key-steps are weakly aligned: there is an ordered one-to-one correspondence between most visual and language key-steps, while some key-steps in one modality are absent in the other. To recover the two sequences, we develop an ordered prototype learning module, which extracts visual and linguistic prototypes representing key-steps. On the other hand, to find weak alignment and perform feature learning, we develop a differentiable weak sequence alignment (DWSA) method that finds ordered one-to-one matching between sequences while allowing some items in a sequence to stay unmatched. We develop an efficient forward and backward algorithm for computing the alignment and the loss derivative with respect to parameters of visual and language feature learning modules. By experiments on two instructional video datasets, we show that our method significantly improves the state of the art.

Introduction

Learning from instructional videos, such as those found on YouTube, presents a significant challenge for unsupervised action segmentation. While these videos provide a rich source of multimodal data, the relationship between the visual performance of a task and its spoken narration is often "weakly aligned." A narrator might describe a step before performing it, or skip a step visually while explaining it in detail.

Visualizing the weak alignment between narrations and video segments across different tasks.

Existing methods often struggle with this noise, either assuming strict alignment or failing to account for "background" segments that contain no relevant key steps. This paper introduces a framework designed to handle these discrepancies by learning to identify and align key steps without human supervision.

Methodology

The proposed framework consists of two primary components: Soft Ordered Prototype Learning (SOPL) and Differentiable Weak Sequence Alignment (DWSA).

Soft Ordered Prototype Learning (SOPL)

SOPL serves as a specialized clustering module that extracts representative feature vectors, or "prototypes," for both visual and linguistic key steps. Unlike traditional clustering methods, SOPL enforces a temporal ordering constraint. This ensures the model learns the logical progression of a task (e.g., "unscrewing a wheel" must happen before "withdrawing the wheel").

Differentiable Weak Sequence Alignment (DWSA)

The core innovation of this work is the DWSA module. It finds an ordered, one-to-one matching between visual and linguistic sequences while explicitly allowing for unmatched items. This capability is crucial for handling instances where a step is mentioned but not shown, or vice versa.

Overview of the proposed framework, integrating visual and linguistic feature extraction with SOPL and DWSA.

By making the alignment process differentiable, the authors enable the model to use alignment error as a training signal. This allows the system to refine its underlying visual and textual feature representations through backpropagation, leading to more accurate segmentation over time.

Experimental Results

The model was evaluated on the ProceL and CrossTask datasets, which contain hundreds of hours of instructional footage. The results demonstrate a significant improvement over previous unsupervised state-of-the-art methods, with an F1 score increase of approximately 4.7%.

Key findings include:

  • Robustness to Noise: The model successfully identifies subtle steps that lack distinct physical actions but are mentioned in narrations.
  • Importance of Weak Alignment: Ablation studies confirmed that allowing for unmatched items (the "weak" aspect of DWSA) is essential; forcing matches significantly degrades performance.

Conclusion

This work provides a robust solution for unsupervised action segmentation by embracing the inherent messiness of instructional media. By combining ordered prototype learning with a differentiable alignment strategy that accounts for missing steps, the framework achieves a new state-of-the-art in localizing key actions within complex, real-world video data.

Summary updated

Abstract

We address the problem of unsupervised localization of key-steps and feature learning in instructional videos using both visual and language instructions. Our key observation is that the sequences of visual and linguistic key-steps are weakly aligned: there is an ordered one-to-one correspondence between most visual and language key-steps, while some key-steps in one modality are absent in the other. To recover the two sequences, we develop an ordered prototype learning module, which extracts visual and linguistic prototypes representing key-steps. On the other hand, to find weak alignment and perform feature learning, we develop a differentiable weak sequence alignment (DWSA) method that finds ordered one-to-one matching between sequences while allowing some items in a sequence to stay unmatched. We develop an efficient forward and backward algorithm for computing the alignment and the loss derivative with respect to parameters of visual and language feature learning modules. By experiments on two instructional video datasets, we show that our method significantly improves the state of the art.
BibTeX
@inproceedings{shen2021learning,
  author = {Yuhan Shen and Lu Wang and Ehsan Elhamifar},
  title = {{Learning To Segment Actions From Visual and Language Instructions via Differentiable Weak Sequence Alignment}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=2htkovEXjuo},
}
Figure