SVIP: Sequence VerIfication for Procedures in Videos

Yicheng Qian, Weixin Luo, Dongze Lian, Xu Tang, Peilin Zhao, Shenghua Gao

TL;DR

SVIP introduces a new computer vision task called Sequence Verification, which determines if two videos perform the exact same series of procedural steps. By combining a Transformer-based architecture with a novel Sequence Alignment Loss, the authors can detect subtle procedural errors—like skipped, added, or swapped steps—even in tasks the model has never seen before.

What Problem Does This Paper Address?

Traditional video understanding focuses on two main areas: Action Recognition (identifying what is happening, like "changing a tire") and Action Detection (finding when specific actions occur). However, these tasks often fail to capture the nuance of a multi-step procedure.

Comparison of Action Recognition, Action Detection, and Sequence Verification tasks.

For example, if a student performs a chemistry experiment but skips a crucial cleaning step or adds chemicals in the wrong order, a standard action recognition model might still classify the video as the correct experiment. There is a significant gap in "open-set" scenarios where we need to verify if a procedure is correct without having predefined labels for every possible mistake. The challenge lies in distinguishing between "positive" pairs (identical procedures) and "negative" pairs (same task, but with step-level transformations).

Methods

The authors propose a baseline model called the CosAlignment Transformer (CAT). Its architecture consists of three main modules:

  1. Intra-step Module: Uses a 2D backbone (ResNet-50) to extract feature maps from sampled frames.
  2. Inter-step Module: Employs a Transformer Encoder to model the long-term temporal dependencies between different steps in the video. Unlike standard models that use global average pooling (which loses order information), CAT preserves the temporal dimension to remain sensitive to step sequences.
  3. Alignment Module (The "Secret Sauce"): To ensure the model truly understands the sequence, the authors introduce a Sequence Alignment Loss ($L_{seq}$). It calculates a cosine similarity correlation matrix between two videos. For a positive pair, the loss enforces that the diagonal of this matrix (matching steps at matching times) should be close to 1, while off-diagonal elements should be close to 0.

Datasets

The paper introduces three benchmarks tailored for sequence verification:

  • COIN-SV: A reorganization of the COIN dataset (instructional videos like "replacing a faucet"). It contains 36 tasks and over 2,000 videos.
  • Diving48-SV: A reorganization of the Diving48 dataset, focusing on the highly structured sequences of competitive diving.
  • CSV (Chemical Sequence Verification): A newly collected scripted dataset. It features 1,940 videos of chemical experiments where volunteers followed specific scripts that intentionally included "errors" such as deletions, additions, and order swaps. This provides a "ground truth" for procedural mistakes that web-scraped data lacks.
Examples of procedural sequences from the COIN-SV, Diving48-SV, and CSV datasets.

Key Insights and Ideas

  • Verification as an Open-Set Solution: By treating this as a verification task (comparing two embeddings) rather than a classification task, the model can be applied to new, unseen procedures without retraining.
  • Order Matters: The authors demonstrate that standard action recognition models are "order-insensitive." CAT proves that explicitly modeling the correlation between steps is vital for catching sequence-level errors.
  • Weighted Distance Ratio (WDR): The authors argue that standard AUC metrics can be misleading because some procedural errors are "more wrong" than others (e.g., swapping two steps vs. skipping five). They introduce WDR, which uses the Levenshtein distance (edit distance) to normalize the embedding distance, ensuring a fairer evaluation across different types of procedural transformations.

Key Results

The CAT model consistently outperformed established action recognition baselines (like TSN, TSM, and Video Swin Transformer) across all datasets:

  • CSV Dataset: CAT achieved an AUC of 83.02%, significantly higher than TSN (59.85%) and TSM (62.38%).
  • Ablation Success: The Sequence Alignment (SA) module was particularly effective at catching swapped steps. On the "alter-order" split of the CSV dataset, adding the SA module improved the AUC by over 6%.
  • Visual Proof: PCA visualizations of the embeddings show that CAT creates much tighter clusters for identical procedures and clearer separation between procedures that differ only by step order.

Limitations and Future Work

  • Data Hunger: The authors noted that the Video Swin Transformer performed poorly, likely because these specialized verification datasets are still relatively small compared to the massive datasets required to train large-scale video transformers effectively.
  • Background Noise: Performance on the COIN-SV dataset was lower (51.13% AUC) due to high background diversity and "in-the-wild" recording conditions, suggesting that better background-foreground separation is needed.
  • Real-time Application: While the paper focuses on "off-line" verification (comparing two finished videos), the authors suggest that Online Sequence Verification (providing an "early warning" as a mistake happens) is a promising but difficult future direction.

Summary updated

Abstract

In this paper, we propose a novel sequence verification task that aims to distinguish positive video pairs performing the same action sequence from negative ones with step-level transformations but still conducting the same task. Such a challenging task resides in an open-set setting without prior action detection or segmentation that requires event-level or even frame-level annotations. To that end, we carefully reorganize two publicly available action-related datasets with step-procedure-task structure. To fully investigate the effectiveness of any method, we collect a scripted video dataset enumerating all kinds of step-level transformations in chemical experiments. Besides, a novel evaluation metric Weighted Distance Ratio is introduced to ensure equivalence for different step-level transformations during evaluation. In the end, a simple but effective baseline based on the transformer encoder with a novel sequence alignment loss is introduced to better characterize long-term dependency between steps, which outperforms other action recognition methods. Codes and data will be released <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> :
BibTeX
@inproceedings{qian2021svip,
  author = {Yicheng Qian and Weixin Luo and Dongze Lian and Xu Tang and Peilin Zhao and Shenghua Gao},
  title = {{SVIP: Sequence VerIfication for Procedures in Videos}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr52688.2022.01927},
}
Figure