Fast Weakly Supervised Action Segmentation Using Mutual Consistency

Yaser Souri, Mohsen Fayyaz, Luca Minciullo, Gianpiero Francesca, Juergen Gall

Abstract

Weakly supervised action segmentation typically suffers from slow inference due to exhaustive transcript searches. We present MuCon, a two-branch neural network that enforces mutual consistency between frame-wise and segment-level predictions. MuCon achieves state-of-the-art accuracy while being 20x faster at inference and 14x faster to train than previous methods by eliminating the need for computationally expensive alignment searches during testing.

Introduction

Action segmentation involves labeling every frame of a video with a specific action (e.g., "pouring milk"). While full supervision is labor-intensive, weakly supervised methods learn from "transcripts"—ordered lists of actions without timestamps. However, existing state-of-the-art methods face a major bottleneck: during inference, they must search through every transcript in the training set to find the best fit for a new video. This makes inference time scale poorly with dataset size. MuCon addresses this by creating a model that is both accurate and fast enough for real-world applications.

Methods: The MuCon Architecture

The proposed MuCon architecture shares a common Temporal Backbone (a Dilated Temporal Convolutional Network) that feeds into two distinct branches:

  1. Frame Classification Branch ($f_c$): Predicts the probability of each action for every individual frame.
  2. Segment Generation Branch ($f_s$): Uses a sequence-to-sequence (RNN) model with attention to predict the transcript and the relative length of each action.
The MuCon architecture featuring frame classification and segment generation branches.

Mutual Consistency (MuCon) Loss

To synchronize these branches, the authors introduce the MuCon loss. This loss forces the frame-wise predictions from the first branch to match the segments predicted by the second branch. To maintain differentiability for backpropagation, a Differentiable Mask Generator creates "soft" temporal masks (e.g., box, bell, or trapezoid shapes) based on predicted segment lengths. These masks act as gates, allowing the model to compare average frame-wise probabilities within a predicted segment against the intended action label.

Experimental Setup

The model was evaluated on two standard benchmarks:

  • Breakfast Dataset: Over 1,700 videos of 10 cooking activities with 48 fine-grained action classes.
  • Hollywood Extended Dataset: 937 video sequences from movies featuring 16 action classes.

Key Results

MuCon demonstrates that redundancy can serve as a powerful form of supervision. By predicting the same information in two formats (frame-wise vs. segment-wise), the branches provide mutual regularization.

Comparison of accuracy (MoF) vs. inference time across different models.

As shown in the performance analysis, the "MuCon-full" model is 20 times faster during inference than the previous state-of-the-art (CDFL) and 14 times faster to train. On the Breakfast dataset, MuCon achieved a Mean over Frames (MoF) accuracy of 48.5%, which is on par with CDFL (48.1%) and significantly better than other fast baselines like ISBA (36.4%). The model effectively avoids "hallucinating" actions because it infers the transcript directly from video features rather than searching a pre-defined list.

Conclusion

MuCon provides a scalable solution for action segmentation by directly predicting transcripts and enforcing consistency. While the model currently uses a Viterbi decoding step to fuse branches, it represents a significant leap in inference efficiency. Future directions include extending this mutual consistency framework to other sequence-to-sequence tasks like speech recognition or video captioning.

Summary updated

Abstract

Action segmentation is the task of predicting the actions for each frame of a video. As obtaining the full annotation of videos for action segmentation is expensive, weakly supervised approaches that can learn only from transcripts are appealing. In this paper, we propose a novel end-to-end approach for weakly supervised action segmentation based on a two-branch neural network. The two branches of our network predict two redundant but different representations for action segmentation and we propose a novel mutual consistency (MuCon) loss that enforces the consistency of the two redundant representations. Using the MuCon loss together with a loss for transcript prediction, our proposed approach achieves the accuracy of state-of-the-art approaches while being 14 times faster to train and 20 times faster during inference. The MuCon loss proves beneficial even in the fully supervised setting.
BibTeX
@article{souri2022fast,
  author = {Yaser Souri and Mohsen Fayyaz and Luca Minciullo and Gianpiero Francesca and Juergen Gall},
  title = {{Fast Weakly Supervised Action Segmentation Using Mutual Consistency}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2022},
  doi = {10.1109/tpami.2021.3089127},
}
Figure