Weakly Supervised Action Segmentation Using Mutual Consistency

Yaser Souri, Mohsen Fayyaz, Juergen Gall

Abstract

Action segmentation is the task of predicting the actions for each frame of a video. As obtaining the full annotation of videos for action segmentation is expensive, weakly supervised approaches that can learn only from transcripts are appealing. In this paper, we propose a novel end-to-end approach for weakly supervised action segmentation based on a two-branch neural network. The two branches of our network predict two redundant but different representations for action segmentation and we propose a novel mutual consistency (MuCon) loss that enforces the consistency of the two redundant representations. Using the MuCon loss together with a loss for transcript prediction, our proposed approach achieves the accuracy of state-of-the-art approaches while being 14 times faster to train and 20 times faster during inference. The MuCon loss proves beneficial even in the fully supervised setting.

Introduction

Action segmentation—identifying what is happening in every single frame of a video—is incredibly labor-intensive to annotate. To avoid this "annotation bottleneck," researchers use weak supervision, where the model is only given a transcript (e.g., "crack egg," then "fry egg," then "plate egg") without any timing information.

The major challenge with existing weakly supervised methods is efficiency. Most rely on the Viterbi algorithm during inference to align a test video against every transcript seen during training to find the best fit. As the training set grows, inference becomes agonizingly slow. This paper introduces a novel two-branch neural network that learns from simple transcripts and achieves state-of-the-art accuracy while being significantly faster to train and deploy.

Methods

The proposed architecture shares a common temporal backbone based on a Dilated Temporal Convolutional Network. It splits into two distinct branches:

  1. Frame Classification Branch ($f_c$): This branch predicts the probability of each action class for every individual frame.
  2. Segment Generation Branch ($f_s$): This is a Sequence-to-Sequence (Seq2Seq) model with attention. Instead of looking at frames, it predicts a sequence of action labels and their corresponding temporal lengths.
The two-branch architecture showing the shared backbone and the Mutual Consistency loss.

The core innovation is the Mutual Consistency (MuCon) loss. Because the two branches represent the same video in different ways (frame-wise vs. segment-wise), they should ideally agree. To make this work, the authors developed a differentiable mask generator. This allows the model to map the predicted segments back onto the frame timeline using various shapes (box, bell, or trapezoid) so the two branches can supervise each other during training. This "mutual supervision" acts as a powerful regularizer, even helping in fully supervised settings.

Results and Discussion

The model was evaluated on two standard benchmarks: the Breakfast Dataset (1,700+ videos, 48 classes) and the Hollywood Extended Dataset (937 videos, 16 classes).

The proposed "MuCon-full" approach provides the best trade-off between speed and accuracy in the field. On the Breakfast dataset, it achieves a Mean over Frames (MoF) accuracy of 48.5%, which is on par with the most accurate (but much slower) methods. Crucially, it is 20x faster during inference than the previous state-of-the-art (CDFL) and 14x faster to train.

Inference time vs. MoF accuracy, highlighting the superior trade-off of the MuCon approach.

Unlike previous methods that search through all training transcripts at test time, this model directly predicts the transcript for a new video. This decouples inference speed from the size of the training set, making it viable for real-world applications.

Conclusion

The MuCon framework demonstrates that enforcing consistency between redundant representations is a powerful tool for weakly supervised learning. While there remains a gap between weakly and fully supervised performance, the efficiency gains are substantial. Future work could involve extending this framework to even weaker forms of supervision, such as unordered sets of actions, or applying it to other sequence-based tasks like speech recognition.

Summary updated

Abstract

Action segmentation is the task of predicting the actions for each frame of a video. As obtaining the full annotation of videos for action segmentation is expensive, weakly supervised approaches that can learn only from transcripts are appealing. In this paper, we propose a novel end-to-end approach for weakly supervised action segmentation based on a two-branch neural network. The two branches of our network predict two redundant but different representations for action segmentation and we propose a novel mutual consistency (MuCon) loss that enforces the consistency of the two redundant representations. Using the MuCon loss together with a loss for transcript prediction, our proposed approach achieves the accuracy of state-of-the-art approaches while being $14$ times faster to train and $20$ times faster during inference. The MuCon loss proves beneficial even in the fully supervised setting.
BibTeX
@article{souri2022weakly,
  author = {Yaser Souri and Mohsen Fayyaz and Juergen Gall},
  title = {{Weakly Supervised Action Segmentation Using Mutual Consistency}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2022},
  doi = {10.1109/tpami.2021.3089127},
}
Figure