Abstract
Weakly supervised action segmentation typically suffers from slow inference due to exhaustive transcript searches. We present MuCon, a two-branch neural network that enforces mutual consistency between frame-wise and segment-level predictions. MuCon achieves state-of-the-art accuracy while being 20x faster at inference and 14x faster to train than previous methods by eliminating the need for computationally expensive alignment searches during testing.
Introduction
Action segmentation involves labeling every frame of a video with a specific action (e.g., "pouring milk"). While full supervision is labor-intensive, weakly supervised methods learn from "transcripts"—ordered lists of actions without timestamps. However, existing state-of-the-art methods face a major bottleneck: during inference, they must search through every transcript in the training set to find the best fit for a new video. This makes inference time scale poorly with dataset size. MuCon addresses this by creating a model that is both accurate and fast enough for real-world applications.
Methods: The MuCon Architecture
The proposed MuCon architecture shares a common Temporal Backbone (a Dilated Temporal Convolutional Network) that feeds into two distinct branches:
- Frame Classification Branch ($f_c$): Predicts the probability of each action for every individual frame.
- Segment Generation Branch ($f_s$): Uses a sequence-to-sequence (RNN) model with attention to predict the transcript and the relative length of each action.
Mutual Consistency (MuCon) Loss
To synchronize these branches, the authors introduce the MuCon loss. This loss forces the frame-wise predictions from the first branch to match the segments predicted by the second branch. To maintain differentiability for backpropagation, a Differentiable Mask Generator creates "soft" temporal masks (e.g., box, bell, or trapezoid shapes) based on predicted segment lengths. These masks act as gates, allowing the model to compare average frame-wise probabilities within a predicted segment against the intended action label.
Experimental Setup
The model was evaluated on two standard benchmarks:
- Breakfast Dataset: Over 1,700 videos of 10 cooking activities with 48 fine-grained action classes.
- Hollywood Extended Dataset: 937 video sequences from movies featuring 16 action classes.
Key Results
MuCon demonstrates that redundancy can serve as a powerful form of supervision. By predicting the same information in two formats (frame-wise vs. segment-wise), the branches provide mutual regularization.
As shown in the performance analysis, the "MuCon-full" model is 20 times faster during inference than the previous state-of-the-art (CDFL) and 14 times faster to train. On the Breakfast dataset, MuCon achieved a Mean over Frames (MoF) accuracy of 48.5%, which is on par with CDFL (48.1%) and significantly better than other fast baselines like ISBA (36.4%). The model effectively avoids "hallucinating" actions because it infers the transcript directly from video features rather than searching a pre-defined list.
Conclusion
MuCon provides a scalable solution for action segmentation by directly predicting transcripts and enforcing consistency. While the model currently uses a Viterbi decoding step to fuse branches, it represents a significant leap in inference efficiency. Future directions include extending this mutual consistency framework to other sequence-to-sequence tasks like speech recognition or video captioning.
Summary updated