Abstract
This paper addresses the challenge of labeling video frames with action classes under weak supervision, where only the temporal ordering of actions is known during training. We employ a Hidden Markov Model (HMM) grounded on a Gated Recurrent Unit (GRU) for frame labeling. Our primary contribution is the Constrained Discriminative Forward Loss (CDFL), which discriminates between the energy of all valid and invalid frame labelings. Unlike prior methods that rely on a single inferred segmentation, CDFL aggregates energy across all valid paths and contrasts them against likely but incorrect "hard" invalid paths. Our evaluation on the Breakfast Action, Hollywood Extended, and 50Salads datasets demonstrates that CDFL achieves superior results compared to the state of the art in both action segmentation and alignment.
Introduction
Action segmentation involves assigning an action label to every frame in a video. While fully supervised methods are effective, they require expensive manual annotation of start and end times for every action. Weakly supervised learning offers a more scalable alternative by using only "transcripts"—the sequence of actions—during training.
The core difficulty in weak supervision is the alignment problem: for a given sequence of actions, there are millions of possible ways to distribute them across video frames. Traditional approaches often attempt to find a single "pseudo ground truth" alignment. However, if this initial guess is inaccurate, the model learns from noise, leading to a "garbage in, garbage out" cycle that limits final accuracy.
Methods: The CDFL Framework
To overcome the limitations of single-path training, we propose an energy-based learning framework centered on the Constrained Discriminative Forward Loss (CDFL). We utilize an HMM-GRU architecture to predict frame-level action probabilities.
Segmentation Graph and Energy Estimation
Instead of selecting one alignment, we construct a fully-connected segmentation graph. This graph is anchored by an initial "Constrained Viterbi" pass but includes a neighborhood of alternative temporal boundaries ("cuts"). Every path through this graph represents a potential segmentation.
We define "valid" paths $\mathcal{P}^V$ as those matching the ground-truth transcript, and "invalid" paths $\mathcal{P}^I$ as those that violate it. The CDFL is calculated by:
- Aggregating Valid Energy: Using a recursive
logaddfunction to sum the energy of all valid paths. - Discriminative Learning: Penalizing "hard" invalid paths—those the model currently assigns low energy (high likelihood) despite being incorrect.
To handle the exponential number of possible paths, we developed a recursive algorithm that computes the CDFL in $O(\Delta^2KN)$ time, where $\Delta$ is the window size, $K$ is the number of actions, and $N$ is the number of frames. This ensures the method remains computationally feasible for long, complex videos.
Results and Discussion
We evaluated our approach on three major benchmarks: Breakfast Action, Hollywood Extended, and 50Salads. The CDFL approach consistently outperformed previous state-of-the-art (SOTA) methods in Mean-over-Frames (MoF) accuracy.
| Dataset | Previous SOTA (Method) | Our CDFL |
|---|---|---|
| Breakfast | 45.7% (D3TW) | 50.2% |
| Hollywood Ext | 33.6% (D3TW) | 45.0% |
| 50Salads | 49.4% (NN-Viterbi) | 54.7% |
Qualitative analysis (Figure 3) shows that the full CDFL formulation ($L_{CDF}$) produces segmentations that most closely align with the ground truth compared to simpler variants. By considering the entire distribution of valid segmentations, the model becomes robust to early training errors and learns to refine boundaries iteratively. The discriminative nature of the loss is particularly effective at pushing the model to distinguish between similar action classes by penalizing high-confidence incorrect predictions.
Conclusion
The Constrained Discriminative Forward Loss provides a robust framework for weakly supervised action segmentation. By moving beyond single-path pseudo-labels to a discriminative energy-based approach, we achieve significant gains in accuracy across multiple benchmarks. While the method relies on an initial anchor for the segmentation graph, its ability to refine boundaries and handle "hard" negatives marks a substantial step forward. Future work may explore extending this framework to even weaker forms of supervision, such as unordered action sets.
Summary updated