Weakly Supervised Energy-Based Learning for Action Segmentation

Jun Li, Peng Lei, Sinisa Todorovic

Abstract

This paper addresses the challenge of labeling video frames with action classes under weak supervision, where only the temporal ordering of actions is known during training. We employ a Hidden Markov Model (HMM) grounded on a Gated Recurrent Unit (GRU) for frame labeling. Our primary contribution is the Constrained Discriminative Forward Loss (CDFL), which discriminates between the energy of all valid and invalid frame labelings. Unlike prior methods that rely on a single inferred segmentation, CDFL aggregates energy across all valid paths and contrasts them against likely but incorrect "hard" invalid paths. Our evaluation on the Breakfast Action, Hollywood Extended, and 50Salads datasets demonstrates that CDFL achieves superior results compared to the state of the art in both action segmentation and alignment.

Introduction

Action segmentation involves assigning an action label to every frame in a video. While fully supervised methods are effective, they require expensive manual annotation of start and end times for every action. Weakly supervised learning offers a more scalable alternative by using only "transcripts"—the sequence of actions—during training.

The core difficulty in weak supervision is the alignment problem: for a given sequence of actions, there are millions of possible ways to distribute them across video frames. Traditional approaches often attempt to find a single "pseudo ground truth" alignment. However, if this initial guess is inaccurate, the model learns from noise, leading to a "garbage in, garbage out" cycle that limits final accuracy.

Methods: The CDFL Framework

To overcome the limitations of single-path training, we propose an energy-based learning framework centered on the Constrained Discriminative Forward Loss (CDFL). We utilize an HMM-GRU architecture to predict frame-level action probabilities.

Overview of the CDFL framework, illustrating the recursive estimation of path energy across valid and invalid segmentations.

Segmentation Graph and Energy Estimation

Instead of selecting one alignment, we construct a fully-connected segmentation graph. This graph is anchored by an initial "Constrained Viterbi" pass but includes a neighborhood of alternative temporal boundaries ("cuts"). Every path through this graph represents a potential segmentation.

We define "valid" paths $\mathcal{P}^V$ as those matching the ground-truth transcript, and "invalid" paths $\mathcal{P}^I$ as those that violate it. The CDFL is calculated by:

  1. Aggregating Valid Energy: Using a recursive logadd function to sum the energy of all valid paths.
  2. Discriminative Learning: Penalizing "hard" invalid paths—those the model currently assigns low energy (high likelihood) despite being incorrect.

To handle the exponential number of possible paths, we developed a recursive algorithm that computes the CDFL in $O(\Delta^2KN)$ time, where $\Delta$ is the window size, $K$ is the number of actions, and $N$ is the number of frames. This ensures the method remains computationally feasible for long, complex videos.

Results and Discussion

We evaluated our approach on three major benchmarks: Breakfast Action, Hollywood Extended, and 50Salads. The CDFL approach consistently outperformed previous state-of-the-art (SOTA) methods in Mean-over-Frames (MoF) accuracy.

Dataset Previous SOTA (Method) Our CDFL
Breakfast 45.7% (D3TW) 50.2%
Hollywood Ext 33.6% (D3TW) 45.0%
50Salads 49.4% (NN-Viterbi) 54.7%
Qualitative comparison of different loss functions against Ground Truth (GT), demonstrating the refinement capability of the CDFL approach.

Qualitative analysis (Figure 3) shows that the full CDFL formulation ($L_{CDF}$) produces segmentations that most closely align with the ground truth compared to simpler variants. By considering the entire distribution of valid segmentations, the model becomes robust to early training errors and learns to refine boundaries iteratively. The discriminative nature of the loss is particularly effective at pushing the model to distinguish between similar action classes by penalizing high-confidence incorrect predictions.

Conclusion

The Constrained Discriminative Forward Loss provides a robust framework for weakly supervised action segmentation. By moving beyond single-path pseudo-labels to a discriminative energy-based approach, we achieve significant gains in accuracy across multiple benchmarks. While the method relies on an initial anchor for the segmentation graph, its ability to refine boundaries and handle "hard" negatives marks a substantial step forward. Future work may explore extending this framework to even weaker forms of supervision, such as unordered action sets.

Summary updated

Abstract

This paper is about labeling video frames with action classes under weak supervision in training, where we have access to a temporal ordering of actions, but their start and end frames in training videos are unknown. Following prior work, we use an HMM grounded on a Gated Recurrent Unit (GRU) for frame labeling. Our key contribution is a new constrained discriminative forward loss (CDFL) that we use for training the HMM and GRU under weak supervision. While prior work typically estimates the loss on a single, inferred video segmentation, our CDFL discriminates between the energy of all valid and invalid frame labelings of a training video. A valid frame labeling satisfies the ground-truth temporal ordering of actions, whereas an invalid one violates the ground truth. We specify an efficient recursive algorithm for computing the CDFL in terms of the logadd function of the segmentation energy. Our evaluation on action segmentation and alignment gives superior results to those of the state of the art on the benchmark Breakfast Action, Hollywood Extended, and 50Salads datasets.
BibTeX
@inproceedings{li2019weakly,
  author = {Jun Li and Peng Lei and Sinisa Todorovic},
  title = {{Weakly Supervised Energy-Based Learning for Action Segmentation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=J-i52jcNEe},
}
Figure