Abstract
Action recognition has become a rapidly developing research field within the last decade. But with the increasing demand for large scale data, the need of hand annotated data for the training becomes more and more impractical. One way to avoid frame-based human annotation is the use of action order information to learn the respective action classes. In this context, we propose a hierarchical approach to address the problem of weakly supervised learning of human actions from ordered action labels by structuring recognition in a coarse-to-fine manner. Given a set of videos and an ordered list of the occurring actions, the task is to infer start and end frames of the related action classes within the video and to train the respective action classifiers without any need for hand labeled frame boundaries. We address this problem by combining a framewise RNN model with a coarse probabilistic inference. This combination allows for the temporal alignment of long sequences and thus, for an iterative training of both elements. While this system alone already generates good results, we show that the performance can be further improved by approximating the number of subactions to the characteristics of the different action classes as well as by the introduction of a regularizing length prior. The proposed system is evaluated on two benchmark datasets, the Breakfast and the Hollywood extended dataset, showing a competitive performance on various weak learning tasks such as temporal action segmentation and action alignment.
Introduction
Traditional action recognition requires "full supervision," where human annotators mark the exact start and end frames for every action. This is incredibly labor-intensive and doesn't scale well to the thousands of hours of video needed for deep learning.
The "weakly supervised" alternative uses only transcripts: an ordered list of actions occurring in a video without timestamps. The challenge here is twofold:
- Temporal Alignment: The model must figure out which frames belong to which action label.
- Modeling Complexity: Long actions have internal structures that a single label can't capture, often leading models to "skip" through states too quickly.
Methodology
The authors use a hierarchical approach that bridges the gap between fine-grained frame data and coarse-grained action sequences.
Hybrid Architecture and Latent Subactions
The system uses a Recurrent Neural Network (RNN) with Gated Recurrent Units (GRUs) to model local temporal context at the frame level. This is paired with a Hidden Markov Model (HMM) that handles global sequence logic. Instead of modeling an action as one giant block, they divide each action into smaller, hidden "subactions" (e.g., "reaching," "grasping") that the model learns automatically.
Iterative Training
The model starts with a simple uniform distribution of actions across the video. It then alternates between training the RNN on the current alignment and using the HMM (via Viterbi decoding) to find a better alignment. This iterative process continues until the system converges on a stable segmentation.
The Length Prior: Addressing the "Skip State" Problem
A common failure in HMMs is the "skip state" problem, where the model assigns only 1–2 frames to a state to maximize probability. The authors introduce a Length Prior (using Half-Gaussian or Half-Poisson distributions) that penalizes states for being too short or unrealistically long, forcing the model to respect the physical duration of human movement.
Key Insights and Results
- The Power of Regularization: The introduction of the length prior was a major breakthrough. Without it, nearly 75% of states were "skip states" (only 1 frame long). With the prior, accuracy on the Breakfast dataset jumped from 32.6% to 37.0%.
- Subactions Capture Nuance: By allowing the number of subactions to be re-estimated during training, the model naturally adapts to the complexity of different actions, leading to a 20% performance boost over models with fixed action lengths.
- The "Sparse" Sweet Spot: Adding just a tiny bit of supervision—annotating only 0.25% of frames (roughly one frame per action instance)—boosted performance from 36.7% to 56.0%, nearly reaching the performance of fully supervised models (61.3%).
Performance Benchmarks
On the Breakfast Dataset, the hybrid model achieved 36.7% Mean over Frames (MoF), significantly outperforming previous state-of-the-art methods like HTK (25.9%) and ECTC (27.7%). On the Hollywood Extended Dataset, the system proved robust to high visual noise, outperforming previous RNN-only approaches.
Conclusion
By combining deep learning's temporal modeling with classical probabilistic constraints, this hybrid approach provides a scalable path for video understanding. The inclusion of a length prior effectively mitigates the "skip state" problem, allowing the model to learn meaningful temporal alignments from minimal supervision.
Summary updated