Abstract
Dense action labeling in video is traditionally a labor-intensive task requiring frame-by-frame annotations. This paper presents Extended Connectionist Temporal Classification (ECTC), a framework designed for weakly supervised learning where only the order of actions is known. By integrating visual similarity constraints into the alignment process, ECTC prevents the "degeneration" common in weak supervision. Our results on the Breakfast Actions and Hollywood2 datasets demonstrate that ECTC achieves performance comparable to fully supervised models while utilizing less than 1% of the manual annotation effort.
Introduction
Identifying the precise temporal boundaries of actions in video—known as dense action labeling—is a critical task for human activity understanding. However, the cost of manual frame-level annotation is a significant barrier to scaling these models. Weakly supervised learning offers a solution by training models using only high-level sequence information (e.g., "cracking an egg" followed by "whisking").
The primary challenge in this setting is the massive search space of possible alignments between the action sequence and the video frames. Standard approaches often fail, leading to "degenerated" solutions where a single action is predicted for the majority of the video.
Methods
The proposed ECTC framework builds upon Recurrent Neural Networks (RNNs), specifically Bidirectional LSTMs, to capture temporal dependencies. It extends the standard Connectionist Temporal Classification (CTC) algorithm, which was originally developed for speech recognition, to better suit the characteristics of video data.
Extended Connectionist Temporal Classification (ECTC)
Unlike speech phonemes, video actions are long-duration events. ECTC addresses this by introducing a visual similarity constraint (a binary term) into the alignment objective. This term calculates the similarity between consecutive frames using k-means clustering and cosine similarity.
By rewarding the model for assigning the same label to visually similar frames, ECTC ensures that the learned alignments are grounded in the actual visual content rather than just satisfying the ordering constraint. The framework uses an extended forward-backward dynamic programming algorithm to efficiently evaluate these constrained paths.
Semi-Supervised Extension
ECTC naturally supports semi-supervised learning. By providing a few "anchor" frames—manually labeled frames representing less than 1% of the total video—the search space for alignments is significantly pruned, leading to accuracy levels that rival fully supervised systems.
Results and Discussion
The framework was evaluated on the Breakfast Actions Dataset (77 hours of cooking) and a subset of Hollywood2.
- Accuracy: On the Breakfast dataset, ECTC achieved 27.7% frame accuracy, outperforming standard CTC (21.8%) and traditional supervised baselines like HMMs (19.7%).
- Annotation Efficiency: Remarkably, with less than 1% of frames labeled as anchors, ECTC performed nearly as well as models trained on 100% frame-by-frame labels.
- Qualitative Consistency: Visualizations indicate that ECTC produces cleaner, more temporally consistent action segments compared to the noisy outputs of standard CTC.
Conclusion
ECTC demonstrates that visual similarity can serve as a powerful proxy for explicit supervision in temporal action labeling. By stabilizing the learning process against degeneration, the framework enables high-performance action recognition with minimal annotation overhead. Future work will explore the integration of richer weak supervision sources, such as natural language descriptions and movie scripts, to further refine boundary precision in complex activities.
Summary updated