Abstract
Recognizing unintentional actions—such as a skater falling or a glass spilling—is a critical component of understanding human behavior in computer vision. However, this task is hindered by the scarcity of precise temporal annotations and the visual similarity between the early stages of successful and failed actions. This paper proposes a self-supervised framework that learns video representations by combining a global temporal contrastive loss with a simplified pairwise clip-ordering task. By focusing on the global context and the chronological flow of events, the model effectively distinguishes between intentional and unintentional actions. Experimental results on the Oops dataset demonstrate that this approach achieves state-of-the-art performance in classification, localization, and anticipation, even outperforming some fully supervised models.
Introduction
In the field of computer vision, action recognition typically focuses on identifying what is happening (e.g., walking, cooking, or jumping). However, a deeper understanding of human behavior requires recognizing intent—specifically, the ability to differentiate between a deliberate action and an accidental one. Predicting unintentional actions is a significant challenge due to two primary factors:
- Annotation Scarcity: While "fail videos" are abundant online, labeling the exact millisecond an action transitions from intended to accidental is labor-intensive and expensive.
- Contextual Subtlety: The visual features at the start of a successful action and a failed one are often nearly identical. Traditional self-supervised methods that rely on local temporal windows often fail to capture the broader context necessary to identify when an action has deviated from its intended path.
Methodology
The proposed framework utilizes a self-supervised approach to learn representations without manual labels during the pre-training phase. The system employs a ResNet3D-18 backbone to process 16-frame clips and relies on two complementary loss functions:
Temporal Contrastive Loss ($L_{temp}$)
Using the InfoNCE framework, the model learns that clips appearing close together in time should have similar feature representations, while clips from distant parts of the video should be distinct. Unlike previous methods that only sampled negatives from a local neighborhood, this model incorporates the entire video as global context, allowing it to learn the unique characteristics of different action stages.
Pairwise Order Loss ($L_{ord}$)
Previous research attempted to use "Video Sorting" (predicting the permutation of multiple shuffled clips), but this task often proved too noisy for effective learning. The authors simplify this into a binary task: given two random clips from the same video, the model must predict which one occurred first. This forces the network to learn the fundamental chronological flow and transitions of an action.
Datasets and Evaluation
The model was primarily evaluated on the Oops Dataset, which contains approximately 20,000 "fail" videos from YouTube. A small labeled subset was used for evaluation, categorizing video segments as intentional, transitional, or unintentional. To ensure a clean training signal, the authors implemented an automatic "cut detection" model to filter out edited compilation videos.
Additionally, the LAD2000 dataset for video anomaly detection was used to demonstrate the generalizability of the learned representations across different tasks.
Key Insights
- Global Context is Essential: Unintentional actions are defined by their relationship to the entire event. A local one-second window is rarely sufficient to determine if a failure has occurred; the model must understand the approach and the subsequent aftermath.
- Simplicity in Supervision: Moving from complex permutation sorting to simple pairwise ordering significantly improved performance. Pairwise ordering provides a more stable and reliable signal for the network to learn temporal progression.
- Complementary Learning: The contrastive loss helps the model identify specific states, while the order loss helps it understand how those states transition over time. Together, they form a robust representation for accident prediction.
Results
The proposed method achieved state-of-the-art results on the Oops dataset across three key downstream tasks:
- Classification: Using a linear classifier with a frozen backbone, the model achieved 59.2% accuracy using only 10% of labeled data. This outperformed the previous best self-supervised method (Video Speed) by 9.3% and surpassed a model pre-trained on the massive Kinetics dataset with full supervision.
- Localization: The model pinpointed the "moment of failure" with 70.5% accuracy (within a 1-second window), representing a 5.2% improvement over previous benchmarks.
- Anticipation: When tasked with predicting a failure 1.5 seconds before it occurs, the model reached 62.4% accuracy, demonstrating its potential for real-time applications.
Conclusion and Future Work
This research highlights the power of global temporal context and simplified ordering tasks in self-supervised learning for action understanding. While the Oops dataset provides a strong foundation, it is somewhat biased toward slapstick-style accidents. Future work will focus on testing these representations on more diverse, "in-the-wild" unintentional actions and exploring their application in robotics to help agents detect and correct their own mistakes in real-time.
Summary updated