Abstract: We introduce a framework that predicts the goals behind observable human action in video. Motivated by evidence in developmental psychology, we leverage video of unintentional action to learn video representations of goals without direct supervision. Our approach models videos as contextual trajectories that represent both low-level motion and high-level action features. Experiments and visualizations show our trained model is able to predict the underlying goals in video of unintentional action. We also propose a method to "automatically correct" unintentional action by leveraging gradient signals of our model to adjust latent trajectories. Although the model is trained with minimal supervision, it is competitive with or outperforms baselines trained on large (supervised) datasets of successfully executed goals, showing that observing unintentional action is crucial to learning about goals in video.
Introduction: The 'Why' Behind the 'What'
Most action recognition systems focus on the what—identifying that a person is "swinging a hammer." However, they often struggle with the why—the underlying goal (e.g., "to hang a picture"). The fundamental challenge is that goals are latent; they aren't directly observable in the pixels. Traditional models trained only on successful actions often conflate the action with the goal.
Drawing inspiration from developmental psychology—which shows that infants learn to understand intentions by observing failed attempts—researchers at Columbia University argue that failure is the missing piece in machine action understanding. Without seeing someone miss a nail or drop a glass, a model cannot easily distinguish the intended outcome from the physical motion.
Methodology: Contextual Trajectories
The proposed framework models video as a contextual trajectory using a two-level architecture designed to decouple visible movements from latent intentions.
- Motion Level: The video is broken into short clips, which are processed by a 3D Convolutional Neural Network (3D CNN) to capture granular, short-term motion features.
- Action Level: These clip embeddings are fed into a Transformer stack. Unlike recurrent networks that process data linearly, the Transformer allows every moment in the video to "attend" to every other moment, creating a context-adjusted latent trajectory of the entire action.
Training without Direct Goal Labels
The model is trained using "indirect" or weak supervision via two primary losses:
- Action Intentionality: The model learns to predict whether a clip occurs before, during, or after a "failure" point.
- Temporal Consistency: A self-supervised task where the model predicts if a sequence of clips is in the correct chronological order.
Action Auto-Correction
Perhaps the most novel technical contribution is the use of adversarial gradients to "correct" failures. If the model identifies a trajectory as "unintentional," the researchers use gradient signals to find the minimal adjustment needed in the latent space to flip that classification to "intentional." This shifted trajectory effectively represents the "intended goal" that the person in the video failed to achieve. By treating the "adversarial delta" as a meaningful feature, the model learns that the goal is a destination in feature space, while the action is merely the path taken to get there.
Datasets and Evaluation
- Oops! Dataset: The primary dataset, containing thousands of "fail" videos (unintentional actions). The authors expanded this with natural language descriptions of goals and failures.
- Kinetics: A standard, large-scale dataset of successful human actions used for pre-training and as a retrieval gallery for the "auto-correct" task.
- SVO Triples: For evaluation, the authors extracted Subject-Verb-Object triples (e.g., "man-open-bottle") from natural language descriptions to provide a structured way to measure goal prediction.
Key Results
- Superior Detection: The model outperformed baselines trained on the massive Kinetics dataset at localizing and classifying unintentional actions, achieving 77.9% accuracy on three-way classification (intentional, transitional, unintentional).
- Goal Prediction: Despite having no direct supervision of goal categories during training, the model’s learned representations were highly effective at predicting SVO triples, performing on par with or better than fully supervised models.
- Successful Retrieval: When the model "auto-corrected" a failure video, the resulting latent vector was able to successfully retrieve videos of the actual intended goal from a separate database of successful actions.
Conclusion and Future Work
This research demonstrates that context matters; by using a Transformer, the model realizes that the same physical motion can have different meanings based on the context. While the "auto-correct" feature currently operates in the latent feature space (meaning it cannot yet generate a synthetic video of the success), it proves that observing failure is crucial for learning about goals. Future work may explore entirely self-supervised ways to detect failure transitions and more complex, hierarchical representations of human intent.
Summary updated