Abstract
Automatic generation of textual video descriptions that are time-aligned with video content is a long-standing goal in computer vision. The task is challenging due to the difficulty of bridging the semantic gap between the visual and natural language domains. This paper addresses the task of automatically generating an alignment between a set of instructions and a first person video demonstrating an activity. The sparse descriptions and ambiguity of written instructions create significant alignment challenges. The key to our approach is the use of egocentric cues to generate a concise set of action proposals, which are then matched to recipe steps using object recognition and computational linguistic techniques. We obtain promising results on both the Extended GTEA Gaze+ dataset and the Bristol Egocentric Object Interactions Dataset.
Introduction
Bridging the "semantic gap" between visual pixels and natural language remains a classic challenge in computer vision. Most existing video-text alignment research relies on narrated videos, where a person describes their actions in real-time, providing a dense, time-synced signal. However, many instructional videos are non-narrated, and written recipes are "sparse"—they omit micro-actions like "reach for the handle." This research addresses the unsupervised alignment problem: mapping high-level instructions to specific segments in a first-person (egocentric) video without manual labels.
Methodology
The authors propose a three-stage pipeline designed to leverage the unique characteristics of first-person video:
- Action Proposal Generation: Instead of analyzing the entire video, the system identifies potential "action segments." A bidirectional LSTM is trained on spatio-temporal features (C3D) combined with egocentric cues: hand pose (segmenting hands to capture manipulation) and head motion (calculating camera rotation to detect task boundaries).
- Task-Specific Object Recognition: Within proposed segments, the system identifies manipulated objects. Accuracy is enhanced via gaze-contingent cropping—training a ResNet-101 on image patches centered where the user is looking.
- Recipe Parsing and Alignment: The Stanford Dependency Parser extracts (action, object) pairs from instructions (e.g., "pour milk"). These are matched to visual detections using Word2Vec embeddings, allowing the system to recognize that "whisking" is semantically related to "scrambling."
Experimental Setup
The system was evaluated on two prominent egocentric datasets: Extended GTEA Gaze+ (EGTEA), featuring 86 cooking videos, and the Bristol Egocentric Object Interactions (BEOID), containing 58 videos of daily tasks. Since these datasets lacked original recipe-to-video mappings, the authors contributed new ground-truth annotations for evaluation.
Results and Discussion
The proposed method achieved a significant performance leap over previous state-of-the-art baselines. On the BEOID dataset, the system reached an Average Precision (AP) of 0.7641, compared to the baseline's 0.2828. On the more complex EGTEA dataset, it achieved an AP of 0.4905, more than doubling the baseline performance.
Key insights from the study include:
- Egocentric Cues as Action Magnets: Hand and head motion are far more predictive of action boundaries than raw visual features, significantly boosting the Intersection over Union (IoU) of action proposals.
- Gaze as a Spatial Filter: Cropping images around the gaze point allows the object recognizer to ignore background clutter, mimicking human focus during task execution.
- Semantic Robustness: Using Word2Vec allows the model to handle the linguistic sparsity of recipes by understanding semantic similarities between different verbs and nouns.
Conclusion
While the current model relies on gaze data from wearable eye-trackers, it demonstrates the power of egocentric cues in solving complex alignment tasks. Future work may involve integrating gaze-prediction algorithms to extend this utility to standard videos. This research provides a foundation for systems that can learn complex human manipulations by simultaneously "watching" a demonstration and "reading" a manual.
Summary updated