Reconstructing and Grounding Narrated Instructional Videos in 3D
Abstract
This paper addresses the challenge of extracting structured 3D information from unconstrained, narrated instructional videos. We present a framework that reconstructs 3D point clouds from individual videos and aligns them into a common coordinate system using a robust "divide-and-conquer" strategy and a 3D alignment graph. This approach effectively handles the intra-class variations common in real-world objects like car engines. Furthermore, we propose an unsupervised method for grounding spoken instructions to 3D locations by leveraging photographer's bias and backprojecting 2D visual cues. Evaluated on a novel dataset of car maintenance videos, our system demonstrates superior performance in cross-video keypoint transfer and 3D text localization compared to baseline methods.
Introduction
Instructional videos on platforms like YouTube are a rich repository of procedural knowledge. However, for computer vision systems, these videos are difficult to process due to significant variations in viewpoints, lighting, and the specific models of objects being shown. Standard Structure from Motion (SfM) techniques fail when objects are not identical across views. Moreover, the lack of large-scale annotated datasets linking narration to 3D coordinates hinders the development of supervised models.
Methodology
Our framework tackles these challenges through a two-stage process: robust 3D reconstruction and unsupervised language grounding.
3D Reconstruction via Alignment Graphs
Rather than attempting a global reconstruction of all videos simultaneously, we reconstruct each video independently to create local 3D point clouds. We then establish correspondences between these models using learned local features (R2D2) and dense flow (GLU-Net). These matches are used to build a 3D Alignment Graph. By finding paths through this graph, we can align models that are too visually distinct for direct matching by utilizing intermediate "bridge" videos that share common features with both.
Unsupervised 3D Text Grounding
To ground language without manual labels, we exploit the "photographer's bias"—the observation that narrators typically center the object they are discussing. We combine cues from the frame center, hand detection, and pre-trained video-text models to generate noisy 2D estimates of mentioned objects. These estimates are backprojected into the 3D space and aggregated. A BERT-based encoder is then trained to map text features to 3D voxel scores, effectively learning the spatial location of parts like "air filters" or "spark plugs" across the entire dataset.
Datasets and Evaluation
We collected a dataset of 174 narrated car maintenance videos featuring six different car models and various maintenance tasks. To evaluate the system, we manually annotated keypoints on a subset of the data, providing a ground truth for 3D alignment and grounding accuracy.
Results and Discussion
The 3D alignment graph achieved an 80% success rate in keypoint transfer within an 8cm tolerance, significantly outperforming direct matching. In text grounding, our model achieved a Percentage of Correct Keypoints (PCK) of 0.41, nearly doubling the chance baseline.
Key findings include:
- Graph Robustness: The alignment graph is essential for bridging the gap between different object instances (e.g., different engine models).
- Effectiveness of Simple Cues: The "Center of Frame" heuristic was the most effective grounding cue, highlighting the consistency of human videography.
- 3D vs. 2D: Grounding in 3D provides a more stable and occlusion-resistant representation of object locations compared to 2D-only approaches.
Conclusion
Our work demonstrates the feasibility of grounding narrated instructions in 3D using unlabelled, real-world video data. While challenges remain for small or non-rigid objects, this approach paves the way for advanced applications such as AR-based instructional manuals that can automatically interpret and spatialize online video tutorials.
Summary updated