Reconstructing and grounding narrated instructional videos in 3D

Dimitri Zhukov, Ignacio Rocco, Ivan Laptev, Josef Sivic, Johannes L. Schönberger, Bugra Tekin, Marc PollefeysarXiv 2021 · arXiv

Reconstructing and Grounding Narrated Instructional Videos in 3D

Abstract

This paper addresses the challenge of extracting structured 3D information from unconstrained, narrated instructional videos. We present a framework that reconstructs 3D point clouds from individual videos and aligns them into a common coordinate system using a robust "divide-and-conquer" strategy and a 3D alignment graph. This approach effectively handles the intra-class variations common in real-world objects like car engines. Furthermore, we propose an unsupervised method for grounding spoken instructions to 3D locations by leveraging photographer's bias and backprojecting 2D visual cues. Evaluated on a novel dataset of car maintenance videos, our system demonstrates superior performance in cross-video keypoint transfer and 3D text localization compared to baseline methods.

Introduction

Instructional videos on platforms like YouTube are a rich repository of procedural knowledge. However, for computer vision systems, these videos are difficult to process due to significant variations in viewpoints, lighting, and the specific models of objects being shown. Standard Structure from Motion (SfM) techniques fail when objects are not identical across views. Moreover, the lack of large-scale annotated datasets linking narration to 3D coordinates hinders the development of supervised models.

Overview of the 3D reconstruction and text grounding process across different narrated videos.

Methodology

Our framework tackles these challenges through a two-stage process: robust 3D reconstruction and unsupervised language grounding.

3D Reconstruction via Alignment Graphs

Rather than attempting a global reconstruction of all videos simultaneously, we reconstruct each video independently to create local 3D point clouds. We then establish correspondences between these models using learned local features (R2D2) and dense flow (GLU-Net). These matches are used to build a 3D Alignment Graph. By finding paths through this graph, we can align models that are too visually distinct for direct matching by utilizing intermediate "bridge" videos that share common features with both.

The 3D reconstruction pipeline: (a) SfM for local models, (b) feature extraction, (c) 2D-2D and 3D-3D matching, and (d) the resulting alignment graph.

Unsupervised 3D Text Grounding

To ground language without manual labels, we exploit the "photographer's bias"—the observation that narrators typically center the object they are discussing. We combine cues from the frame center, hand detection, and pre-trained video-text models to generate noisy 2D estimates of mentioned objects. These estimates are backprojected into the 3D space and aggregated. A BERT-based encoder is then trained to map text features to 3D voxel scores, effectively learning the spatial location of parts like "air filters" or "spark plugs" across the entire dataset.

Datasets and Evaluation

We collected a dataset of 174 narrated car maintenance videos featuring six different car models and various maintenance tasks. To evaluate the system, we manually annotated keypoints on a subset of the data, providing a ground truth for 3D alignment and grounding accuracy.

Results and Discussion

The 3D alignment graph achieved an 80% success rate in keypoint transfer within an 8cm tolerance, significantly outperforming direct matching. In text grounding, our model achieved a Percentage of Correct Keypoints (PCK) of 0.41, nearly doubling the chance baseline.

Key findings include:

  • Graph Robustness: The alignment graph is essential for bridging the gap between different object instances (e.g., different engine models).
  • Effectiveness of Simple Cues: The "Center of Frame" heuristic was the most effective grounding cue, highlighting the consistency of human videography.
  • 3D vs. 2D: Grounding in 3D provides a more stable and occlusion-resistant representation of object locations compared to 2D-only approaches.

Conclusion

Our work demonstrates the feasibility of grounding narrated instructions in 3D using unlabelled, real-world video data. While challenges remain for small or non-rigid objects, this approach paves the way for advanced applications such as AR-based instructional manuals that can automatically interpret and spatialize online video tutorials.

Summary updated

Abstract

Narrated instructional videos often show and describe manipulations of similar objects, e.g., repairing a particular model of a car or laptop. In this work we aim to reconstruct such objects and to localize associated narrations in 3D. Contrary to the standard scenario of instance-level 3D reconstruction, where identical objects or scenes are present in all views, objects in different instructional videos may have large appearance variations given varying conditions and versions of the same product. Narrations may also have large variation in natural language expressions. We address these challenges by three contributions. First, we propose an approach for correspondence estimation combining learnt local features and dense flow. Second, we design a two-step divide and conquer reconstruction approach where the initial 3D reconstructions of individual videos are combined into a 3D alignment graph. Finally, we propose an unsupervised approach to ground natural language in obtained 3D reconstructions. We demonstrate the effectiveness of our approach for the domain of car maintenance. Given raw instructional videos and no manual supervision, our method successfully reconstructs engines of different car models and associates textual descriptions with corresponding objects in 3D.
BibTeX
@misc{zhukov2021reconstructing,
  author = {Dimitri Zhukov and Ignacio Rocco and Ivan Laptev and Josef Sivic and Johannes L. Schönberger and Bugra Tekin and Marc Pollefeys},
  title = {{Reconstructing and grounding narrated instructional videos in 3D}},
  year = {2021},
  eprint = {2109.04409},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2109.04409},
}
Figure