Reconstructing 3D objects from 2D images is a cornerstone of computer vision, yet single-image methods face inherent ambiguities regarding occluded or unseen parts of an object. While video sequences provide the necessary temporal cues to resolve these ambiguities, research has been hindered by a lack of high-quality video datasets with 3D ground truth. Presented at CVPR 2021, this paper introduces SAIL-VOS 3D, a massive synthetic dataset, and Video2Mesh, a baseline model designed to leverage temporal information for superior 3D reconstruction.
The Challenge of 3D Video Reconstruction
Single-image 3D reconstruction is an "ill-posed" problem; a model must essentially guess the geometry of an object's back or hidden sections. Video should theoretically solve this as movement reveals different perspectives over time. However, two major hurdles remain:
- Data Scarcity: Annotating 3D meshes in real-world video is prohibitively expensive. Existing datasets are often static, small, or lack "amodal" annotations (labeling the full shape of an object, including occluded parts).
- Model Design: Most current models treat video frames as independent images, ignoring the rich temporal context available in preceding frames.
SAIL-VOS 3D: A High-Fidelity Synthetic Benchmark
To overcome the data bottleneck, the researchers utilized the GTA-V game engine to create a photorealistic synthetic dataset. SAIL-VOS 3D consists of 484 videos and approximately 237,000 frames across 178 object categories. Every frame is densely annotated with 3D meshes, depth maps, instance-level segmentation, and amodal segmentation. This allows models to learn "object permanence"—understanding the complete structure of an object even when it is partially blocked from view.
Video2Mesh: Leveraging Temporal Priors
The authors propose Video2Mesh to demonstrate the value of temporal cues. Unlike traditional methods that start from a generic sphere for every frame, Video2Mesh uses a temporal mesh prediction strategy:
- Tracking: The model detects and tracks objects across frames using Faster R-CNN.
- Reference Meshes: If an object was seen in the previous frame, the model uses that frame's predicted mesh as its starting point. For new objects, it uses a class-specific mean mesh.
- Refinement: A Graph Convolutional Network (GCN) and "VertAlign" (aligning image features with mesh vertices) deform the reference mesh to match the current frame's visual evidence.
Key Results and Insights
- Superior Performance: Video2Mesh achieved a 10.21 APmesh on the SAIL-VOS 3D benchmark, outperforming the single-view state-of-the-art baseline, MeshR-CNN (9.68).
- Synthetic-to-Real Transfer: The study demonstrates that high-quality synthetic data is an excellent pre-training tool. Models pre-trained on SAIL-VOS 3D nearly doubled their performance on real-world data (Pix3D) when training samples were scarce (improving from 9.0 to 16.4 APmesh).
- Temporal Consistency: By using the previous frame's prediction as a prior, the model maintains geometric consistency and accumulates knowledge about an object's shape over time.
Limitations and Future Directions
While Video2Mesh sets a new baseline, it relies heavily on accurate tracking; tracking failures can propagate errors through the temporal prior. Additionally, the current model does not explicitly account for complex camera trajectories. Despite these challenges, SAIL-VOS 3D provides a critical foundation for developing the next generation of video-based 3D perception algorithms.
Summary updated