SAIL-VOS 3D: A Synthetic Dataset and Baselines for Object Detection and 3D Mesh Reconstruction From Video Data

Yuan-Ting Hu, Jiahong Wang, Raymond A. Yeh, Alexander G. Schwing

Reconstructing 3D objects from 2D images is a cornerstone of computer vision, yet single-image methods face inherent ambiguities regarding occluded or unseen parts of an object. While video sequences provide the necessary temporal cues to resolve these ambiguities, research has been hindered by a lack of high-quality video datasets with 3D ground truth. Presented at CVPR 2021, this paper introduces SAIL-VOS 3D, a massive synthetic dataset, and Video2Mesh, a baseline model designed to leverage temporal information for superior 3D reconstruction.

The Challenge of 3D Video Reconstruction

Single-image 3D reconstruction is an "ill-posed" problem; a model must essentially guess the geometry of an object's back or hidden sections. Video should theoretically solve this as movement reveals different perspectives over time. However, two major hurdles remain:

  1. Data Scarcity: Annotating 3D meshes in real-world video is prohibitively expensive. Existing datasets are often static, small, or lack "amodal" annotations (labeling the full shape of an object, including occluded parts).
  2. Model Design: Most current models treat video frames as independent images, ignoring the rich temporal context available in preceding frames.

SAIL-VOS 3D: A High-Fidelity Synthetic Benchmark

To overcome the data bottleneck, the researchers utilized the GTA-V game engine to create a photorealistic synthetic dataset. SAIL-VOS 3D consists of 484 videos and approximately 237,000 frames across 178 object categories. Every frame is densely annotated with 3D meshes, depth maps, instance-level segmentation, and amodal segmentation. This allows models to learn "object permanence"—understanding the complete structure of an object even when it is partially blocked from view.

A sequence from the SAIL-VOS 3D dataset showing the original video frames (top) and the corresponding 3D mesh reconstructions (bottom) that maintain temporal consistency.

Video2Mesh: Leveraging Temporal Priors

The authors propose Video2Mesh to demonstrate the value of temporal cues. Unlike traditional methods that start from a generic sphere for every frame, Video2Mesh uses a temporal mesh prediction strategy:

  • Tracking: The model detects and tracks objects across frames using Faster R-CNN.
  • Reference Meshes: If an object was seen in the previous frame, the model uses that frame's predicted mesh as its starting point. For new objects, it uses a class-specific mean mesh.
  • Refinement: A Graph Convolutional Network (GCN) and "VertAlign" (aligning image features with mesh vertices) deform the reference mesh to match the current frame's visual evidence.

Key Results and Insights

  • Superior Performance: Video2Mesh achieved a 10.21 APmesh on the SAIL-VOS 3D benchmark, outperforming the single-view state-of-the-art baseline, MeshR-CNN (9.68).
  • Synthetic-to-Real Transfer: The study demonstrates that high-quality synthetic data is an excellent pre-training tool. Models pre-trained on SAIL-VOS 3D nearly doubled their performance on real-world data (Pix3D) when training samples were scarce (improving from 9.0 to 16.4 APmesh).
  • Temporal Consistency: By using the previous frame's prediction as a prior, the model maintains geometric consistency and accumulates knowledge about an object's shape over time.

Limitations and Future Directions

While Video2Mesh sets a new baseline, it relies heavily on accurate tracking; tracking failures can propagate errors through the temporal prior. Additionally, the current model does not explicitly account for complex camera trajectories. Despite these challenges, SAIL-VOS 3D provides a critical foundation for developing the next generation of video-based 3D perception algorithms.

Summary updated

Abstract

Extracting detailed 3D information of objects from video data is an important goal for holistic scene understanding. While recent methods have shown impressive results when reconstructing meshes of objects from a single image, results often remain ambiguous as part of the object is unobserved. Moreover, existing image-based datasets for mesh reconstruction don't permit to study models which integrate temporal information. To alleviate both concerns we present SAIL-VOS 3D: a synthetic video dataset with frame-by-frame mesh annotations which extends SAIL-VOS. We also develop first baselines for reconstruction of 3D meshes from video data via temporal models. We demonstrate efficacy of the proposed baseline on SAIL-VOS 3D and Pix3D, showing that temporal information improves reconstruction quality. Resources and additional information are available at http://sailvos.web.illinois.edu.
BibTeX
@inproceedings{hu2021sail,
  author = {Yuan-Ting Hu and Jiahong Wang and Raymond A. Yeh and Alexander G. Schwing},
  title = {{SAIL-VOS 3D: A Synthetic Dataset and Baselines for Object Detection and 3D Mesh Reconstruction From Video Data}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr46437.2021.00147},
}
Figure