TL;DR
This paper introduces SAIL-VOS 3D, a novel synthetic dataset for 3D object mesh reconstruction from video, featuring frame-by-frame mesh annotations. They also present "Video2Mesh," a temporal model that leverages this video data to achieve improved 3D mesh reconstruction quality compared to single-image methods, especially for occluded objects.
What Problem Does This Paper Address?
Understanding the 3D shape of objects is crucial for many computer vision tasks, like scene understanding and robotics. While single-image 3D reconstruction has seen significant progress, it often struggles with ambiguity due to unobserved parts of an object. Existing datasets for mesh reconstruction are typically image-based, making it difficult to study how temporal information from videos can help resolve these ambiguities and improve reconstruction accuracy. This limitation hinders the development of methods that can leverage the rich cues present in video sequences, especially for dynamic scenes and occluded objects.
Methods
The core contribution is the Video2Mesh framework. It tackles 3D mesh reconstruction from video in two main stages:
- Detection and Tracking: First, it uses a standard object detector (Faster R-CNN) to find bounding boxes and class labels for objects in each frame. To establish correspondences across frames, it solves an assignment problem based on bounding box IoU and class label similarity. This allows the model to track objects over time.
- Temporal Mesh Prediction: For each detected object, the model predicts a 3D mesh. Crucially, it uses a "reference mesh" to guide the prediction. This reference mesh is either a class-specific mean shape (if the object is not tracked from the previous frame) or the predicted mesh from the previous frame (if tracked). A deep network then deforms this reference mesh based on the object's ROI features from the current image frame. The refinement process involves multiple stages of a MeshRefine module, which uses a Graph Convolutional Network (GCN) to update vertex positions. This temporal dependency allows the model to infer the shape of occluded or unseen parts by leveraging information from previous frames.
Datasets
- SAIL-VOS 3D: A newly introduced synthetic dataset collected from the GTA-V game engine. It comprises 484 videos with over 237,000 frames, featuring frame-by-frame annotations including RGB frames, camera matrices, depth data, instance-level segmentation, amodal segmentation, and 3D meshes. It's designed to capture dynamic scenes with diverse scenarios, unlike many existing static indoor datasets.
- Pix3D: A real-world image dataset used for evaluating the transferability of the trained model and for comparison with existing methods.
Key Insights and Ideas
- Temporal Information is Key: The paper strongly advocates for the use of temporal information in video for 3D reconstruction. By tracking objects and using previous frames' predictions as references, the model can infer shapes more accurately, especially for occluded parts.
- Synthetic Data for Rich Annotations: Generating high-quality, frame-by-frame 3D mesh annotations for real videos is prohibitively expensive and time-consuming. Using a game engine like GTA-V allows for the creation of such a dataset, enabling research into temporal 3D reconstruction.
- Amodal Reconstruction Matters: The dataset provides amodal annotations, meaning the full 3D shape is annotated even if parts are occluded or out-of-view in a specific frame. This is crucial for training models that can infer complete object shapes.
- Reference Mesh Strategy: Using a class-specific mean shape or the previous frame's prediction as a reference mesh is a practical way to initialize and guide the temporal mesh prediction process.
Key Results
- SAIL-VOS 3D Performance: The proposed Video2Mesh method outperforms strong baselines like MeshR-CNN on the SAIL-VOS 3D dataset, particularly in terms of APmesh (average precision for mesh reconstruction).
- Occlusion and Clip Length: Video2Mesh shows significant improvements on slightly and heavily occluded objects, as well as on longer video clips, demonstrating the benefit of temporal information.
- Transferability: The model trained on synthetic SAIL-VOS 3D data can be fine-tuned on the real-world Pix3D dataset, showing good performance even with limited real data, highlighting the utility of the synthetic dataset.
- Ablation Study: An ablation study confirms that incorporating temporal prediction significantly boosts performance, more so than just using a mean shape or transformation estimation.
Limitations and Future Work
- Synthetic Domain Gap: While the model shows good transferability, it was trained on synthetic data. The paper acknowledges that testing on real data (like DAVIS) shows reasonable results but suggests that depth data could be further explored to improve transferability.
- Computational Cost: Temporal models inherently involve more computation than single-image methods. The paper mentions that during training, the recursion is not unrolled for efficiency, but this might imply inference could be computationally intensive.
- Future Directions: The authors hope their SAIL-VOS 3D dataset will spur further research into temporal 3D reconstruction. They also suggest exploring depth data as a cue for transferability.
Summary updated