TL;DR
The authors introduce MVImgNet, a massive real-world multi-view image dataset containing 6.5 million frames from over 219,000 videos across 238 object categories. By providing a large-scale "soft bridge" between 2D and 3D vision, MVImgNet enables effective pretraining that significantly boosts performance in both 3D reconstruction (NeRF, Multi-view Stereo) and 2D view-consistent image understanding.
What Problem Does This Paper Address?
In 2D computer vision, pretraining on massive datasets like ImageNet to learn universal representations is a standard, highly successful paradigm. However, 3D computer vision has lacked a true generic counterpart to ImageNet.
Existing 3D datasets suffer from major limitations:
- Synthetic datasets (like ShapeNet) are clean but suffer from a severe domain gap when applied to messy, real-world scenarios.
- Real-world 3D datasets (like ScanObjectNN or ScanNet) are either limited in scale or restricted to specific indoor/outdoor scene domains due to the laborious nature of 3D scanning and manual labeling.
To bypass the bottleneck of expensive 3D scanning, the authors propose using multi-view images captured from everyday smartphones. This approach provides implicit 3D-aware signals (via multi-view consistency) while allowing for scalable, crowd-sourced data collection.
Methods
The authors developed an efficient, automated pipeline to turn raw smartphone videos into a highly annotated multi-view dataset:
- Crowdsourced Video Capture: Over 1,000 collectors shot short (~10 second) videos of daily objects, capturing $180^\circ$ or $360^\circ$ views.
- Sparse Reconstruction: Structure-from-Motion (SfM) via COLMAP was used to automatically compute camera intrinsics and extrinsics for frames extracted from the videos.
- Foreground Segmentation: An off-the-shelf tool (CarveKit) automatically generated binary masks to isolate the primary objects from their backgrounds.
- Dense Reconstruction & Fusion: Multi-view Stereo (MVS) depth maps were generated and fused using the foreground masks to produce a clean 3D point cloud for each object.
Datasets
The paper introduces two primary datasets:
- MVImgNet: The core multi-view image dataset containing 6.5 million frames from 219,188 videos spanning 238 object classes (e.g., laptops, shoes, bags). Categories are aligned with the WordNet taxonomy.
- MVPNet: A derived 3D object point cloud dataset containing 87,200 samples across 150 categories, created by filtering and manually cleaning the dense reconstructions from MVImgNet.
The authors evaluate these datasets against existing benchmarks like CO3D (a concurrent multi-view dataset), DTU (for multi-view stereo), and ScanObjectNN (for 3D point cloud classification).
Key Insights and Ideas
- The "Soft Bridge" Concept: Multi-view images inherently contain 3D structural information through geometric consistency across frames. This allows models to learn 3D priors without requiring explicit, perfectly scanned 3D meshes during initial training.
- Cross-Dimensional Benefits: Pretraining on MVImgNet doesn't just help 3D tasks; it significantly improves 2D image models by teaching them viewpoint invariance—the ability to recognize an object regardless of the angle it's viewed from.
- Data Scalability Power: The authors demonstrate a clear power-law relationship: as the volume of multi-view training data scales up, the generalization performance of downstream models (like generalizable NeRFs) improves continuously without hitting an obvious ceiling.
Key Results
1. 3D Radiance Field Reconstruction (NeRF)
When evaluating generalizable NeRFs (using IBRNet as a baseline) on unseen real-world objects, pretraining on MVImgNet yielded superior visual quality and quantitative metrics compared to training from scratch or pretraining on the competing CO3D dataset.
2. Data-Efficient Multi-View Stereo (MVS)
Self-supervised pretraining on MVImgNet drastically reduces the amount of target data needed. An MVS model pretrained on MVImgNet and finetuned on just 5% of the DTU dataset achieved a 2mm depth accuracy of 50.10%, outperforming the train-from-scratch baseline (48.61%).
3. Real-World 3D Point Cloud Classification
Using the derived MVPNet dataset for self-supervised pretraining (via PointMAE) and finetuning on the challenging ScanObjectNN benchmark achieved state-of-the-art results, reaching an overall accuracy of 84.13% on the PB_T50_RS variant.
Limitations and Future Work
- Human-Centric Bias: The dataset focuses heavily on everyday objects found in human daily life. It lacks representation for nature-centric categories (e.g., wild animals, plants), which might limit its performance on those specific domains.
- Simple Backgrounds: The data processing pipeline intentionally filters out complex backgrounds to isolate objects. Consequently, MVImgNet is optimized for object-centric tasks and cannot be directly applied to complex, large-scale scene understanding without further adaptation techniques like domain adaptation or knowledge distillation.
Summary updated