MVImgNet: A Large-scale Dataset of Multi-view Images

Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, Guanying Chen, Shuguang Cui, Xiaoguang Han

TL;DR

The authors introduce MVImgNet, a massive real-world multi-view image dataset containing 6.5 million frames from over 219,000 videos across 238 object categories. By providing a large-scale "soft bridge" between 2D and 3D vision, MVImgNet enables effective pretraining that significantly boosts performance in both 3D reconstruction (NeRF, Multi-view Stereo) and 2D view-consistent image understanding.

What Problem Does This Paper Address?

In 2D computer vision, pretraining on massive datasets like ImageNet to learn universal representations is a standard, highly successful paradigm. However, 3D computer vision has lacked a true generic counterpart to ImageNet.

Existing 3D datasets suffer from major limitations:

  1. Synthetic datasets (like ShapeNet) are clean but suffer from a severe domain gap when applied to messy, real-world scenarios.
  2. Real-world 3D datasets (like ScanObjectNN or ScanNet) are either limited in scale or restricted to specific indoor/outdoor scene domains due to the laborious nature of 3D scanning and manual labeling.

To bypass the bottleneck of expensive 3D scanning, the authors propose using multi-view images captured from everyday smartphones. This approach provides implicit 3D-aware signals (via multi-view consistency) while allowing for scalable, crowd-sourced data collection.

Methods

The authors developed an efficient, automated pipeline to turn raw smartphone videos into a highly annotated multi-view dataset:

Figure 2. The efficient data acquisition pipeline of MVImgNet.
  1. Crowdsourced Video Capture: Over 1,000 collectors shot short (~10 second) videos of daily objects, capturing $180^\circ$ or $360^\circ$ views.
  2. Sparse Reconstruction: Structure-from-Motion (SfM) via COLMAP was used to automatically compute camera intrinsics and extrinsics for frames extracted from the videos.
  3. Foreground Segmentation: An off-the-shelf tool (CarveKit) automatically generated binary masks to isolate the primary objects from their backgrounds.
  4. Dense Reconstruction & Fusion: Multi-view Stereo (MVS) depth maps were generated and fused using the foreground masks to produce a clean 3D point cloud for each object.

Datasets

The paper introduces two primary datasets:

  • MVImgNet: The core multi-view image dataset containing 6.5 million frames from 219,188 videos spanning 238 object classes (e.g., laptops, shoes, bags). Categories are aligned with the WordNet taxonomy.
  • MVPNet: A derived 3D object point cloud dataset containing 87,200 samples across 150 categories, created by filtering and manually cleaning the dense reconstructions from MVImgNet.

The authors evaluate these datasets against existing benchmarks like CO3D (a concurrent multi-view dataset), DTU (for multi-view stereo), and ScanObjectNN (for 3D point cloud classification).

Key Insights and Ideas

  • The "Soft Bridge" Concept: Multi-view images inherently contain 3D structural information through geometric consistency across frames. This allows models to learn 3D priors without requiring explicit, perfectly scanned 3D meshes during initial training.
  • Cross-Dimensional Benefits: Pretraining on MVImgNet doesn't just help 3D tasks; it significantly improves 2D image models by teaching them viewpoint invariance—the ability to recognize an object regardless of the angle it's viewed from.
  • Data Scalability Power: The authors demonstrate a clear power-law relationship: as the volume of multi-view training data scales up, the generalization performance of downstream models (like generalizable NeRFs) improves continuously without hitting an obvious ceiling.
Figure 5. PSNR of IBRNet pretrained by a different number of MVImgNet data (higher is better).

Key Results

1. 3D Radiance Field Reconstruction (NeRF)

When evaluating generalizable NeRFs (using IBRNet as a baseline) on unseen real-world objects, pretraining on MVImgNet yielded superior visual quality and quantitative metrics compared to training from scratch or pretraining on the competing CO3D dataset.

NeRF qualitative comparison of train-from-scratch, CO3D-pretrained, and MVImgNet-pretrained models.

2. Data-Efficient Multi-View Stereo (MVS)

Self-supervised pretraining on MVImgNet drastically reduces the amount of target data needed. An MVS model pretrained on MVImgNet and finetuned on just 5% of the DTU dataset achieved a 2mm depth accuracy of 50.10%, outperforming the train-from-scratch baseline (48.61%).

3. Real-World 3D Point Cloud Classification

Using the derived MVPNet dataset for self-supervised pretraining (via PointMAE) and finetuning on the challenging ScanObjectNN benchmark achieved state-of-the-art results, reaching an overall accuracy of 84.13% on the PB_T50_RS variant.

Limitations and Future Work

  • Human-Centric Bias: The dataset focuses heavily on everyday objects found in human daily life. It lacks representation for nature-centric categories (e.g., wild animals, plants), which might limit its performance on those specific domains.
  • Simple Backgrounds: The data processing pipeline intentionally filters out complex backgrounds to isolate objects. Consequently, MVImgNet is optimized for object-centric tasks and cannot be directly applied to complex, large-scale scene understanding without further adaptation techniques like domain adaptation or knowledge distillation.

Summary updated

Abstract

Being data-driven is one of the most iconic properties of deep learning algorithms. The birth of ImageNet drives a remarkable trend of "learning from large-scale data" in computer vision. Pretraining on ImageNet to obtain rich universal representations has been manifested to benefit various 2D visual tasks, and becomes a standard in 2D vision. However, due to the laborious collection of real-world 3D data, there is yet no generic dataset serving as a counterpart of ImageNet in 3D vision, thus how such a dataset can impact the 3D community is unraveled. To remedy this defect, we introduce MVImgNet, a large-scale dataset of multi-view images, which is highly convenient to gain by shooting videos of real-world objects in human daily life. It contains 6.5 million frames from 219,188 videos crossing objects from 238 classes, with rich annotations of object masks, camera parameters, and point clouds. The multi-view attribute endows our dataset with 3D-aware signals, making it a soft bridge between 2D and 3D vision. We conduct pilot studies for probing the potential of MVImgNet on a variety of 3D and 2D visual tasks, including radiance field reconstruction, multi-view stereo, and view-consistent image understanding, where MVImgNet demonstrates promising performance, remaining lots of possibilities for future explorations. Besides, via dense reconstruction on MVImgNet, a 3D object point cloud dataset is derived, called MVPNet, covering 87,200 samples from 150 categories, with the class label on each point cloud. Experiments show that MVPNet can benefit the real-world 3D object classification while posing new challenges to point cloud understanding. MVImgNet and MVPNet will be publicly available, hoping to inspire the broader vision community.
BibTeX
@inproceedings{yu2023mvimgnet,
  author = {Xianggang Yu and Mutian Xu and Yidan Zhang and Haolin Liu and Chongjie Ye and Yushuang Wu and Zizheng Yan and Chenming Zhu and Zhangyang Xiong and Tianyou Liang and Guanying Chen and Shuguang Cui and Xiaoguang Han},
  title = {{MVImgNet: A Large-scale Dataset of Multi-view Images}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
}
Figure