MVMO: A Multi-Object Dataset for Wide Baseline Multi-View Semantic Segmentation

Aitor Alvarez-Gila, Joost van de Weijer, Yaxing Wang, Estíbaliz Garrote

TL;DR

The MVMO (Multi-View, Multi-Object) dataset is a massive new synthetic benchmark featuring 116,000 scenes, each captured from 25 wide-baseline camera angles. By providing high-density object environments with complex occlusions and photorealistic path-traced renders, it challenges current semantic segmentation models to move beyond single-view limitations and exploit complementary information across vastly different perspectives.

What Problem Does This Paper Address?

Semantic segmentation—the task of assigning a class label to every pixel in an image—is a cornerstone of computer vision. However, most research relies on single-view inputs, which struggle when objects are heavily occluded, distant, or viewed from awkward angles.

While multi-view setups (like those on self-driving cars) can help, existing datasets typically feature "close baselines," where cameras are positioned near each other. This results in highly redundant information. The authors argue that we lack data for wide-baseline scenarios—where cameras are spread far apart (e.g., a ceiling-mounted camera vs. a floor-level one). In these cases, the appearance of an object can change drastically, and what is hidden in one view might be perfectly visible in another. MVMO is designed to fill this gap, providing a playground for research into cross-view semantic transfer and multi-view fusion.

Methods

The researchers used Blender’s Python API to procedurally generate 116,000 unique 3D scenes.

  • Scene Composition: Each scene contains 15–20 objects randomly selected from 10 classes in the ModelNet10 dataset (e.g., chairs, toilets, desks).
  • Rendering: To ensure realism, they used the Cycles path-tracing engine, which simulates physically accurate light behavior, including inter-reflections and specularities.
  • Camera Setup: 25 cameras are distributed across a hemisphere surrounding the scene at four different heights (levels L0 to L3), as shown in the camera rig visualization:
Fig. 1: Top: two scenes from the proposed 116,000 scene MVMO dataset and the 25 equidistributed camera locations. Bottom: rendered views and semantic ground truth for the 5 camera poses (highlighted) used in our experiments.

Datasets

The paper introduces the MVMO dataset, which stands out from existing benchmarks due to its combination of wide baselines and high object density. The authors provide a detailed comparison in Table 1:

Table 1: Datasets for multi/cross-view semantic segmentation. The table shows the lack of datasets with wide baseline and high object density addressed by MVMO.

The dataset is split into a training set (100k scenes) and two test sets (4k scenes each): "Same Objects" (SO) and "Other Objects" (OO), the latter testing the model's ability to generalize to unseen 3D shapes within the same categories.

Key Insights and Ideas

  • The Zenithal Advantage: Models trained on the "top-down" view (L0.cam0) generally perform better. From this bird's-eye perspective, there are fewer inter-object occlusions, making the segmentation task fundamentally easier.
  • The Fragility of Direct Transfer: A model trained on a side view (L3) fails significantly when tested on a top view (L0), and vice versa. This confirms that standard CNN features are not naturally invariant to the massive geometric shifts found in wide-baseline setups.
  • Homography is Not Enough: The authors tried using a planar homography (a geometric transformation that assumes the scene is flat) to transfer labels between views. It failed to provide significant gains, proving that 3D depth information is essential for matching pixels across wide baselines.
  • Common Failure Modes: The authors identified four primary challenges that multi-view data could help solve: self-occlusion, inter-object occlusion, small/distant objects, and specular ambiguities.
Fig. 3: Failure cases from monocular models. a) self-occlusion b) inter-object occlusion c) small objects d) ambiguity from specular inter-reflection. Last row shows a second view that could help solve the ambiguity.

Key Results

The authors established baselines using a U-Net with a ResNet50 backbone. The results (measured in Intersection over Union, or IoU) highlight the difficulty of cross-view transfer:

  • In-View Performance: When training and testing on the same camera angle, models achieved high scores (e.g., 80.55% IoU for the zenithal cam0).
  • Cross-View Drop: Transferring a model from a top view (cam0) to a low side view (cam13) caused performance to plummet to roughly 14% IoU.
  • Horizontal Invariance: Transferring between cameras at the same height (e.g., cam8 to cam12) showed much smaller drops, as the objects were placed with random rotations, making the model somewhat robust to horizontal orbiting.

Detailed IoU results for these experiments are available in Table 2:

Table 2: IoU results for direct cross-view semantic transfer. Five models trained on 100% of the train set (100k scenes).

Limitations and Future Work

While MVMO is a powerful tool, it has limitations:

  1. Synthetic Gap: As a synthetic dataset, models trained on MVMO may require domain adaptation to work on real-world images.
  2. Material Uniformity: Objects use "Uniform Object Materials," meaning a single material is applied to an entire shape, which is simpler than real-world multi-material objects.
  3. Static Scenes: The dataset does not currently account for temporal dynamics or moving objects.

The authors suggest that future research should focus on developing new theoretical approaches for unsupervised cross-view transfer and 3D-aware fusion to better exploit the complementary information MVMO provides.

Summary updated

Abstract

We present MVMO (Multi-View, Multi-Object dataset): a synthetic dataset of 116,000 scenes containing randomly placed objects of 10 distinct classes and captured from 25 camera locations in the upper hemisphere. MVMO comprises photorealistic, path-traced image renders, together with semantic segmentation ground truth for every view. Unlike existing multi-view datasets, MVMO features wide baselines between cameras and high density of objects, which lead to large disparities, heavy occlusions and view-dependent object appearance. Single view semantic segmentation is hindered by self and inter-object occlusions that could benefit from additional viewpoints. Therefore, we expect that MVMO will propel research in multi-view semantic segmentation and cross-view semantic transfer. We also provide baselines that show that new research is needed in such fields to exploit the complementary information of multi-view setups.
BibTeX
@inproceedings{alvarezgila2022mvmo,
  author = {Aitor Alvarez-Gila and Joost van de Weijer and Yaxing Wang and Estíbaliz Garrote},
  title = {{MVMO: A Multi-Object Dataset for Wide Baseline Multi-View Semantic Segmentation}},
  booktitle = {ICIP},
  year = {2022},
  url = {https://openreview.net/forum?id=D9BJY1K-pDN},
}

Loading related content...

Figure