MuSHRoom: Multi-Sensor Hybrid Room Dataset for Joint 3D Reconstruction and Novel View Synthesis

Xuqian Ren, Wenjia Wang, Dingding Cai, Tuuli Tuominen, Juho Kannala, Esa Rahtu

TL;DR

MuSHRoom is a new multi-sensor room-scale dataset designed to bridge the gap between 3D geometric reconstruction and photorealistic novel view synthesis (NVS). By providing consumer-grade RGB-D data (from Kinects and iPhones) alongside high-precision laser-scanned ground truth meshes, it challenges current algorithms to perform both tasks well under real-world conditions. The authors also introduce a more rigorous evaluation protocol that mimics actual AR/VR use cases by testing models on entirely independent camera trajectories.

What Problem Does This Paper Address?

To build immersive metaverse, AR, and VR experiences, algorithms must understand both the structure of a room (3D reconstruction) and how it looks from any angle (novel view synthesis). Historically, the computer vision community has treated these as separate problems.

As a result, existing datasets are fragmented: some provide excellent imagery for view synthesis but lack ground-truth 3D meshes, while others provide 3D scans but use synthetic, over-cleaned data. Furthermore, standard evaluation protocols for view synthesis usually involve uniformly sampling test frames from a single video sequence. This doesn't match real-world AR/VR applications, where a user might first scan a room and later view it from completely different, unmapped positions and angles.

To address these gaps, the authors introduce MuSHRoom (Multi-Sensor Hybrid Room Dataset) to force a unified, robust approach to both geometry and photorealism.

Methods

The authors developed a comprehensive data collection and post-processing pipeline to capture 10 diverse real-world indoor environments.

+-------------------------------------------------------------------+
|                        Data Capture Modes                         |
+---------------------------------+---------------------------------+
|          Consumer Tech          |        Industrial Tech          |
|  (Kinect / iPhone 12 Pro Max)   |     (Faro Focus 3D X130)        |
|  - Long sequence (training)     |  - High-precision point clouds  |
|  - Short sequence (testing)     |  - Expert-cleaned GT meshes     |
+---------------------------------+---------------------------------+

Key methodological contributions include:

  • Multi-Sensor Tracking: They used an Azure Kinect (fusing RGB-D and IMU data via the Spectacular AI SDK) and an iPhone 12 Pro Max (via Polycam and COLMAP) to capture consumer-grade data.
  • A Realistic Evaluation Protocol: Instead of uniform frame splitting, they use the long video sequence for training and an entirely separate short sequence (with a different trajectory and camera angles) for testing NVS performance.
  • Advanced Mesh Culling: To fairly evaluate 3D reconstruction in non-square or unbounded real-world rooms, they developed a new culling protocol that projects meshes onto the $xy$, $yz$, and $xz$ planes to cleanly trim away artifacts before computing geometric metrics.

Datasets

The primary contribution is the MuSHRoom dataset itself, which features:

  • 10 real-world rooms with varying shapes, lighting conditions, and objects (e.g., classrooms, saunas, copy rooms).
  • 40 total sequences (1 long and 1 short sequence per room, across 2 different consumer devices).
  • Expert-cleaned ground-truth meshes generated via an industrial Faro Focus 3D X130 laser scanner.

Unlike synthetic datasets, MuSHRoom preserves real-world challenges like motion blur, specular reflections, transparent surfaces, and dramatic illumination changes.

Overview of the MuSHRoom dataset capturing setup and rooms

Key Insights and Ideas

  • The Geometry vs. Photorealism Trade-off: Off-the-shelf pipelines are heavily specialized. Radiance field methods (like NeRF variants) and 3D Gaussian Splatting (3DGS) yield stunning images but produce noisy, incomplete, or "hollow" 3D meshes. Conversely, Signed Distance Function (SDF) methods produce highly accurate geometry but sacrifice fine visual details.
  • Trajectory Gaps Matter: Testing view synthesis on an independent trajectory (the authors' new protocol) causes a noticeable drop in rendering quality across all models. This highlights how fragile current NVS methods are when forced to generalize to unseen viewpoints.
  • The Vulnerability of Depth Sensors: Real-world artifacts like reflections on TV screens or transparent glass doors break consumer depth sensors, leading to invalid depth data that severely degrades downstream 3D reconstruction.

Key Results

The authors benchmarked several prominent pipelines, including Nerfacto, Depth-Nerfacto, MonoSDF, and Splatfacto (a 3D Gaussian Splatting implementation).

  • Novel View Synthesis: Splatfacto achieved the highest rendering quality. On Kinect data, it scored a 26.07 PSNR when tested within the same sequence. However, under the tougher cross-sequence protocol, its performance dropped to 23.28 PSNR.
  • 3D Reconstruction: MonoSDF dominated geometric accuracy, achieving an F-score of 0.8753 on Kinect data. In comparison, the photorealistic Splatfacto lagged far behind with an F-score of just 0.4697, illustrating the severe geometric degradation in pure rendering pipelines.
Qualitative comparison of different methods on the MuSHRoom dataset

Limitations and Future Work

While MuSHRoom provides an excellent testbed, the authors note that a truly unified end-to-end framework capable of delivering both perfect geometry and flawless photorealism on consumer hardware does not yet exist. Furthermore, handling severe real-world noise—such as extreme illumination changes in saunas or missing depth data on glass surfaces—remains an open problem for the community to solve using this benchmark.

Summary updated

Abstract

Metaverse technologies demand accurate, real-time, and immersive modeling on consumer-grade hardware for both non-human perception (e.g., drone/robot/autonomous car navigation) and immersive technologies like AR/VR, requiring both structural accuracy and photorealism. However, there exists a knowledge gap in how to apply geometric reconstruction and photorealism modeling (novel view synthesis) in a unified framework. To address this gap and promote the development of robust and immersive modeling and rendering with consumer-grade devices, we propose a real-world Multi-Sensor Hybrid Room Dataset (MuSHRoom). Our dataset presents exciting challenges and requires state-of-the-art methods to be cost-effective, robust to noisy data and devices, and can jointly learn 3D reconstruction and novel view synthesis instead of treating them as separate tasks, making them ideal for real-world applications. We benchmark several famous pipelines on our dataset for joint 3D mesh reconstruction and novel view synthesis. Our dataset and benchmark show great potential in promoting the improvements for fusing 3D reconstruction and high-quality rendering in a robust and computationally efficient end-to-end fashion. The dataset and code are available at the project website: https://xuqianren.github.io/publications/MuSHRoom/.
BibTeX
@inproceedings{ren2024mushroom,
  author = {Xuqian Ren and Wenjia Wang and Dingding Cai and Tuuli Tuominen and Juho Kannala and Esa Rahtu},
  title = {{MuSHRoom: Multi-Sensor Hybrid Room Dataset for Joint 3D Reconstruction and Novel View Synthesis}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00445},
}
Figure