TL;DR
MuSHRoom is a new multi-sensor room-scale dataset designed to bridge the gap between 3D geometric reconstruction and photorealistic novel view synthesis (NVS). By providing consumer-grade RGB-D data (from Kinects and iPhones) alongside high-precision laser-scanned ground truth meshes, it challenges current algorithms to perform both tasks well under real-world conditions. The authors also introduce a more rigorous evaluation protocol that mimics actual AR/VR use cases by testing models on entirely independent camera trajectories.
What Problem Does This Paper Address?
To build immersive metaverse, AR, and VR experiences, algorithms must understand both the structure of a room (3D reconstruction) and how it looks from any angle (novel view synthesis). Historically, the computer vision community has treated these as separate problems.
As a result, existing datasets are fragmented: some provide excellent imagery for view synthesis but lack ground-truth 3D meshes, while others provide 3D scans but use synthetic, over-cleaned data. Furthermore, standard evaluation protocols for view synthesis usually involve uniformly sampling test frames from a single video sequence. This doesn't match real-world AR/VR applications, where a user might first scan a room and later view it from completely different, unmapped positions and angles.
To address these gaps, the authors introduce MuSHRoom (Multi-Sensor Hybrid Room Dataset) to force a unified, robust approach to both geometry and photorealism.
Methods
The authors developed a comprehensive data collection and post-processing pipeline to capture 10 diverse real-world indoor environments.
+-------------------------------------------------------------------+
| Data Capture Modes |
+---------------------------------+---------------------------------+
| Consumer Tech | Industrial Tech |
| (Kinect / iPhone 12 Pro Max) | (Faro Focus 3D X130) |
| - Long sequence (training) | - High-precision point clouds |
| - Short sequence (testing) | - Expert-cleaned GT meshes |
+---------------------------------+---------------------------------+
Key methodological contributions include:
- Multi-Sensor Tracking: They used an Azure Kinect (fusing RGB-D and IMU data via the Spectacular AI SDK) and an iPhone 12 Pro Max (via Polycam and COLMAP) to capture consumer-grade data.
- A Realistic Evaluation Protocol: Instead of uniform frame splitting, they use the long video sequence for training and an entirely separate short sequence (with a different trajectory and camera angles) for testing NVS performance.
- Advanced Mesh Culling: To fairly evaluate 3D reconstruction in non-square or unbounded real-world rooms, they developed a new culling protocol that projects meshes onto the $xy$, $yz$, and $xz$ planes to cleanly trim away artifacts before computing geometric metrics.
Datasets
The primary contribution is the MuSHRoom dataset itself, which features:
- 10 real-world rooms with varying shapes, lighting conditions, and objects (e.g., classrooms, saunas, copy rooms).
- 40 total sequences (1 long and 1 short sequence per room, across 2 different consumer devices).
- Expert-cleaned ground-truth meshes generated via an industrial Faro Focus 3D X130 laser scanner.
Unlike synthetic datasets, MuSHRoom preserves real-world challenges like motion blur, specular reflections, transparent surfaces, and dramatic illumination changes.
Key Insights and Ideas
- The Geometry vs. Photorealism Trade-off: Off-the-shelf pipelines are heavily specialized. Radiance field methods (like NeRF variants) and 3D Gaussian Splatting (3DGS) yield stunning images but produce noisy, incomplete, or "hollow" 3D meshes. Conversely, Signed Distance Function (SDF) methods produce highly accurate geometry but sacrifice fine visual details.
- Trajectory Gaps Matter: Testing view synthesis on an independent trajectory (the authors' new protocol) causes a noticeable drop in rendering quality across all models. This highlights how fragile current NVS methods are when forced to generalize to unseen viewpoints.
- The Vulnerability of Depth Sensors: Real-world artifacts like reflections on TV screens or transparent glass doors break consumer depth sensors, leading to invalid depth data that severely degrades downstream 3D reconstruction.
Key Results
The authors benchmarked several prominent pipelines, including Nerfacto, Depth-Nerfacto, MonoSDF, and Splatfacto (a 3D Gaussian Splatting implementation).
- Novel View Synthesis: Splatfacto achieved the highest rendering quality. On Kinect data, it scored a 26.07 PSNR when tested within the same sequence. However, under the tougher cross-sequence protocol, its performance dropped to 23.28 PSNR.
- 3D Reconstruction: MonoSDF dominated geometric accuracy, achieving an F-score of 0.8753 on Kinect data. In comparison, the photorealistic Splatfacto lagged far behind with an F-score of just 0.4697, illustrating the severe geometric degradation in pure rendering pipelines.
Limitations and Future Work
While MuSHRoom provides an excellent testbed, the authors note that a truly unified end-to-end framework capable of delivering both perfect geometry and flawless photorealism on consumer hardware does not yet exist. Furthermore, handling severe real-world noise—such as extreme illumination changes in saunas or missing depth data on glass surfaces—remains an open problem for the community to solve using this benchmark.
Summary updated