TL;DR
Researchers have introduced MegaSynth, a massive procedurally generated dataset of 700,000 3D scenes designed to scale up Large Reconstruction Models (LRMs). By abandoning the need for realistic "semantics" (like recognizable furniture) in favor of abstract geometric primitives, the authors generated a dataset 50 times larger than previous benchmarks, leading to significant gains in reconstruction quality and generalization across real-world domains.
What Problem Does This Paper Address?
While Large Reconstruction Models (LRMs) have revolutionized 3D object reconstruction, scaling them to full scenes remains a bottleneck. Real-world scene datasets are notoriously difficult to collect; the current gold standard, DL3DV, contains only 10,000 scenes, compared to nearly a million for object-level datasets like Objaverse.
Furthermore, real-world data often suffers from "suboptimal distribution"—limited camera motion, noisy content, and inaccurate annotations. Existing synthetic scene datasets attempt to be photorealistic, but modeling complex human-centric semantics (like how a chair relates to a table) makes them computationally expensive and difficult to scale. This paper asks: Do we actually need realistic objects to learn how to reconstruct 3D space?
Methods
The core innovation is the MegaSynth Generation Pipeline, which prioritizes geometric complexity over semantic realism.
- Non-Semantic Procedural Generation: Instead of placing 3D models of real objects, MegaSynth populates "room" boxes with random combinations of geometric primitives (cubes, spheres, cylinders, cones). These are augmented with height-fields to create concave/convex details and thin wireframe structures to simulate fine-grained geometry.
- Controllable Complexity: The authors heuristically regulate factors like camera pose distribution, material properties (increasing the probability of tricky specular and glass surfaces), and lighting.
- Lighting Diversity: Scenes are lit using a mix of ambient light, simulated sunlight (through randomized windows), and luminous objects/light bulbs to mimic complex real-world conditions.
- Mixed-Data Training: The authors propose two strategies: pre-training on MegaSynth followed by fine-tuning on real data, or joint training on both. They also introduce a Geometry Loss ($L_{loc}$), leveraging the perfect ground-truth depth available in synthetic data to supervise the center locations of predicted 3D Gaussians, which stabilizes training for wide-coverage scenes.
Datasets
- MegaSynth (New): 700,000 synthesized non-semantic scenes.
- DL3DV: The largest clean real-world scene dataset (10K scenes), used for training and evaluation.
- Hypersim: A synthetic indoor dataset used for zero-shot generalization testing.
- MipNeRF360 & Tanks & Temples (TT): Standard real-world benchmarks for out-of-domain evaluation.
Key Insights and Ideas
- Reconstruction is Low-Level: The most striking finding is that models trained only on abstract shapes perform comparably to those trained on real data. This suggests that 3D reconstruction is fundamentally a low-level geometric task (matching pixels and structures) rather than a high-level semantic one.
- Scale Trumps Realism: A massive, "ugly" synthetic dataset can be more effective than a small, "pretty" real one. The sheer scale (700K vs 10K) allows the model to see a much wider variety of geometric configurations.
- Metadata is a Superpower: Unlike real-world captures, synthetic data provides "noise-free" camera poses and perfect depth maps. Using this metadata for direct geometric supervision ($L_{loc}$) was critical for preventing training instability in complex scenes.
Key Results
The inclusion of MegaSynth led to consistent improvements across all tested architectures (GS-LRM and Long-LRM):
- Quantitative Gains: Models trained with MegaSynth showed a 1.2 to 1.8 dB PSNR improvement across diverse image domains.
- Zero-Shot Generalization: The performance boost was often larger on out-of-domain datasets (like Hypersim) than on the in-domain DL3DV data, proving the dataset's ability to teach generalizable geometric priors.
- Efficiency: The feed-forward models achieved these results in 0.1–0.3 seconds, representing a 2000x speedup over traditional optimization-based methods like 3DGS, which take several minutes per scene.
Limitations and Future Work
While MegaSynth is a massive step forward, the authors note a few areas for improvement:
- Semantic Gap: While semantics aren't required for reconstruction, there is still a distribution gap between primitive-based scenes and the real world. The model still struggles with extremely complex real-world geometry patterns that aren't well-represented by basic primitives.
- Indoor Focus: MegaSynth is currently optimized for indoor-like "box" environments. Extending the procedural logic to handle vast outdoor landscapes (mountains, forests, cityscapes) is a logical next step.
- Downstream Tasks: Preliminary results show MegaSynth helps with monocular depth estimation; exploring its utility for other 3D tasks like SLAM or navigation is a promising direction.
Summary updated