MegaSynth: Scaling Up 3D Scene Reconstruction with Synthesized Data

Hanwen Jiang, Zexiang Xu, Desai Xie, Ziwen Chen, Haian Jin, Fujun Luan, Zhixin Shu, Kai Zhang, Sai Bi, Xin Sun, Jiuxiang Gu, Qixing Huang, Georgios Pavlakos, Hao Tan

TL;DR

Researchers have introduced MegaSynth, a massive procedurally generated dataset of 700,000 3D scenes designed to scale up Large Reconstruction Models (LRMs). By abandoning the need for realistic "semantics" (like recognizable furniture) in favor of abstract geometric primitives, the authors generated a dataset 50 times larger than previous benchmarks, leading to significant gains in reconstruction quality and generalization across real-world domains.

What Problem Does This Paper Address?

While Large Reconstruction Models (LRMs) have revolutionized 3D object reconstruction, scaling them to full scenes remains a bottleneck. Real-world scene datasets are notoriously difficult to collect; the current gold standard, DL3DV, contains only 10,000 scenes, compared to nearly a million for object-level datasets like Objaverse.

Furthermore, real-world data often suffers from "suboptimal distribution"—limited camera motion, noisy content, and inaccurate annotations. Existing synthetic scene datasets attempt to be photorealistic, but modeling complex human-centric semantics (like how a chair relates to a table) makes them computationally expensive and difficult to scale. This paper asks: Do we actually need realistic objects to learn how to reconstruct 3D space?

Methods

The core innovation is the MegaSynth Generation Pipeline, which prioritizes geometric complexity over semantic realism.

  1. Non-Semantic Procedural Generation: Instead of placing 3D models of real objects, MegaSynth populates "room" boxes with random combinations of geometric primitives (cubes, spheres, cylinders, cones). These are augmented with height-fields to create concave/convex details and thin wireframe structures to simulate fine-grained geometry.
  2. Controllable Complexity: The authors heuristically regulate factors like camera pose distribution, material properties (increasing the probability of tricky specular and glass surfaces), and lighting.
  3. Lighting Diversity: Scenes are lit using a mix of ambient light, simulated sunlight (through randomized windows), and luminous objects/light bulbs to mimic complex real-world conditions.
  4. Mixed-Data Training: The authors propose two strategies: pre-training on MegaSynth followed by fine-tuning on real data, or joint training on both. They also introduce a Geometry Loss ($L_{loc}$), leveraging the perfect ground-truth depth available in synthetic data to supervise the center locations of predicted 3D Gaussians, which stabilizes training for wide-coverage scenes.
The MegaSynth generation pipeline creates abstract but geometrically rich scenes.

Datasets

  • MegaSynth (New): 700,000 synthesized non-semantic scenes.
  • DL3DV: The largest clean real-world scene dataset (10K scenes), used for training and evaluation.
  • Hypersim: A synthetic indoor dataset used for zero-shot generalization testing.
  • MipNeRF360 & Tanks & Temples (TT): Standard real-world benchmarks for out-of-domain evaluation.

Key Insights and Ideas

  • Reconstruction is Low-Level: The most striking finding is that models trained only on abstract shapes perform comparably to those trained on real data. This suggests that 3D reconstruction is fundamentally a low-level geometric task (matching pixels and structures) rather than a high-level semantic one.
  • Scale Trumps Realism: A massive, "ugly" synthetic dataset can be more effective than a small, "pretty" real one. The sheer scale (700K vs 10K) allows the model to see a much wider variety of geometric configurations.
  • Metadata is a Superpower: Unlike real-world captures, synthetic data provides "noise-free" camera poses and perfect depth maps. Using this metadata for direct geometric supervision ($L_{loc}$) was critical for preventing training instability in complex scenes.

Key Results

The inclusion of MegaSynth led to consistent improvements across all tested architectures (GS-LRM and Long-LRM):

  • Quantitative Gains: Models trained with MegaSynth showed a 1.2 to 1.8 dB PSNR improvement across diverse image domains.
  • Zero-Shot Generalization: The performance boost was often larger on out-of-domain datasets (like Hypersim) than on the in-domain DL3DV data, proving the dataset's ability to teach generalizable geometric priors.
  • Efficiency: The feed-forward models achieved these results in 0.1–0.3 seconds, representing a 2000x speedup over traditional optimization-based methods like 3DGS, which take several minutes per scene.
Figure 3. Reconstruction visualization on the in-domain DL3DV data. The results are from Long-LRM at resolution 256. We present both indoor and outdoor results in the first and second rows, respectively. With our MegaSynth (denoted as ‘w. MegaSynth’), the model performs better on thin structures (e.g., bottom left), complicated lighting (e.g., top middle), and cluttered scenes (e.g., top right).

Limitations and Future Work

While MegaSynth is a massive step forward, the authors note a few areas for improvement:

  • Semantic Gap: While semantics aren't required for reconstruction, there is still a distribution gap between primitive-based scenes and the real world. The model still struggles with extremely complex real-world geometry patterns that aren't well-represented by basic primitives.
  • Indoor Focus: MegaSynth is currently optimized for indoor-like "box" environments. Extending the procedural logic to handle vast outdoor landscapes (mountains, forests, cityscapes) is a logical next step.
  • Downstream Tasks: Preliminary results show MegaSynth helps with monocular depth estimation; exploring its utility for other 3D tasks like SLAM or navigation is a promising direction.

Summary updated

Abstract

We propose scaling up 3D scene reconstruction by training with synthesized data. At the core of our work is MegaSynth, a procedurally generated 3D dataset comprising 700K scenes - over 50 times larger than the prior real dataset DL3DV - dramatically scaling the training data. To enable scalable data generation, our key idea is eliminating semantic information, removing the need to model complex semantic priors such as object affordances and scene composition. Instead, we model scenes with basic spatial structures and geometry primitives, offering scalability. Besides, we control data complexity to facilitate training while loosely aligning it with real-world data distribution to benefit real-world generalization. We explore training LRMs with both MegaSynth and available real data. Experiment results show that joint training or pre-training with MegaSynth improves reconstruction quality by 1.2 to 1.8 dB PSNR across diverse image domains. Moreover, models trained solely on MegaSynth perform comparably to those trained on real data, underscoring the low-level nature of 3D reconstruction. Additionally, we provide an in-depth analysis of MegaSynth's properties for enhancing model capability, training stability, and generalization, as well as application to other tasks.
BibTeX
@inproceedings{jiang2025megasynth,
  author = {Hanwen Jiang and Zexiang Xu and Desai Xie and Ziwen Chen and Haian Jin and Fujun Luan and Zhixin Shu and Kai Zhang and Sai Bi and Xin Sun and Jiuxiang Gu and Qixing Huang and Georgios Pavlakos and Hao Tan},
  title = {{MegaSynth: Scaling Up 3D Scene Reconstruction with Synthesized Data}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=qc3aWHUA5b},
}
Figure