From Programs to Poses: Factored Real-World Scene Generation via Learned Program Libraries

Joy Hsu, Emily Jin, Jiajun Wu, Niloy J. Mitra

Abstract

Real-world scenes, such as those in ScanNet, are difficult to capture, with highly limited data available. Generating realistic scenes with varied object poses remains an open and challenging task. In this work, we propose FactoredScenes, a framework that synthesizes realistic 3D scenes by leveraging the underlying structure of rooms while learning the variation of object poses from lived-in scenes. We introduce a factored representation that decomposes scenes into hierarchically organized concepts of room programs and object poses. To encode structure, FactoredScenes learns a library of functions capturing reusable layout patterns from which scenes are drawn, then uses large language models to generate high-level programs, regularized by the learned library. To represent scene variations, FactoredScenes learns a program-conditioned model to hierarchically predict object poses, and retrieves and places 3D objects in a scene. We show that FactoredScenes generates realistic, real-world rooms that are difficult to distinguish from real ScanNet scenes.

Introduction

Generating realistic 3D scenes requires a delicate balance between structural integrity and natural variation. While a room must follow functional blueprints (e.g., chairs surrounding a table), it must also exhibit the "lived-in" quality of real-world environments, where objects are rarely perfectly aligned to a grid. Most existing AI models struggle with this duality, often producing either overly rigid CAD-like layouts or chaotic arrangements where furniture placement lacks common sense.

FactoredScenes decomposes 3D scenes into high-level programs and low-level object poses.

FactoredScenes addresses this by introducing a factored representation that separates high-level structural "programs" from low-level "poses." This allows the model to learn structural regularities from clean, synthetic data while capturing realistic object orientations from noisy, real-world scans.

The Challenge of Real-World Scene Generation

Creating high-quality 3D scene datasets is labor-intensive. While 2D image datasets are vast, 3D datasets like ScanNet are relatively small and contain "noisy" real-world data where objects are subtly rotated and shifted. Existing generative models typically treat scenes as collections of independent bounding boxes, leading to two primary failures:

  1. Structural Incoherence: Models often fail to maintain functional relationships, such as nightstands being placed far from beds.
  2. Axis-Alignment Bias: Many models are restricted to predicting boxes aligned strictly to X and Y axes, failing to capture the natural rotations that make a room look authentic.

Methodology: The FactoredScenes Pipeline

The core of FactoredScenes is its hierarchical decomposition of the scene generation process. The probability of a scene is factored into a sequence of conditional steps:

The FactoredScenes pipeline: from learning a library of layout patterns to generating programs, layouts, and final realistic scenes.

1. Library Learning

Using a "wake-sleep" algorithm inspired by DreamCoder, the system analyzes professionally designed synthetic scenes (3D-Front) to learn a library of reusable functions. Instead of treating a dining set as five independent objects, it learns a cluster_placement function that captures the inherent relationship between a table and its chairs.

2. Program Generation

A Large Language Model (LLM) leverages this learned library to synthesize high-level programs for new rooms. By using a library of spatial functions, the LLM can focus on high-level reasoning and common-sense layout rules without needing to predict precise geometric coordinates directly.

3. Hierarchical Pose Prediction

To capture "lived-in" variations, FactoredScenes employs a program-conditioned model to predict object poses. This model is hierarchical: it first predicts the pose of a "primary" object (e.g., a table) and then predicts the poses of "dependent" objects (e.g., chairs) relative to the primary one. This captures the relational nature of object orientation in real-world settings.

4. Object Retrieval and Placement

Finally, the system retrieves 3D meshes from ShapeNet that match the predicted dimensions and places them according to the generated poses to complete the scene.

Experimental Results

FactoredScenes was evaluated using datasets including 3D-Front for structural learning, ScanNet for pose variation, and ShapeNet for object geometry.

  • Realism: On the Fréchet Inception Distance (FID) metric, FactoredScenes outperformed state-of-the-art models by 38.3% for bedrooms and 40.1% for living rooms.
  • Library Utility: The learned library provided a 644.1% relative improvement in function reuse compared to standard LLM-only approaches, demonstrating the effectiveness of learning domain-specific abstractions.
  • Human Evaluation: In a perceptual study, human participants could only distinguish real ScanNet scenes from FactoredScenes generations 67% of the time, highlighting the high degree of visual realism achieved.

Conclusion

FactoredScenes demonstrates that factoring scene generation into structural programs and realistic poses is a powerful approach for synthesizing complex 3D environments. By combining the reasoning capabilities of LLMs with learned spatial libraries and hierarchical pose models, the framework produces scenes that are both structurally sound and naturally varied. Future work will explore improving LLM placement logic and incorporating human-in-the-loop feedback for more complex scene synthesis.

Summary updated

Abstract

Real-world scenes, such as those in ScanNet, are difficult to capture, with highly limited data available. Generating realistic scenes with varied object poses remains an open and challenging task. In this work, we propose FactoredScenes, a framework that synthesizes realistic 3D scenes by leveraging the underlying structure of rooms while learning the variation of object poses from lived-in scenes. We introduce a factored representation that decomposes scenes into hierarchically organized concepts of room programs and object poses. To encode structure, FactoredScenes learns a library of functions capturing reusable layout patterns from which scenes are drawn, then uses large language models to generate high-level programs, regularized by the learned library. To represent scene variations, FactoredScenes learns a program-conditioned model to hierarchically predict object poses, and retrieves and places 3D objects in a scene. We show that FactoredScenes generates realistic, real-world rooms that are difficult to distinguish from real ScanNet scenes.
BibTeX
@inproceedings{hsu2025from,
  author = {Joy Hsu and Emily Jin and Jiajun Wu and Niloy J. Mitra},
  title = {{From Programs to Poses: Factored Real-World Scene Generation via Learned Program Libraries}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=Ew8bJkSt3g},
}
Figure