3D Indoor Scene Generation via Diffusion with User Interaction Awareness

Misong Kim, SeoungJae Jeong, Hyoseok Hwang, HyeongYeop KangJournal of the Korea Computer Graphics Society 2025 · OpenReview

Introduction

While modern generative AI has made significant strides in creating visually stunning 3D room layouts, these models often overlook a fundamental requirement: human utility. Current 3D indoor scene synthesis, which has evolved from simple Variational Autoencoders (VAEs) to sophisticated Diffusion models, excels at capturing the statistical distribution of furniture—placing beds against walls or rugs on floors. However, they frequently lack "interaction awareness."

A common failure in AI-generated rooms is the placement of objects in functionally inaccessible locations, such as a coffee table placed too far from a sofa or a dining chair blocked by other furniture. This paper argues that for applications in VR, robotics, and architectural design, visual plausibility is insufficient; functional usability must be a core objective.

Methodology

The proposed framework builds upon DiffuScene, a state-of-the-art diffusion model, by introducing a "reachability-aware" guidance system that influences the model during the sampling process.

The proposed pipeline integrates a U-Net based diffusion process with an accessibility guidance module that provides gradients to refine object placement.

1. Human Pose Sampling

Using VPoser (a variational human pose prior) and SMPL-X (a holistic 3D human body model), the researchers sampled thousands of natural human poses, specifically focusing on sitting and standing positions to model typical indoor interactions.

2. Reachability Map Construction

The system quantifies the 3D space around a human body by calculating two primary factors:

  • Reach Frequency: The statistical likelihood of a hand reaching a specific voxel in space.
  • Pose Energy: The biomechanical effort required to maintain a specific pose.

By combining these, the authors created a 3D map where high-value regions represent locations that are both easy and ergonomic for a human to reach.

3. Diffusion Guidance

During the denoising steps of the Diffusion model, the system calculates a "reward" based on the reachability map for specific object pairs (e.g., chair-table). It then utilizes the gradient of this reward to adjust object placement in real-time, effectively "pulling" furniture into functional configurations without requiring the model to be retrained from scratch.

Experimental Results

The model was evaluated using the 3D-FRONT dataset, specifically focusing on complex interaction environments like the LivingRoom and LivingDiningRoom categories.

A comparison of reachability levels, showing how the guidance system optimizes the distance between a chair and a table for better human accessibility.

Quantitative Performance

The researchers compared their method against the standard DiffuScene baseline using three key metrics:

  • Accessibility Score: The proposed model achieved a score of 0.59, a significant improvement over the baseline's 0.44, indicating better alignment with human reach zones.
  • Spatial Consistency (IoU): The method demonstrated a lower Intersection over Union (0.0024 vs 0.0033), resulting in fewer physical collisions or overlaps between objects.
  • Visual Quality (FID): The FID score was slightly higher than the baseline (15.67 vs 15.10). This suggests a minor trade-off where the model occasionally prioritizes functional utility over the exact visual distribution of the training data, though the scenes remain highly realistic.

Discussion and Future Work

The core innovation of this work is treating human accessibility as a differentiable signal, turning a subjective design quality into a mathematical optimization constraint. This "plug-and-play" guidance is computationally efficient and adaptable to various base models.

However, limitations remain. The current reachability maps are based on static poses, whereas real human interaction is dynamic (involving walking or leaning). Furthermore, the study focused on common furniture pairs; future work aims to generalize these interaction rules to more complex scenarios, such as opening cabinets or interacting with kitchen appliances, and to enable the AI to autonomously learn which objects require reachability constraints.

Summary updated

BibTeX
@article{kim2025d,
  author = {Misong Kim and SeoungJae Jeong and Hyoseok Hwang and HyeongYeop Kang},
  title = {{3D Indoor Scene Generation via Diffusion with User Interaction Awareness}},
  journal = {Journal of the Korea Computer Graphics Society},
  year = {2025},
  url = {https://openreview.net/forum?id=aQwHlxJTnb},
}
Figure