Plan2Scene: Converting Floorplans to 3D Scenes

Madhawa Vidanapathirana, Qirui Wu, Yasutaka Furukawa, Angel X. Chang, Manolis Savva

Abstract

We address the task of converting a floorplan and a set of associated photos of a residence into a textured 3D mesh model, a task which we call Plan2Scene. Our system 1) lifts a floorplan image to a 3D mesh model; 2) synthesizes surface textures based on the input photos; and 3) infers textures for unobserved surfaces using a graph neural network architecture. To train and evaluate our system we create indoor surface texture datasets, and augment a dataset of floorplans and photos from prior work with rectified surface crops and additional annotations. Our approach handles the challenge of producing tileable textures for dominant surfaces such as floors, walls, and ceilings from a sparse set of unaligned photos that only partially cover the residence. Qualitative and quantitative evaluations show that our system produces realistic 3D interior models, outperforming baseline approaches on a suite of texture quality metrics and as measured by a holistic user study.

1. Introduction

Creating digital 3D replicas of existing homes is a complex task. While professional photogrammetry and depth-sensing hardware can produce high-quality results, they require expert operation and exhaustive photo coverage. In contrast, most residential data available online—such as on real estate platforms like Zillow or Redfin—consists only of a 2D floorplan and a few wide-angle photos.

Overview of the Plan2Scene pipeline, converting a 2D floorplan and unlocalized photos into a textured 3D mesh.

This "sparse" data presents three major challenges: missing camera poses, incomplete coverage (many surfaces are never seen), and lighting inconsistencies across photos. Plan2Scene addresses these by treating photos as hints for material synthesis rather than direct texture maps.

2. Methodology

The Plan2Scene pipeline is divided into three primary stages: geometry lifting, texture synthesis, and material propagation.

2.1 Geometry Lifting

The system first converts the 2D floorplan into a 3D "shell" consisting of walls, floors, and ceilings. Common 3D objects, such as toilets and sinks, are placed based on icons identified in the floorplan.

2.2 Neural Texture Synthesis

Instead of simple projection, the system extracts "rectified" square patches from input photos using semantic segmentation and plane detection. These crops are processed by an encoder-decoder network to learn a "texture embedding." This allows the generation of tileable, seamless textures free of lighting artifacts. A key innovation is the separation of median color (in HSV space) from the texture pattern, enabling a single model to handle diverse materials. To ensure quality, a VGG-based "textureness score" filters out noisy crops (e.g., those with heavy shadows or reflections).

2.3 GNN Propagation

In a typical residence, approximately 60% of surfaces are not visible in any input photo. Plan2Scene uses a Graph Neural Network (GNN) to propagate texture information across the residence. By treating rooms as nodes and doors as edges, the system ensures stylistic consistency (e.g., matching flooring in adjacent rooms) based on the observed "theme" of the home.

3. Results and Evaluation

Plan2Scene was evaluated against several baselines, including direct cropping and texture retrieval.

Comparison between the baseline Rent3D (a) and the proposed Plan2Scene (b) approach, highlighting the improved texture quality and completeness.

3.1 Quantitative and Qualitative Performance

Quantitatively, Plan2Scene achieved superior scores in FID (distribution similarity) and TILE (seamlessness). Qualitatively, the synthesis method avoids the tiling artifacts common in simpler methods and maintains higher color accuracy. In a forced-choice user study with 18 participants, Plan2Scene's results were preferred approximately 70% of the time over baseline approaches.

3.2 Limitations

Despite its success, the system faces challenges with segmentation errors (e.g., misidentifying a rug as the floor), global illumination shifts in photos, and complex geometric patterns like ornate mosaics.

4. Conclusion

Plan2Scene demonstrates a robust framework for generating realistic 3D interiors from sparse, unlocalized data. Future work will focus on generating more complex architectural details and improving robustness to varying lighting conditions across residences.

Summary updated

Abstract

We address the task of converting a floorplan and a set of associated photos of a residence into a textured 3D mesh model, a task which we call Plan2Scene. Our system 1) lifts a floorplan image to a 3D mesh model; 2) synthesizes surface textures based on the input photos; and 3) infers textures for unobserved surfaces using a graph neural network architecture. To train and evaluate our system we create indoor surface texture datasets, and augment a dataset of floorplans and photos from prior work with rectified surface crops and additional annotations. Our approach handles the challenge of producing tileable textures for dominant surfaces such as floors, walls, and ceilings from a sparse set of unaligned photos that only partially cover the residence. Qualitative and quantitative evaluations show that our system produces realistic 3D interior models, outperforming baseline approaches on a suite of texture quality metrics and as measured by a holistic user study.
BibTeX
@inproceedings{vidanapathirana2021plan,
  author = {Madhawa Vidanapathirana and Qirui Wu and Yasutaka Furukawa and Angel X. Chang and Manolis Savva},
  title = {{Plan2Scene: Converting Floorplans to 3D Scenes}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=v595kTODVIx},
}
Figure