Plan2Scene: Converting Floorplans to 3D Scenes

Madhawa Vidanapathirana, Qirui Wu, Yasutaka Furukawa, Angel X. Chang, Manolis Savva

Abstract

We address the task of converting a floorplan and a set of associated photos of a residence into a textured 3D mesh model, a task which we call Plan2Scene. Our system 1) lifts a floorplan image to a 3D mesh model; 2) synthesizes surface textures based on the input photos; and 3) infers textures for unobserved surfaces using a graph neural network architecture. To train and evaluate our system we create indoor surface texture datasets, and augment a dataset of floorplans and photos from prior work with rectified surface crops and additional annotations. Our approach handles the challenge of producing tileable textures for dominant surfaces such as floors, walls, and ceilings from a sparse set of unaligned photos that only partially cover the residence. Qualitative and quantitative evaluations show that our system produces realistic 3D interior models, outperforming baseline approaches on a suite of texture quality metrics and as measured by a holistic user study.

Introduction

Creating realistic 3D models of indoor spaces is notoriously difficult. While professional CAD tools exist, they require expert labor. Automated methods like photogrammetry or depth-sensing (LiDAR) often struggle with reflective surfaces, require exhaustive scanning of every corner, and produce "messy" geometry.

Plan2Scene identifies a massive, untapped data source: real estate listings. These typically include a 2D floorplan and a few wide-angle photos. However, using this data for 3D reconstruction is challenging because photos lack camera pose metadata, lighting varies wildly between rooms, and coverage is sparse—many walls, floors, or entire rooms are never photographed. Plan2Scene aims to produce "clean" architectural textures (floors, walls, ceilings) that look like actual materials rather than distorted snapshots.

Overview of the Plan2Scene pipeline, converting a floorplan and sparse photos into a textured 3D mesh.

Methodology

The Plan2Scene pipeline consists of three main stages:

1. Geometry Lifting

The system takes a vectorized floorplan and extrudes it into a 3D mesh. Common objects like doors and toilets are placed using rule-based defaults to provide a structural foundation.

2. Texture Synthesis for Observed Surfaces

For surfaces visible in photos, the system segments the image (e.g., identifying "floor" vs "wall"), rectifies the perspective to get a flat "crop," and feeds it into an encoder-decoder neural network. This network is trained to produce tileable textures—patterns that can be repeated infinitely without visible seams. A key innovation is the separation of color (HSV) from pattern, allowing the model to handle different colored versions of the same material (like painted plaster) more effectively.

3. Texture Propagation for Unobserved Surfaces

Since approximately 60% of surfaces in the dataset are unobserved, the system treats the house as a graph where rooms are nodes and doors are edges. A Graph Neural Network (GNN) propagates texture information across the house. If the living room has a specific wood floor, the GNN learns that the adjacent hallway likely shares that material, ensuring inter-room consistency based on architectural common sense.

Datasets

The researchers introduced and augmented several datasets to train their models:

  • Rent3D++: An extension of the Rent3D dataset with detailed room boundaries, photo-to-room assignments, and annotated icons for objects like sinks and stoves.
  • Stationary Textures Dataset: A curated collection of 516 high-quality texture exemplars (wood, carpet, tile, plaster) used to train the synthesis engine.
  • Substance-mapped Textures: 146 tileable textures used as a baseline for retrieval-based texturing.

Results and Evaluation

Plan2Scene was compared against several baselines, including direct photo tiling and texture retrieval.

Qualitative comparison between the Rent3D baseline and Plan2Scene's output.
  • Quantitative Performance: Plan2Scene outperformed baselines across a suite of metrics, including TILE (measuring texture seamlessness) and SUBS (measuring material type matching accuracy).
  • Qualitative Performance: In a user study with 18 participants, Plan2Scene's outputs were preferred approximately 70% of the time over the next best baseline. The GNN successfully maintained visual themes across houses, avoiding a "patchwork" appearance.

Conclusion and Future Work

Plan2Scene demonstrates the ability to "hallucinate" realistic, consistent textures for rooms and surfaces not captured in input photos. However, limitations remain. Semantic errors in initial segmentation can lead to incorrect material synthesis (e.g., mistaking a rug for the floor). The model also struggles with extreme white balance shifts and highly structured, non-repeating patterns. Future work may involve more dynamic object reconstruction from photos to replace current rule-based furniture placement.

Summary updated

Abstract

We address the task of converting a floorplan and a set of associated photos of a residence into a textured 3D mesh model, a task which we call Plan2Scene. Our system 1) lifts a floorplan image to a 3D mesh model; 2) synthesizes surface textures based on the input photos; and 3) infers textures for unobserved surfaces using a graph neural network architecture. To train and evaluate our system we create indoor surface texture datasets, and augment a dataset of floorplans and photos from prior work with rectified surface crops and additional annotations. Our approach handles the challenge of producing tileable textures for dominant surfaces such as floors, walls, and ceilings from a sparse set of unaligned photos that only partially cover the residence. Qualitative and quantitative evaluations show that our system produces realistic 3D interior models, outperforming baseline approaches on a suite of texture quality metrics and as measured by a holistic user study.
BibTeX
@inproceedings{vidanapathirana2021plan,
  author = {Madhawa Vidanapathirana and Qirui Wu and Yasutaka Furukawa and Angel X. Chang and Manolis Savva},
  title = {{Plan2Scene: Converting Floorplans to 3D Scenes}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=v595kTODVIx},
}
Figure