Abstract
We address the task of converting a floorplan and a set of associated photos of a residence into a textured 3D mesh model, a task which we call Plan2Scene. Our system 1) lifts a floorplan image to a 3D mesh model; 2) synthesizes surface textures based on the input photos; and 3) infers textures for unobserved surfaces using a graph neural network architecture. To train and evaluate our system we create indoor surface texture datasets, and augment a dataset of floorplans and photos from prior work with rectified surface crops and additional annotations. Our approach handles the challenge of producing tileable textures for dominant surfaces such as floors, walls, and ceilings from a sparse set of unaligned photos that only partially cover the residence. Qualitative and quantitative evaluations show that our system produces realistic 3D interior models, outperforming baseline approaches on a suite of texture quality metrics and as measured by a holistic user study.
1. Introduction
Creating digital 3D replicas of existing homes is a complex task. While professional photogrammetry and depth-sensing hardware can produce high-quality results, they require expert operation and exhaustive photo coverage. In contrast, most residential data available online—such as on real estate platforms like Zillow or Redfin—consists only of a 2D floorplan and a few wide-angle photos.
This "sparse" data presents three major challenges: missing camera poses, incomplete coverage (many surfaces are never seen), and lighting inconsistencies across photos. Plan2Scene addresses these by treating photos as hints for material synthesis rather than direct texture maps.
2. Methodology
The Plan2Scene pipeline is divided into three primary stages: geometry lifting, texture synthesis, and material propagation.
2.1 Geometry Lifting
The system first converts the 2D floorplan into a 3D "shell" consisting of walls, floors, and ceilings. Common 3D objects, such as toilets and sinks, are placed based on icons identified in the floorplan.
2.2 Neural Texture Synthesis
Instead of simple projection, the system extracts "rectified" square patches from input photos using semantic segmentation and plane detection. These crops are processed by an encoder-decoder network to learn a "texture embedding." This allows the generation of tileable, seamless textures free of lighting artifacts. A key innovation is the separation of median color (in HSV space) from the texture pattern, enabling a single model to handle diverse materials. To ensure quality, a VGG-based "textureness score" filters out noisy crops (e.g., those with heavy shadows or reflections).
2.3 GNN Propagation
In a typical residence, approximately 60% of surfaces are not visible in any input photo. Plan2Scene uses a Graph Neural Network (GNN) to propagate texture information across the residence. By treating rooms as nodes and doors as edges, the system ensures stylistic consistency (e.g., matching flooring in adjacent rooms) based on the observed "theme" of the home.
3. Results and Evaluation
Plan2Scene was evaluated against several baselines, including direct cropping and texture retrieval.
3.1 Quantitative and Qualitative Performance
Quantitatively, Plan2Scene achieved superior scores in FID (distribution similarity) and TILE (seamlessness). Qualitatively, the synthesis method avoids the tiling artifacts common in simpler methods and maintains higher color accuracy. In a forced-choice user study with 18 participants, Plan2Scene's results were preferred approximately 70% of the time over baseline approaches.
3.2 Limitations
Despite its success, the system faces challenges with segmentation errors (e.g., misidentifying a rug as the floor), global illumination shifts in photos, and complex geometric patterns like ornate mosaics.
4. Conclusion
Plan2Scene demonstrates a robust framework for generating realistic 3D interiors from sparse, unlocalized data. Future work will focus on generating more complex architectural details and improving robustness to varying lighting conditions across residences.
Summary updated