TL;DR
This paper introduces Top2Pano, a novel framework for generating high-quality 360° indoor panoramas from simple 2D top-down views. By combining volumetric occupancy estimation with a diffusion-based refinement process, Top2Pano achieves impressive geometric consistency and photorealism, even generalizing to schematic floorplans.
What Problem Does This Paper Address?
Creating realistic and geometrically accurate 360° indoor panoramas is a challenging task, especially when starting from limited information like 2D top-down views or floorplans. Existing methods often struggle with inferring 3D structure, maintaining consistency, and achieving photorealism. This gap limits applications in virtual reality, interior design, and robotics, where immersive visual experiences are crucial. The paper aims to bridge this gap by developing a method that can synthesize compelling indoor panoramas from these sparse inputs.
Methods
Top2Pano employs a three-stage pipeline:
- Volumetric Occupancy Estimation (OccRecon): The system first estimates the 3D volumetric occupancy of the indoor scene. It takes a top-down view and its segmentation (obtained using SAM) as input. This module is designed to infer spatial structures and layout configurations, even without explicit semantic information in the input. A diffusion-based encoder-decoder framework is used, leveraging 2D diffusion models for efficiency.
- Coarse Panorama Rendering: Using the estimated 3D occupancy map and a specified camera position, the system renders coarse depth and color panoramas. This is achieved through volumetric rendering for depth and direct projection of colors from the top-down view for the color panorama. Structural constraints on walls and floors are enforced to ensure geometric consistency.
- Photorealistic Panorama Synthesis (PanoGen): The coarse depth and color panoramas serve as conditions for a diffusion-based synthesis module, PanoGen, which is built upon ControlNet. This stage refines the panoramas, restoring fine details and ensuring photorealism while maintaining structural coherence. PanoGen also supports stylized synthesis, allowing for diverse design variations based on textual or image-based controls.
Datasets
The paper uses two datasets for training and evaluation:
- Matterport3D [3]: A widely used dataset for indoor scene understanding.
- Gibson [36]: Another dataset for indoor environments.
The authors generated top-down views from the 3D models in these datasets using Blender, rendering them with an orthographic camera to create inputs that resemble floorplans, thus enhancing generalization capabilities.
Key Insights and Ideas
- Leveraging Volumetric Occupancy for Structure: Inferring 3D volumetric occupancy from a 2D top-down view is a key step to overcome the inherent ambiguity and lack of depth information.
- Diffusion Models for Both Structure and Synthesis: The paper effectively uses diffusion models not only for the final photorealistic synthesis (PanoGen) but also for estimating the 3D occupancy (OccRecon), demonstrating their versatility.
- Semantic-Free Input for Generalization: By using segmentation from SAM and avoiding reliance on explicit semantic labels in the input, the model can generalize better to simpler inputs like schematic floorplans.
- Coarse-to-Fine Refinement: The two-stage approach (coarse rendering followed by diffusion-based refinement) effectively balances geometric accuracy with photorealistic detail.
- Stylized Synthesis with ControlNet: The integration of ControlNet allows for flexible style control, enabling users to generate panoramas with specific aesthetic themes.
Key Results
- State-of-the-Art Performance: Top2Pano outperforms baseline methods on both Matterport3D and Gibson datasets across PSNR, SSIM, FID, and LPIPS metrics. For instance, on Matterport3D, Top2Pano achieves the best FID (30.84) and LPIPS (0.6029) scores among the compared methods.
- Generalization to Floorplans: The model demonstrates strong generalization capabilities, producing high-quality, geometry-consistent panoramas from schematic and even hand-drawn floorplans.
- Stylized Synthesis: The framework successfully generates panoramas with diverse interior design aesthetics, guided by text or images.
- Ablation Study: Removing key components like coarse colored panoramas or structural reinforcement leads to noticeable degradation in image quality, furniture placement, and spatial understanding, highlighting the importance of each module.
Limitations and Future Work
- Failure Cases: The paper acknowledges failure cases, particularly with complex geometries like false ceilings, missing furniture (e.g., flat-screen TVs), and incorrect stair directions. These often stem from the inherent ambiguity of the 2D input, leading to hallucinations.
- Limited Vertical Field of View (FoV): The generated panoramas exhibit a limited vertical FoV, which is attributed to the constraints of the training data. The authors suggest that future work with datasets containing full vertical FoV panoramas could improve this aspect.
- Trade-off in Stylization: When using stylized synthesis, there's a trade-off between fidelity to the original top-down view and the desired style, especially when the input is a colored top-down view. Reducing the influence of the coarse color panorama can enhance stylization but might compromise realism.
Future work could focus on addressing these limitations, potentially by incorporating more diverse training data or exploring novel architectural designs that better handle complex vertical structures and fine details.
Summary updated