Introduction
Reconstructing a 3D room layout from a single 360° equirectangular image is a fundamental task in computer vision, typically approached via either 1D regression or 2D segmentation. While 1D regression methods like HorizonNet achieve high accuracy for visible areas, they require massive decoders and struggle with occlusions. Conversely, 2D segmentation methods handle occlusions better but rely on non-differentiable post-processing that sacrifices geometric precision.
Seg2Reg introduces a differentiable bridge between these two paradigms. By treating the floor plan as a 2D density field and employing "flattened" volume rendering—a technique inspired by Neural Radiance Fields (NeRF)—the model achieves state-of-the-art accuracy. Remarkably, it does so with a decoder nearly 1,000 times smaller than previous top-performing models, effectively resolving the disconnection between training objectives and geometric goals.
Methodology
The core innovation of this work is the Seg2Reg rendering layer, which transforms 2D segmentation-like predictions into precise 1D geometric layouts through a differentiable process.
Density Field Representation and Rendering
Instead of standard pixel-wise classification, the network predicts a 2D "density logit map." In this representation, high density corresponds to walls or areas outside the room, while low density indicates the room's interior. To determine the layout boundary, the model casts rays from the camera origin across this 2D density map.
As shown in the rendering process, the system uses alpha-blending to calculate the expected distance where a ray "hits" a wall. Because this process is entirely differentiable, the 2D density map is trained specifically to optimize the final 1D layout accuracy. This mathematical "decoder" replaces the need for millions of parameters found in traditional regression decoders, allowing for an ultra-lightweight MLP-based design (0.015M parameters vs. 13M-92M in LGT-Net).
Occlusion Awareness and LayoutWarp
To handle hidden corners, Seg2Reg samples "secondary views" from within the predicted room. These views allow the model to see behind obstacles, merging multiple perspectives into a complete floor plan. Additionally, the authors introduce LayoutWarp, a 3D-aware data augmentation technique. Unlike standard 2D transforms that break panoramic geometry, LayoutWarp performs transformations in 3D space (e.g., perturbing wall corners) and warps the 2D image accordingly, enhancing generalization to irregular, non-Manhattan rooms.
Experiments and Results
The model was evaluated using the PanoLayoutStudio codebase across four major benchmarks: MatterportLayout, Zillow Indoor (ZInd), PanoContext, and Stanford2D3D.
Seg2Reg achieved state-of-the-art performance, notably reaching a 3D IoU of 82.83% on the MatterportLayout test set, surpassing the previous best of 81.55% (LGT-Net). Beyond accuracy, the model demonstrated extreme efficiency and robustness. In cross-dataset evaluations, models trained with LayoutWarp showed significantly fewer geometric failures when encountering non-right-angled walls, effectively overcoming the common "Manhattan bias."
Discussion and Conclusion
The success of Seg2Reg highlights that differentiability is the "missing piece" in room reconstruction. By making the rendering process differentiable, loss can flow directly from 3D geometry back to 2D pixel predictions. This shift allows the "decoder" to be defined by mathematical projection rather than learned parameters.
While highly effective, the method currently depends on equirectangular projections and the quality of secondary view sampling. Future work may involve integrating Seg2Reg with NeRF-based mesh reconstructions or exploring few-shot learning to improve performance in data-scarce environments. Overall, Seg2Reg provides a highly efficient and accurate framework for 360° room layout reconstruction.
Summary updated