Abstract
Recovering the 3D layout of a room from a single panorama is a significant challenge in computer vision, primarily due to scale ambiguity in standard central projections. This paper introduces a novel method to recover the full 3D layout—including absolute scale—from a single non-central panorama. By integrating deep learning for feature extraction with new linear geometric solvers, the proposed approach eliminates the need for prior assumptions like camera height. The method achieves state-of-the-art performance in both Manhattan and Atlanta world environments, providing a robust solution for metric 3D reconstruction from a single viewpoint.
Introduction
In computer vision, "layout recovery" involves transforming a 2D panorama into a 3D model of a room's structural boundaries (walls, floor, and ceiling). While existing deep learning models like HorizonNet or AtlantaNet excel at identifying room shapes, they suffer from fundamental scale ambiguity. In a standard central panorama, where all light rays pass through a single point, a small room close to the camera is indistinguishable from a larger room further away. Consequently, traditional methods require external information, such as the camera's height from the floor, to produce metric measurements.
This research addresses the challenge of recovering absolute scale without prior knowledge by revisiting non-central panoramas. Unlike central projections, non-central images involve light rays that do not meet at a single point. These images contain subtle geometric distortions that encode the missing scale information, allowing the system to "triangulate" distances from a single viewpoint.
Methodology: A Two-Stage Pipeline
The authors propose a hybrid pipeline that combines the pattern recognition capabilities of deep learning with the mathematical rigor of classical geometry.
1. Neural Network as Line Extractor
The first stage adapts the architecture of HorizonNet, utilizing a ResNet-50 backbone followed by RNN/LSTM layers. The network is trained to predict the pixel positions of ceiling-wall and floor-wall boundaries, as well as the probability of wall-wall intersections (corners) for every column in the panorama. This stage effectively converts the raw image into structured geometric features.
2. Geometric Solvers
The core innovation lies in the subsequent geometric solvers. Once the network identifies the structural lines, the authors employ two new Direct Linear Transformation (DLT)-like solvers to reconstruct the 3D geometry. By representing 3D lines using Plücker coordinates, they derive linear equations that jointly solve for the room's height and the distance to each wall. This framework is flexible enough to handle both Manhattan worlds (orthogonal walls) and Atlanta worlds (vertical walls with arbitrary horizontal angles).
Datasets and Training
Since non-central projection datasets are not standard in the field, the authors generated a synthetic dataset. This dataset consists of approximately 650 unique room layouts (ranging from 6 to 10 walls) and over 2,500 synthetic images. Each image includes ground truth 3D layout information, enabling the network to learn the specific distortions inherent in non-central circular panoramas.
Key Results
The proposed method was evaluated against state-of-the-art (SOTA) models like HorizonNet and AtlantaNet. Because SOTA models typically provide only "up-to-scale" results, the comparison utilized 3D Intersection over Union (IoU) and absolute Corner Error (CE).
- Manhattan World: The method achieved a 93.88% 3D IoU, significantly outperforming AtlantaNet (83.94%) and HorizonNet (82.66%).
- Atlanta World: The method reached a 91.67% 3D IoU, compared to AtlantaNet's 80.01%.
- Absolute Scale Accuracy: In Manhattan environments, the Corner Error was only 0.223 meters, demonstrating the system's ability to recover physical dimensions with high precision from a single image.
Conclusion and Future Work
This work demonstrates that geometric distortion in non-central panoramas is a valuable feature for 3D reconstruction rather than a drawback. By breaking the reliance on camera height assumptions, this method paves the way for more autonomous robotic mapping. Future research will focus on bridging the gap between synthetic and real-world data, as well as optimizing the computational complexity of non-central modeling for real-time applications in specialized robotic sensors and catadioptric camera systems.
Summary updated