Introduction
In the evolving landscape of urban augmented reality (AR) and 3D building reconstruction, precise spatial localization remains a significant hurdle. While current AR navigation systems are proficient at identifying specific buildings, they frequently struggle with vertical granularity—specifically, determining which floor a particular business or point of interest occupies. Recognizing "floor-level lines"—the horizontal boundaries separating adjacent floors—is a critical task for enabling floor-aware applications. However, this task is complicated by perspective distortion, environmental occlusions, and a lack of specialized training data.
This paper introduces FloorLevel-Net, a deep learning framework designed to detect and vertically order floor-level lines in real-world street-view imagery. By integrating a novel height-attention mechanism with geometry-aware post-processing, the system achieves high accuracy and real-time performance.
The Challenge of Vertical Localization
Detecting floor boundaries in the wild is difficult for three primary reasons:
- Data Scarcity: Existing facade datasets typically focus on semantic labels (e.g., "window," "door") or rectified, perfectly flat images. They lack the geometric floor-level annotations necessary for real-world perspective views.
- Perspective Distortion: Street-view images are rarely captured head-on. Buildings are viewed from various angles, causing parallel floor lines to converge toward vanishing points.
- Visual Clutter: Urban environments are filled with occlusions such as trees, lamp posts, and signage that obscure the structural lines of a building.
While traditional methods often rely on detecting repetitive patterns like window rows, these approaches fail when faced with irregular architectural designs or significant partial occlusions.
Methodology: FloorLevel-Net
FloorLevel-Net employs a two-stage approach to overcome these challenges, combining deep learning with classical geometric constraints.
Multi-task Learning with Height-Attention (HA)
The core of the framework is a Convolutional Neural Network (CNN) that performs multi-task learning. It simultaneously segments semantic building components and predicts the vertical distribution of floor lines.
The primary innovation is the Height-Attention (HA) mechanism. Recognizing that floors follow a strict vertical hierarchy (e.g., the second floor is always above the first), the HA layer encodes this spatial logic. It pools features horizontally and applies 1D convolutions to generate a vertical attention map. This forces the network to prioritize vertical ordering, allowing it to distinguish between different floor levels even in complex or cluttered scenes.
Geometry-Aware Post-processing
To refine the raw, often noisy output of the neural network, the system applies a geometric optimization stage. The detected pixels are grouped and fitted to polylines, which are then subjected to two physical constraints:
- Vanishing Point Constraint: All floor lines on a single facade must converge at the same 3D vanishing point.
- Order Constraint: Lines must maintain a consistent bottom-to-top sequence, preventing the model from producing logically impossible floor overlaps.
Data Strategy and Augmentation
To address the lack of labeled perspective data, the authors developed a synthetic-to-real transfer strategy. They utilized 150 rectified images from the CMP Facade Dataset and applied a "warping" scheme using building quads extracted from Google Street View (GSV). This resulted in 3,000 pairs of synthetic images that mimic realistic perspective distortions. The model was then evaluated on a diverse test set of 600 real-world images from sources including eTrims, arcDataset, and new captures from London and Hong Kong.
Experimental Results
FloorLevel-Net demonstrates significant improvements over standard segmentation baselines:
- Accuracy: On complex GSV images, the model achieved a pixel-wise accuracy of 0.605, more than doubling the performance of standard models like PSPNet (0.321) trained on the same data.
- Efficiency: The pipeline is highly optimized, processing images in approximately 0.19 seconds. This is nearly six times faster than traditional geometric filtering methods (~1.18s), making it suitable for real-time AR applications.
- Robustness: The integration of semantic context allows the network to "hallucinate" missing lines. For instance, if a tree obscures a floor boundary, the network uses the positions of windows above and below to infer the correct line placement.
Conclusion and Future Work
FloorLevel-Net provides a robust solution for floor-level line detection by treating verticality as a strong structural prior. While the system is highly effective, it faces limitations in scenarios involving extreme occlusion (e.g., a large bus blocking the ground floor) or highly reflective glass facades that confuse window detection.
Future research will explore the use of video sequences and Structure from Motion (SfM) to provide temporal consistency. By leveraging multiple frames, the system could achieve even greater stability and distance estimation for next-generation urban navigation and AR experiences.
Summary updated