FloorLevel-Net: Recognizing Floor-Level Lines With Height-Attention-Guided Multi-Task Learning

Mengyang Wu, Wei Zeng, Chi-Wing Fu

Introduction

In the evolving landscape of urban augmented reality (AR) and 3D building reconstruction, precise spatial localization remains a significant hurdle. While current AR navigation systems are proficient at identifying specific buildings, they frequently struggle with vertical granularity—specifically, determining which floor a particular business or point of interest occupies. Recognizing "floor-level lines"—the horizontal boundaries separating adjacent floors—is a critical task for enabling floor-aware applications. However, this task is complicated by perspective distortion, environmental occlusions, and a lack of specialized training data.

This paper introduces FloorLevel-Net, a deep learning framework designed to detect and vertically order floor-level lines in real-world street-view imagery. By integrating a novel height-attention mechanism with geometry-aware post-processing, the system achieves high accuracy and real-time performance.

The Challenge of Vertical Localization

Detecting floor boundaries in the wild is difficult for three primary reasons:

  1. Data Scarcity: Existing facade datasets typically focus on semantic labels (e.g., "window," "door") or rectified, perfectly flat images. They lack the geometric floor-level annotations necessary for real-world perspective views.
  2. Perspective Distortion: Street-view images are rarely captured head-on. Buildings are viewed from various angles, causing parallel floor lines to converge toward vanishing points.
  3. Visual Clutter: Urban environments are filled with occlusions such as trees, lamp posts, and signage that obscure the structural lines of a building.

While traditional methods often rely on detecting repetitive patterns like window rows, these approaches fail when faced with irregular architectural designs or significant partial occlusions.

Methodology: FloorLevel-Net

FloorLevel-Net employs a two-stage approach to overcome these challenges, combining deep learning with classical geometric constraints.

Multi-task Learning with Height-Attention (HA)

The core of the framework is a Convolutional Neural Network (CNN) that performs multi-task learning. It simultaneously segments semantic building components and predicts the vertical distribution of floor lines.

The primary innovation is the Height-Attention (HA) mechanism. Recognizing that floors follow a strict vertical hierarchy (e.g., the second floor is always above the first), the HA layer encodes this spatial logic. It pools features horizontally and applies 1D convolutions to generate a vertical attention map. This forces the network to prioritize vertical ordering, allowing it to distinguish between different floor levels even in complex or cluttered scenes.

Geometry-Aware Post-processing

To refine the raw, often noisy output of the neural network, the system applies a geometric optimization stage. The detected pixels are grouped and fitted to polylines, which are then subjected to two physical constraints:

  • Vanishing Point Constraint: All floor lines on a single facade must converge at the same 3D vanishing point.
  • Order Constraint: Lines must maintain a consistent bottom-to-top sequence, preventing the model from producing logically impossible floor overlaps.

Data Strategy and Augmentation

To address the lack of labeled perspective data, the authors developed a synthetic-to-real transfer strategy. They utilized 150 rectified images from the CMP Facade Dataset and applied a "warping" scheme using building quads extracted from Google Street View (GSV). This resulted in 3,000 pairs of synthetic images that mimic realistic perspective distortions. The model was then evaluated on a diverse test set of 600 real-world images from sources including eTrims, arcDataset, and new captures from London and Hong Kong.

Experimental Results

FloorLevel-Net demonstrates significant improvements over standard segmentation baselines:

  • Accuracy: On complex GSV images, the model achieved a pixel-wise accuracy of 0.605, more than doubling the performance of standard models like PSPNet (0.321) trained on the same data.
  • Efficiency: The pipeline is highly optimized, processing images in approximately 0.19 seconds. This is nearly six times faster than traditional geometric filtering methods (~1.18s), making it suitable for real-time AR applications.
  • Robustness: The integration of semantic context allows the network to "hallucinate" missing lines. For instance, if a tree obscures a floor boundary, the network uses the positions of windows above and below to infer the correct line placement.

Conclusion and Future Work

FloorLevel-Net provides a robust solution for floor-level line detection by treating verticality as a strong structural prior. While the system is highly effective, it faces limitations in scenarios involving extreme occlusion (e.g., a large bus blocking the ground floor) or highly reflective glass facades that confuse window detection.

Future research will explore the use of video sequences and Structure from Motion (SfM) to provide temporal consistency. By leveraging multiple frames, the system could achieve even greater stability and distance estimation for next-generation urban navigation and AR experiences.

Summary updated

Abstract

The ability to recognize the position and order of the floor-level lines that divide adjacent building floors can benefit many applications, for example, urban augmented reality (AR). This work tackles the problem of locating floor-level lines in street-view images, using a supervised deep learning approach. Unfortunately, very little data is available for training such a network $-$ current street-view datasets contain either semantic annotations that lack geometric attributes, or rectified facades without perspective priors. To address this issue, we first compile a new dataset and develop a new data augmentation scheme to synthesize training samples by harassing (i) the rich semantics of existing rectified facades and (ii) perspective priors of buildings in diverse street views. Next, we design FloorLevel-Net, a multi-task learning network that associates explicit features of building facades and implicit floor-level lines, along with a height-attention mechanism to help enforce a vertical ordering of floor-level lines. The generated segmentations are then passed to a second-stage geometry post-processing to exploit self-constrained geometric priors for plausible and consistent reconstruction of floor-level lines. Quantitative and qualitative evaluations conducted on assorted facades in existing datasets and street views from Google demonstrate the effectiveness of our approach. Also, we present context-aware image overlay results and show the potentials of our approach in enriching AR-related applications.
BibTeX
@article{wu2021floorlevel,
  author = {Mengyang Wu and Wei Zeng and Chi-Wing Fu},
  title = {{FloorLevel-Net: Recognizing Floor-Level Lines With Height-Attention-Guided Multi-Task Learning}},
  journal = {IEEE Trans. Image Process.},
  year = {2021},
  doi = {10.1109/tip.2021.3096090},
}
Figure