Abstract
Extracting building footprints from aerial images is essential for precise urban mapping with photogrammetric computer vision technologies. Existing approaches mainly assume that the roof and footprint of a building are well overlapped, which may not hold in off-nadir aerial images as there is often a big offset between them. In this paper, we propose an offset vector learning scheme, which turns the building footprint extraction problem in off-nadir images into an instance-level joint prediction problem of the building roof and its corresponding "roof to footprint" offset vector. Thus the footprint can be estimated by translating the predicted roof mask according to the predicted offset vector. We further propose a simple but effective feature-level offset augmentation module, which can significantly refine the offset vector prediction by introducing little extra cost. Moreover, a new dataset, Buildings in Off-Nadir Aerial Images (BONAI), is created and released in this paper. It contains 268,958 building instances across 3,300 aerial images with fully annotated instance-level roof, footprint, and corresponding offset vector for each building. Experiments on the BONAI dataset demonstrate that our method achieves the state-of-the-art, outperforming other competitors by 3.37 to 7.39 points in F1-score.
Introduction
Traditional remote sensing algorithms typically operate on "near-nadir" imagery, where the camera looks directly downward. In such scenarios, a building's roof and its ground-level footprint are nearly perfectly aligned. However, modern aerial imaging frequently utilizes "off-nadir" perspectives, capturing scenes at significant angles. This perspective shift reveals building facades and creates a substantial spatial offset between the visible roof and the actual footprint on the ground.
Standard segmentation models often fail in these conditions because they attempt to predict the footprint directly, which is frequently occluded by the building's own structure. This often leads to the model incorrectly identifying the visible facade as part of the footprint.
Methodology: The LOFT Framework
To address the challenges of off-nadir imagery, we introduce LOFT (Learning OFfset vecTor). This framework decouples footprint extraction into two distinct, manageable tasks:
- Roof Segmentation: Identifying the visible roof mask, which is typically unobstructed.
- Offset Prediction: Predicting a 2D vector representing the transformation from the roof's center to the footprint's center.
Architecture
LOFT is built upon the Mask R-CNN architecture, augmented with a specialized Offset Head. While the standard mask head delineates the roof's shape, the offset head analyzes structural cues—such as the visible facade—to determine the direction and magnitude of the displacement to the ground.
During inference, the final footprint is recovered by translating the predicted roof mask by the predicted offset vector.
Feature-level Offset Augmentation (FOA)
To enhance robustness across various camera angles, we propose FOA. Unlike traditional image-level rotation, FOA performs rotations on abstract feature maps during training. This allows the model to learn from diverse perspective angles with minimal computational overhead.
The BONAI Dataset
A major contribution of this work is the release of BONAI (Buildings in Off-Nadir Aerial Images), the first large-scale benchmark for this task. It consists of 3,300 high-resolution aerial images (0.3m–0.6m resolution) covering 268,958 building instances across six major Chinese cities. Each building includes comprehensive instance-level annotations for roofs, footprints, and offset vectors.
Experimental Results
LOFT was evaluated against state-of-the-art methods on the BONAI dataset. It achieved an F1-score of 64.31%, surpassing the previous best (CM R-CNN) by 3.37 points. Notably, LOFT showed superior performance in boundary precision, with a Boundary AP of 53.40, compared to 45.80 for standard Mask R-CNN.
Furthermore, LOFT demonstrates excellent generalization; when tested on the near-nadir WHU Building Dataset, it achieved a 91.97% F1-score, matching models specifically designed for top-down imagery. This suggests that LOFT is a versatile solution for both vertical and oblique aerial photography.
Conclusion
By treating the building facade as a spatial clue rather than an obstacle, the LOFT framework effectively solves the footprint extraction problem in off-nadir imagery. The decoupling of roof segmentation and offset prediction prevents "hallucination" errors common in direct segmentation approaches. Future work will explore integrating explicit facade segmentation to further refine 3D structural understanding and offset accuracy.
Summary updated