Abstract
SkyScapes introduces a high-resolution aerial dataset and a novel multi-task deep learning model designed to bridge the gap between wide-area aerial surveys and the centimeter-level precision required for autonomous driving HD maps. By treating semantic edge detection as an auxiliary task, the proposed "SkyScapesNet" significantly improves the segmentation of fine-grained details like lane markings and complex urban boundaries.
Introduction
High-definition (HD) maps are essential for autonomous driving and advanced driver assistance systems (ADAS). Traditionally, these maps are generated using mobile mapping systems—vans equipped with LiDAR and cameras. While precise, this process is slow, expensive, and limited by the vehicle's field of view and street-level occlusions.
Aerial imagery offers a potential solution by capturing vast urban areas instantaneously. However, existing aerial datasets are often too coarse, focusing on broad categories like "buildings" or "roads" without the granularity needed for navigation. To be useful for HD mapping, a system must distinguish between different types of lane markings, sidewalks, and specific vehicle zones at a centimeter-level resolution. SkyScapes addresses two main challenges: the lack of highly detailed aerial data and the inability of standard segmentation models to handle the extreme scale variance between massive buildings and tiny lane lines.
The SkyScapes Dataset
The primary contribution of this work is the SkyScapes dataset, which provides a new benchmark for fine-grained aerial scene understanding. It consists of 16 non-overlapping RGB images captured via helicopter over Munich, Germany, with a Ground Sampling Distance (GSD) of approximately 13 cm/pixel.
The dataset features 31 semantic categories, including 19 dense classes and 12 specific lane-marking sub-categories such as zebra zones, stop lines, and dashed lines. Each image underwent a rigorous three-level manual quality check, totaling roughly 200 man-hours of annotation per image to ensure the precision required for HD mapping applications.
SkyScapesNet Architecture
The authors propose SkyScapesNet, a multi-task ensemble-model network based on the Fully Convolutional DenseNet (FC-DenseNet). Its architecture is specifically tuned for the "small object, large context" problem inherent in aerial imagery.
The network incorporates several key innovations:
- Multi-Task Learning: The model simultaneously predicts dense semantics, multi-class edges, and binary edges. This auxiliary edge detection forces the model to pay closer attention to boundaries, reducing the "bleeding" effect common in standard segmentation.
- CRASPP (Concatenated Reverse ASPP): This module uses varying dilation rates in both forward and reverse orders to capture features across a wide range of scales.
- FRSR (Full-Resolution Separable Residual): A residual pooling stream helps the model retain high-resolution spatial information often lost during downsampling.
- LKBR (Large-Kernel with Boundary Refinement): Large convolutional kernels are used to refine the extraction of tiny objects and sharpen region outlines.
Key Insights and Results
A significant finding is that adding edge detection as a side-task improves primary segmentation. By penalizing the model for fuzzy boundaries, it becomes much better at separating closely packed objects, such as individual lanes on a highway. Furthermore, SkyScapesNet addresses the "Scale Paradox"—the need to understand 50-meter buildings and 10-centimeter lane lines simultaneously—through multi-scale branching.
Performance Metrics
- SkyScapes-Dense: SkyScapesNet achieved a 40.13% mIoU, a 1.93% improvement over the DeepLabv3+ baseline.
- SkyScapes-Lane: On the challenging lane-marking task, the model reached 51.93% mIoU, outperforming the state-of-the-art by 3.06%.
- Generalization: When trained on SkyScapes and tested on the ISPRS Potsdam dataset, the model achieved 47.46% mIoU without any fine-tuning, demonstrating the robustness of the learned features across different sensors and urban environments.
Conclusion
While SkyScapes provides a high-quality benchmark, the dataset's size (16 original images) and extreme class imbalance remain challenges. Future work will focus on expanding geographic diversity and exploring the fusion of aerial-derived maps with ground-level sensor data to create a complete HD mapping pipeline.
Summary updated