Vehicle Instance Segmentation from Aerial Image and Video Using a Multi-Task Learning Residual Fully Convolutional Network

Lichao Mou, Xiao Xiang ZhuarXiv 2018 · DOI · arXiv

TL;DR

This paper introduces a boundary-aware multi-task learning framework designed to solve the tricky problem of separating "touching" vehicles in high-resolution aerial imagery. By combining a Residual Fully Convolutional Network (ResFCN) with a dual-branch architecture that simultaneously predicts vehicle regions and their semantic boundaries, the authors achieve state-of-the-art instance segmentation performance on both standard benchmarks and a challenging new UAV dataset.

What Problem Does This Paper Address?

In remote sensing, extracting vehicles from aerial images usually falls into two buckets: vehicle detection (drawing bounding boxes) or semantic segmentation (labeling every "vehicle" pixel). However, both fall short for advanced applications like traffic monitoring. Detection lacks precise pixel-level shapes, and semantic segmentation often fails to distinguish between individual cars parked bumper-to-bumper, merging them into a single "blob."

This paper tackles vehicle instance segmentation, which requires identifying exactly which pixels belong to which specific vehicle. The primary challenge is the "touching vehicle" problem, where the lack of clear gaps makes it nearly impossible for standard deep learning models to tell where one car ends and the next begins.

The challenge: standard semantic segmentation (bottom left) often merges touching vehicles, whereas instance segmentation (bottom right) separates them.

Methods

The authors propose a Boundary-aware Residual Fully Convolutional Network (B-ResFCN). The architecture is built on two key pillars:

  1. Residual Learning (ResNet) as a Backbone: Unlike traditional feedforward networks (like VGG), the residual blocks in ResNet create "shortcut" paths. The authors theoretically demonstrate that these paths allow low-level visual information—like edges and boundaries—to flow directly to deeper layers, which is critical for segmenting small objects like cars.
  2. Multi-Task Learning (MTL) Architecture: The network splits into two identical branches after the feature extraction stage. One branch predicts the vehicle mask (the body of the car), while the other predicts the semantic boundary (the outline). By training these tasks together, the model learns to pay extra attention to the edges that separate individual instances.
The ResFCN architecture incorporates multi-level contextual features from different stages to capture both fine-grained details and global context.

To produce high-resolution outputs, the model fuses features from multiple stages of the network (8x, 16x, and 32x downsampled layers) and uses upsampling to return to the original image size.

The overall multi-task architecture showing the two branches for segmentation masks and boundary maps.

Datasets

The study utilizes two primary datasets:

  • ISPRS Potsdam: A standard, high-quality benchmark for aerial semantic labeling with 5 cm spatial resolution.
  • Busy Parking Lot UAV Video Dataset: A newly introduced dataset by the authors. It features 15 cm resolution video from a UAV over a mall in Massachusetts. This dataset is significantly more challenging due to high vehicle density, varying colors, complex shadows, and motion blur.
A sample frame from the new Busy Parking Lot UAV Video dataset, highlighting the high density of vehicles.

Key Insights and Ideas

  • Boundaries are the Key: The "aha moment" is that vehicle segmentation isn't just about finding vehicle pixels; it's about finding the non-vehicle pixels that separate them. Explicitly modeling boundaries provides the necessary "cut" to split merged blobs.
  • Residuals Preserve Detail: The authors provide a mathematical look at backpropagation in ResNets, showing that the additive nature of residual blocks prevents the "washing out" of fine edge details that typically happens in very deep networks.
  • Context Matters for Small Objects: By fusing features from early (fine-grained) and late (semantic) stages of the network, the model can identify a pixel as part of a "car" while simultaneously knowing exactly where that car's edge is.

Key Results

The proposed B-ResFCN consistently outperformed standard FCN variants (VGG, Inception, and Xception backbones).

  • Instance-Level F1 Score: On the Potsdam dataset, the B-ResFCN achieved a staggering 98.31% instance-level F1 score.
  • Generalization: On the more difficult Busy Parking Lot dataset, the model maintained a high F1 score of 93.29%, significantly outperforming the ResFCN without the boundary branch (which scored 88.73%).
  • Qualitative Success: Visual results show the model successfully separating long rows of parked cars that other state-of-the-art models merged into single continuous strips.

Limitations and Future Work

While the results are impressive, the authors note that performance drops in areas with extreme shadows or significant motion blur, which are common in real-world UAV footage. The current approach treats each video frame independently; the authors suggest that incorporating temporal information (tracking vehicles across frames) could further improve robustness. Additionally, while the multi-task approach is effective, it relies on high-quality boundary annotations, which can be labor-intensive to produce for new datasets.

Summary updated

Abstract

Object detection and semantic segmentation are two main themes in object retrieval from high-resolution remote sensing images, which have recently achieved remarkable performance by surfing the wave of deep learning and, more notably, convolutional neural networks (CNNs). In this paper, we are interested in a novel, more challenging problem of vehicle instance segmentation, which entails identifying, at a pixel-level, where the vehicles appear as well as associating each pixel with a physical instance of a vehicle. In contrast, vehicle detection and semantic segmentation each only concern one of the two. We propose to tackle this problem with a semantic boundary-aware multi-task learning network. More specifically, we utilize the philosophy of residual learning (ResNet) to construct a fully convolutional network that is capable of harnessing multi-level contextual feature representations learned from different residual blocks. We theoretically analyze and discuss why residual networks can produce better probability maps for pixel-wise segmentation tasks. Then, based on this network architecture, we propose a unified multi-task learning network that can simultaneously learn two complementary tasks, namely, segmenting vehicle regions and detecting semantic boundaries. The latter subproblem is helpful for differentiating closely spaced vehicles, which are usually not correctly separated into instances. Currently, datasets with pixel-wise annotation for vehicle extraction are ISPRS dataset and IEEE GRSS DFC2015 dataset over Zeebrugge, which specializes in semantic segmentation. Therefore, we built a new, more challenging dataset for vehicle instance segmentation, called the Busy Parking Lot UAV Video dataset, and we make our dataset available at http://www.sipeo.bgu.tum.de/download so that it can be used to benchmark future vehicle instance segmentation algorithms.
BibTeX
@misc{mou2018vehicle,
  author = {Lichao Mou and Xiao Xiang Zhu},
  title = {{Vehicle Instance Segmentation from Aerial Image and Video Using a Multi-Task Learning Residual Fully Convolutional Network}},
  year = {2018},
  eprint = {1805.10485},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1805.10485},
}
Figure