FastFCN: Rethinking Dilated Convolution in the Backbone for Semantic Segmentation

Huikai Wu, Junge Zhang, Kaiqi Huang, Kongming Liang, Yizhou YuarXiv 2019 · arXiv

Semantic segmentation—the task of assigning a label to every pixel in an image—is a cornerstone of computer vision but remains computationally intensive. For years, "Dilated Convolution" has been the standard for maintaining high-resolution feature maps while preserving a wide field of view. However, this comes at a significant cost in terms of speed and memory.

In "FastFCN: Rethinking Dilated Convolution in the Backbone for Semantic Segmentation," researchers from the Chinese Academy of Sciences and Deepwise AI Lab introduce Joint Pyramid Upsampling (JPU). This novel module allows networks to use standard, efficient classification backbones while achieving state-of-the-art accuracy at triple the speed.

The Resolution Dilemma

Standard Fully Convolutional Networks (FCNs) typically downsample images by a factor of 32 to extract high-level features. While effective for classification, this 32x reduction loses the fine spatial details necessary for precise segmentation boundaries.

To address this, researchers previously turned to Dilated (Atrous) Convolutions. By removing the final downsampling layers and using dilated kernels, they could maintain an 8x resolution. While accurate, this approach creates a massive computational bottleneck. For instance, a ResNet-101 backbone using dilated convolutions requires roughly 4x more computation and memory than its classification counterpart.

Comparison of different segmentation architectures: (a) FCN, (b) EncoderDecoder, and (c) DilatedFCN.

Joint Pyramid Upsampling (JPU)

The core contribution of FastFCN is the Joint Pyramid Upsampling (JPU) module. Rather than modifying the backbone to maintain high resolution throughout, FastFCN uses a standard "off-the-shelf" classification backbone that downsamples to 32x.

The JPU module treats the recovery of detail as a joint upsampling problem. It extracts the last three feature maps from the backbone (at 8x, 16x, and 32x resolution) and processes them through three main steps:

  1. Embedding: Each map is passed through a regular convolution to reduce dimensions and map them into a shared embedding space.
  2. Fusion: These maps are upsampled and concatenated into a unified feature representation.
  3. Multi-Scale Context: The module applies four parallel separable convolutions with varying dilation rates (1, 2, 4, and 8). This pyramid structure learns to map low-resolution features back to high-resolution details, guided by the earlier, higher-resolution layers.
Overview of the FastFCN framework featuring the JPU module.

By moving the dilation logic out of the backbone's 100+ layers and into this single, lightweight module, computational costs are drastically reduced.

Key Insights

  • Mathematical Foundation: The authors provide a mathematical proof that the high-resolution feature maps of a DilatedFCN can be approximated as a joint upsampling problem, justifying the JPU approach.
  • Backbone Agnosticism: JPU acts as a "plug-in" head. Any standard classification model (like ResNet) can be converted into a high-performance segmenter without specialized pre-training.
  • Efficiency: The use of depthwise separable convolutions within the JPU further minimizes parameter count and FLOPs.

Results and Performance

FastFCN demonstrates significant improvements in both speed and accuracy across major benchmarks:

  • Speed: FastFCN runs 3x faster than equivalent DilatedFCNs. On a Titan-Xp GPU, a ResNet-101 based FastFCN achieved ~23 FPS, compared to ~10 FPS for the dilated version.
  • Accuracy:
    • On Pascal Context, it achieved an mIoU of 53.13%, a record at the time.
  • On ADE20K, it reached a final score of 0.5584, outperforming previous state-of-the-art models.
  • Versatility: JPU was successfully integrated into other architectures like PSPNet and DeepLabV3, consistently boosting both speed and performance.

Conclusion

FastFCN effectively resolves the bottleneck of dilated convolutions in segmentation backbones. By introducing the JPU module, the researchers have enabled high-resolution scene parsing that is both accurate and efficient enough for more demanding real-time applications. Future work may focus on optimizing the remaining "context heads" or adapting JPU for mobile architectures.

Summary updated

Abstract

Modern approaches for semantic segmentation usually employ dilated convolutions in the backbone to extract high-resolution feature maps, which brings heavy computation complexity and memory footprint. To replace the time and memory consuming dilated convolutions, we propose a novel joint upsampling module named Joint Pyramid Upsampling (JPU) by formulating the task of extracting high-resolution feature maps into a joint upsampling problem. With the proposed JPU, our method reduces the computation complexity by more than three times without performance loss. Experiments show that JPU is superior to other upsampling modules, which can be plugged into many existing approaches to reduce computation complexity and improve performance. By replacing dilated convolutions with the proposed JPU module, our method achieves the state-of-the-art performance in Pascal Context dataset (mIoU of 53.13%) and ADE20K dataset (final score of 0.5584) while running 3 times faster.
BibTeX
@misc{wu2019fastfcn,
  author = {Huikai Wu and Junge Zhang and Kaiqi Huang and Kongming Liang and Yizhou Yu},
  title = {{FastFCN: Rethinking Dilated Convolution in the Backbone for Semantic Segmentation}},
  year = {2019},
  eprint = {1903.11816},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1903.11816},
}
Figure