Semantic segmentation—the task of assigning a label to every pixel in an image—is a cornerstone of computer vision but remains computationally intensive. For years, "Dilated Convolution" has been the standard for maintaining high-resolution feature maps while preserving a wide field of view. However, this comes at a significant cost in terms of speed and memory.
In "FastFCN: Rethinking Dilated Convolution in the Backbone for Semantic Segmentation," researchers from the Chinese Academy of Sciences and Deepwise AI Lab introduce Joint Pyramid Upsampling (JPU). This novel module allows networks to use standard, efficient classification backbones while achieving state-of-the-art accuracy at triple the speed.
The Resolution Dilemma
Standard Fully Convolutional Networks (FCNs) typically downsample images by a factor of 32 to extract high-level features. While effective for classification, this 32x reduction loses the fine spatial details necessary for precise segmentation boundaries.
To address this, researchers previously turned to Dilated (Atrous) Convolutions. By removing the final downsampling layers and using dilated kernels, they could maintain an 8x resolution. While accurate, this approach creates a massive computational bottleneck. For instance, a ResNet-101 backbone using dilated convolutions requires roughly 4x more computation and memory than its classification counterpart.
Joint Pyramid Upsampling (JPU)
The core contribution of FastFCN is the Joint Pyramid Upsampling (JPU) module. Rather than modifying the backbone to maintain high resolution throughout, FastFCN uses a standard "off-the-shelf" classification backbone that downsamples to 32x.
The JPU module treats the recovery of detail as a joint upsampling problem. It extracts the last three feature maps from the backbone (at 8x, 16x, and 32x resolution) and processes them through three main steps:
- Embedding: Each map is passed through a regular convolution to reduce dimensions and map them into a shared embedding space.
- Fusion: These maps are upsampled and concatenated into a unified feature representation.
- Multi-Scale Context: The module applies four parallel separable convolutions with varying dilation rates (1, 2, 4, and 8). This pyramid structure learns to map low-resolution features back to high-resolution details, guided by the earlier, higher-resolution layers.
By moving the dilation logic out of the backbone's 100+ layers and into this single, lightweight module, computational costs are drastically reduced.
Key Insights
- Mathematical Foundation: The authors provide a mathematical proof that the high-resolution feature maps of a DilatedFCN can be approximated as a joint upsampling problem, justifying the JPU approach.
- Backbone Agnosticism: JPU acts as a "plug-in" head. Any standard classification model (like ResNet) can be converted into a high-performance segmenter without specialized pre-training.
- Efficiency: The use of depthwise separable convolutions within the JPU further minimizes parameter count and FLOPs.
Results and Performance
FastFCN demonstrates significant improvements in both speed and accuracy across major benchmarks:
- Speed: FastFCN runs 3x faster than equivalent DilatedFCNs. On a Titan-Xp GPU, a ResNet-101 based FastFCN achieved ~23 FPS, compared to ~10 FPS for the dilated version.
- Accuracy:
- On Pascal Context, it achieved an mIoU of 53.13%, a record at the time.
- On ADE20K, it reached a final score of 0.5584, outperforming previous state-of-the-art models.
- Versatility: JPU was successfully integrated into other architectures like PSPNet and DeepLabV3, consistently boosting both speed and performance.
Conclusion
FastFCN effectively resolves the bottleneck of dilated convolutions in segmentation backbones. By introducing the JPU module, the researchers have enabled high-resolution scene parsing that is both accurate and efficient enough for more demanding real-time applications. Future work may focus on optimizing the remaining "context heads" or adapting JPU for mobile architectures.
Summary updated