Abstract
This paper introduces a novel method to automatically learn dilation factors in convolutional neural networks (CNNs) rather than relying on manual heuristics. By allowing dilation to be fractional and unique to each feature channel, the authors enable networks to adaptively optimize their receptive fields. This approach leads to state-of-the-art semantic segmentation results on challenging street-scene datasets like Cityscapes and CamVid with minimal computational overhead.
Introduction
In semantic segmentation—the task of assigning a category (such as "car," "tree," or "road") to every pixel—context is a critical factor. Identifying a large truck requires a wide receptive field, whereas identifying a thin pole requires fine-grained local detail.
Dilated convolutions (or atrous convolutions) are the standard mechanism for expanding a network's view without sacrificing image resolution. However, traditional dilation factors are fixed integers (e.g., 1, 2, 4) that researchers must tune manually. This paper addresses two primary limitations of this approach: the tedious nature of manual tuning and the rigidity of applying the same dilation factor to every channel in a layer, regardless of the feature type.
Methods
The authors propose a Learnable Channel-based Dilated Convolution layer. This layer introduces several key innovations to make the receptive field a dynamic, learnable part of the architecture:
- Continuous Dilation Factors: Instead of being restricted to integers, the dilation factor ($d$) is treated as a real-valued, learnable parameter. This allows the network to use backpropagation to "nudge" the dilation factor toward an optimal value during training.
- Bilinear Interpolation: To handle fractional dilation factors where kernel weights might land between pixels, the authors employ bilinear interpolation to sample values from the input feature map.
- Channel-wise Optimization: The method allows each individual channel in a convolutional layer to have its own dilation factor. This enables a single layer to simultaneously capture fine details in some channels and broad context in others.
- End-to-End Training: The layer is fully differentiable, allowing the network to learn the best "viewing distance" for its filters alongside the filter weights themselves.
Experiments and Results
The researchers evaluated their approach on two major benchmarks for autonomous driving: Cityscapes (high-resolution street scenes with 19 classes) and CamVid (street-view video with 11 classes).
Quantitative Performance
The method consistently improved performance over strong baselines:
- Cityscapes: On the Deeplab-LargeFOV baseline, learning the dilation factors improved the Mean Intersection over Union (mIoU) from 58.91% to 63.31%. When integrated into the more powerful PSPNet, it pushed the mIoU to 79.9%.
- CamVid: The approach achieved a new state-of-the-art result at the time of publication, reaching 73.9% mIoU.
Qualitative and Efficiency Gains
Visually, the models showed a marked improvement in distinguishing between easily confused classes of different scales, such as "Person" vs. "Rider" and "Truck" vs. "Bus." Despite the added complexity of bilinear interpolation, the computational overhead remained low: inference time increased by only 3% to 8%, and training time increased by approximately 10% to 15% compared to base models.
Discussion and Conclusion
The authors found that the learned dilation distributions were remarkably stable, converging to similar effective configurations regardless of initialization. By capturing both local and wide context simultaneously within a single layer, the proposed method demonstrates that a "one size fits all" dilation factor is suboptimal.
While highly effective, the method currently requires manual range constraints (e.g., [1, 4]) to prevent optimization from diverging. Future work may explore removing these boundaries and testing the generalization of learnable dilation on more unconstrained datasets like COCO or ImageNet. Nonetheless, the proposed layer offers a "plug-and-play" improvement for existing architectures like Deeplab and PSPNet without requiring extra supervision.
Summary updated