Abstract: We present semantic segmentation experiments with a model capable to perform predictions on four benchmark datasets: Cityscapes, ScanNet, WildDash and KITTI. We employ a ladder-style convolutional architecture featuring a modified DenseNet-169 model in the downsampling datapath, and only one convolution in each stage of the upsampling datapath. Due to limited computing resources, we perform the training only on Cityscapes Fine train+val, ScanNet train, WildDash val and KITTI train. We evaluate the trained model on the test subsets of the four benchmarks in concordance with the guidelines of the Robust Vision Challenge ROB 2018. The performed experiments reveal several interesting findings which we describe and discuss.
Introduction
Most semantic segmentation models are specialists, excelling on specific datasets like Cityscapes but struggling when faced with the "photographer bias" or the messy reality of the real world. To be truly useful for autonomous systems, a model must be robust. It needs to handle diverse environments (indoor vs. outdoor), recognize when it is looking at something it hasn't been trained on (out-of-distribution detection), and maintain accuracy across different sensor qualities and weather conditions. This research addresses these challenges within the context of the Robust Vision Challenge (ROB) 2018, which requires a single model to perform well across multiple distinct benchmarks simultaneously.
Methodology
The authors employ a Ladder-DenseNet architecture, a fully convolutional neural network designed for high-resolution prediction.
- The Backbone: A modified DenseNet-169 serves as the downsampling (encoder) path to extract rich semantic features.
- The Ladder Path: To recover spatial detail lost during downsampling, the model uses a ladder-style upsampling path. Standard feature concatenations were replaced with summations to improve computational efficiency without sacrificing accuracy.
- Pyramid Loss: To combat overfitting in the upsampling layers, a novel auxiliary loss was introduced. Classification heads were attached at multiple scales (8x, 16x, 32x, and 64x subsampled resolutions), calculating cross-entropy loss against the ground truth at those specific scales.
- Training Strategy: Because the indoor dataset (ScanNet) is significantly larger than the driving datasets, the team used oversampling to maintain a 2:1 ratio of indoor to outdoor images. Crucially, they mixed indoor and outdoor images within the same batches to ensure Batch Normalization statistics remained stable across domains.
Datasets and Label Space
The model was trained and evaluated on four primary benchmarks with a combined label space of 39 object classes (19 driving + 20 indoor):
- Cityscapes: High-resolution urban driving scenes from Germany.
- ScanNet: A massive indoor dataset of 2.5 million views.
- WildDash: A "stress test" dataset featuring hazardous driving scenarios like extreme blur and weather.
- KITTI: A standard autonomous driving dataset.
Key Insights
Low Domain Interference
A significant finding was that training on indoor and outdoor data simultaneously resulted in almost zero performance hit. The model rarely "hallucinated" indoor objects in driving scenes (less than 0.01% of pixels), suggesting that deep networks have sufficient capacity to keep these domains distinct.
Foreign Classes as Uncertainty Signals
In the WildDash "stress test," the model often predicted indoor classes (from ScanNet) when it encountered visual anomalies or out-of-distribution patches. By remapping these "indoor" predictions to a "Void" or "Ignore" class, the authors boosted their WildDash score by 9.1 percentage points. This suggests that multi-domain training provides a built-in mechanism to detect when a model is confused.
Upsampling and Batch Composition
While deep layers proved robust, fine-grained upsampling layers were prone to overfitting to specific camera characteristics. The "Pyramid Loss" was essential to regularizing these layers. Furthermore, mixing domains in every training batch was vital; without it, Batch Normalization layers would drift, hurting performance when switching between indoor and outdoor contexts.
Results
The model, identified as LDN2_ROB, achieved competitive results across the ROB 2018 challenge:
| Dataset | Metric | Result | Challenge Rank |
|---|---|---|---|
| Cityscapes | class IoU | 77.1 | 2nd |
| ScanNet | class IoU | 44.0 | 2nd |
| KITTI | class IoU | 63.5 | 3rd |
| WildDash | category iIoU | 54.5 | 3rd |
Conclusion
Despite the success of the Pyramid Loss, some overfitting still occurs in the highest-resolution stages of the upsampling path, where accuracy sometimes decreased slightly as the model moved from coarse to fine resolutions. Future work may involve incorporating even more diverse datasets and investigating more sophisticated ways to use "foreign class" activations as a formal measure of predictive uncertainty.
Summary updated