Incremental and Multi-Task Learning Strategies for Coarse-To-Fine Semantic Segmentation

Mazen Mel, Umberto Michieli, Pietro ZanuttighTechnologies 2019 · DOI

TL;DR

This paper tackles the challenge of semantic segmentation by proposing novel incremental and multi-task learning strategies. Instead of retraining models from scratch for increasingly detailed scene understanding, they leverage previously learned coarse-grained information to efficiently learn finer-grained categories, achieving competitive results with improved efficiency.

What Problem Does This Paper Address?

Semantic segmentation, the task of assigning a label to every pixel in an image, is crucial for understanding visual scenes. However, real-world applications often require varying levels of detail. For instance, a robot might first need to identify broad categories like "movable objects" and "structures" for navigation, but later needs to distinguish between "books," "monitors," and "cups" for interaction. Retraining a deep neural network from scratch for each new, more detailed set of classes is computationally expensive and inefficient. This paper addresses this by developing methods to incrementally adapt existing segmentation models to new, finer classes, and also explores a multi-task approach to learn multiple levels of semantic understanding simultaneously.

Methods

The authors propose three main incremental learning strategies, all built upon the DeepLab V3+ architecture.

  1. Incremental with Softmax: The output (softmax probabilities) from a model trained on a coarse set of classes is concatenated with the input RGB images for training a new model on a finer set of classes. This provides richer contextual information than just the input image.
  2. Incremental with Argmax: Similar to the softmax approach, but instead of the full probability distribution, only the predicted class index (argmax) from the coarse model is used as an additional input. This is more compact but loses uncertainty information.
  3. Incremental with Edges: The edges of the predicted semantic map from the coarse model are used as an additional input. This focuses the learning on boundary information, which is often challenging to segment accurately.

Additionally, they propose a Joint Learning of Multiple Representations approach. Here, a single DeepLab V3+ encoder is used with a modified decoder capable of outputting multiple semantic maps at different levels of granularity (e.g., 5, 15, and 41 classes) simultaneously. This is trained end-to-end with a combined loss function.

Datasets

The primary dataset used for training and evaluation is the New York University Depth v2 (NYUDv2) dataset. This dataset contains RGB and depth images of indoor scenes. The authors utilize three different label sets from this dataset, representing hierarchical levels of semantic precision: 5 classes, 15 classes, and 41 classes.

Key Insights and Ideas

  • Leveraging Coarse-to-Fine Information: The core idea is that knowledge gained from segmenting broad categories can significantly aid in learning to segment finer, more specific categories. This is more efficient than starting from scratch.
  • Softmax vs. Argmax as Input: Using the full softmax output from a previous stage provides more information and leads to slightly better results than just using the predicted class index (argmax), despite being more computationally complex.
  • Edges as a Useful Cue: Incorporating edge information from previous segmentation stages helps the model focus on object boundaries, leading to improved accuracy, particularly for challenging shapes.
  • Multi-Task Learning for Efficiency and Accuracy: Jointly learning multiple levels of semantic segmentation in a single pass offers a significant speedup (no multiple inference steps) and can even improve accuracy compared to training individual models sequentially.
  • Hierarchical Splitting Matters: The way classes are split hierarchically significantly impacts the difficulty. Some splits are highly unbalanced, with few pixels belonging to certain finer classes, posing a challenge for learning.

Key Results

  • Incremental Learning: The incremental approaches (especially with softmax and edges) showed improvements in Pixel Accuracy (PA), mean Class Accuracy (mCA), and mean Intersection over Union (mIoU) when transitioning from 4 to 13 classes, compared to the baseline DeepLab V3+ trained directly on 13 classes. For example, the incremental (edges) approach achieved 69.8% mCA and 53.9% mIoU for 13 classes, outperforming the baseline's 68.2% mCA and 51.4% mIoU.
  • Multi-Task Learning: The joint multi-task learning approach achieved competitive results across all label sets (4, 13, and 40 classes). For the 4-class setting, it achieved 83.2% PA, 82.3% mCA, and 72.0% mIoU, slightly outperforming the baseline's 82.5% PA, 82.2% mCA, and 70.3% mIoU. While the improvements were sometimes modest, they were achieved in a single training pass.
  • Qualitative Results: Visualizations showed that the incremental methods (especially softmax and edges) produced cleaner segmentations with fewer artifacts compared to the baseline and argmax methods. The multi-task approach also showed improvements in recognizing finer details and object boundaries.

Limitations and Future Work

  • Computational Constraints: The authors note that combining multiple cues can be limited by GPU memory constraints.
  • Incremental Inference Time: While the incremental strategies improve training efficiency, they can require multiple inference passes, which might not be suitable for real-time applications. The multi-task approach is better suited for real-time scenarios.
  • Future Work: The authors plan to explore improving the multi-task approach by adding other tasks (e.g., depth prediction, surface normal estimation) without using them as inputs. They also aim to investigate using more lightweight network architectures to better combine multiple cues.

Summary updated

Abstract

The semantic understanding of a scene is a key problem in the computer vision field. In this work, we address the multi-level semantic segmentation task where a deep neural network is first trained to recognize an initial, coarse, set of a few classes. Then, in an incremental-like approach, it is adapted to segment and label new objects’ categories hierarchically derived from subdividing the classes of the initial set. We propose a set of strategies where the output of coarse classifiers is fed to the architectures performing the finer classification. Furthermore, we investigate the possibility to predict the different levels of semantic understanding together, which also helps achieve higher accuracy. Experimental results on the New York University Depth v2 (NYUDv2) dataset show promising insights on the multi-level scene understanding.
BibTeX
@inproceedings{mel2019incremental,
  author = {Mazen Mel and Umberto Michieli and Pietro Zanuttigh},
  title = {{Incremental and Multi-Task Learning Strategies for Coarse-To-Fine Semantic Segmentation}},
  booktitle = {Technologies},
  year = {2019},
  doi = {10.3390/technologies8010001},
}
Figure