Empirical Study of Multi-Task Hourglass Model for Semantic Segmentation Task

Darwin Saire, Adín Ramírez Rivera

TL;DR

This paper tackles the persistent problem of lost spatial precision in semantic segmentation, particularly at object boundaries. They propose a multi-task learning approach using hourglass models, where auxiliary tasks like edge detection and contour prediction help refine the model's internal representations, leading to improved segmentation accuracy without post-processing.

What Problem Does This Paper Address?

Semantic segmentation (SS) is crucial for understanding images by assigning a class label to every pixel. While Convolutional Neural Networks (CNNs) have revolutionized SS, a significant challenge remains: the loss of spatial precision, especially around object edges and for small objects. This results in blurry or inaccurate boundaries in the segmented output. The authors hypothesize that this issue isn't due to a single faulty operation but a combination of factors, including insufficient feature refinement. They aim to address this by enhancing the model's internal "latent space" through multi-task learning.

Methods

The core of this work is a multi-task learning (MTL) framework applied to hourglass (encoder-decoder) CNN architectures. The primary task is semantic segmentation. To improve its performance, they introduce several auxiliary tasks:

  • Edge Detection: Aims to identify object boundaries.
  • Semantic Contours: Similar to edge detection but focuses on pixel-level dense classification of object contours.
  • Distance Transform: Provides geometric information about the distance of pixels to object edges. This is further processed into a quantized distance transform for easier integration.

The key idea is that by sharing a common latent space, these contour-focused auxiliary tasks can guide the model to learn more robust representations that specifically enhance the understanding of object boundaries, thereby mitigating the spatial precision loss. They empirically investigate how these auxiliary tasks influence the latent space and, consequently, the final segmentation results.

Datasets

The authors evaluated their methodology on three standard datasets:

  • Cityscapes: A large-scale dataset for urban scene understanding, featuring 19 semantic classes.
  • CamVid: A dataset focused on road scene understanding with 11 classes.
  • Freiburg Forest: A dataset for forest environments with six classes.

They employed data augmentation techniques, including random cropping and transformations, to increase the training data size.

Key Insights and Ideas

  • Latent Space Clustering is Key: The paper strongly suggests that the improved segmentation performance is directly linked to a more organized, "clustered" latent space. When auxiliary tasks are added, features that are similar across tasks tend to group together in this latent space.
  • Hourglass Architectures are Crucial: The authors specifically focus on hourglass models because their encoder-decoder structure with a decoding stage is particularly well-suited for reconstructing spatial information and benefiting from the latent space manipulation.
  • Contour Tasks are Highly Beneficial: Auxiliary tasks that focus on object edges and contours (edge detection, semantic contours, distance transform) are shown to be particularly effective in improving spatial precision.
  • No Post-Processing Needed: A significant finding is that their MTL approach improves segmentation quality to the point where traditional post-processing steps (like Conditional Random Fields) are not required to achieve state-of-the-art results.

Key Results

  • Improved Segmentation Accuracy: The multi-task approach consistently improved the Intersection over Union (IoU) metric across all tested datasets (CamVid, Cityscapes, Freiburg Forest) compared to single-task hourglass models. For instance, on CamVid, they achieved an IoU of 0.6259 with MTL compared to 0.5895 without it.
  • Better Boundary Definition: Qualitative results (visualizations) and quantitative analysis (using trimap evaluation) demonstrate a significant improvement in the accuracy of object boundaries. Activation maps become better aligned with object contours.
  • Reduced Overfitting: The auxiliary tasks act as a regularizer, helping the models generalize better and reducing the risk of overfitting, especially on object edges.
  • State-of-the-Art Performance: The paper claims to improve the state-of-the-art for hourglass models on these datasets without any post-processing.

Limitations and Future Work

  • Focus on Hourglass Models: The current study is limited to hourglass architectures. The authors acknowledge that other architectures might benefit differently from MTL.
  • Computational Cost: While not a primary focus, the paper notes that training with multiple tasks increases training time, though testing time remains efficient as only the segmentation task is performed.
  • Future Directions:
    • Investigate using clustering frameworks to explicitly impose biases on the latent space, potentially reducing the number of required auxiliary tasks.
  • Explore the influence of these contour-based auxiliary tasks on architectures other than hourglass models.
  • Extend the work to video segmentation, which requires handling temporal consistency in addition to spatial accuracy.
  • Explore optical flow as a potential auxiliary task for video segmentation.

Summary updated

Abstract

The semantic segmentation (SS) task aims to create a dense classification by labeling at the pixel level each object present on images. Convolutional neural network (CNN) approaches have been widely used, and exhibited the best results in this task. However, the loss of spatial precision on the results is a main drawback that has not been solved. In this work, we propose to use a multi-task approach by complementing the semantic segmentation task with edge detection, semantic contour, and distance transform tasks. We propose that by sharing a common latent space, the complementary tasks can produce more robust representations that can enhance the semantic labels. We explore the influence of contour-based tasks on latent space, as well as their impact on the final results of SS. We demonstrate the effectiveness of learning in a multi-task setting for hourglass models in the Cityscapes, CamVid, and Freiburg Forest datasets by improving the state-of-the-art without any refinement post-processing.
BibTeX
@inproceedings{saire2021empirical,
  author = {Darwin Saire and Adín Ramírez Rivera},
  title = {{Empirical Study of Multi-Task Hourglass Model for Semantic Segmentation Task}},
  booktitle = {IEEE Access},
  year = {2021},
  url = {https://openreview.net/forum?id=6KHYI-Q0QgS},
}
Figure