Dilated Continuous Random Field for Semantic Segmentation

Xi Mo, Xiangyu Chen, Cuncong Zhong, Rui Li, Kaidong Li, Usman Sajid

TL;DR

This paper introduces DilatedCRF, a novel end-to-end trainable framework for semantic segmentation that relaxes the computational constraints of traditional Conditional Random Fields (CRFs). By employing a Dilated Sparse Convolution (DSConv) module and adaptive pooling, DilatedCRF achieves state-of-the-art performance while significantly reducing computational cost and inference time.

What Problem Does This Paper Address?

Semantic segmentation, particularly in applications like robotic bin-picking, often requires refining initial predictions to achieve fine-grained accuracy. Traditional methods like DenseCRF use mean field approximation to optimize energy terms, which is computationally intensive and can be a bottleneck for real-time applications. The challenge lies in developing a CRF-based refinement method that is both accurate and efficient, capable of integrating seamlessly with deep learning models without requiring specialized hardware or complex, non-differentiable operations. Furthermore, the precision of human-annotated data, like affordance maps for suction, can be inconsistent, necessitating robust post-processing.

Methods

The core innovation is the Dilated Sparse Convolution (DSConv) module. This module replaces computationally expensive fully connected layers in traditional CRF approaches with a more efficient, channel-wise convolution. It uses dilated sparse convolutions to capture global relationships between features and then employs Adaptive Global Average Pooling (AGAP) and Adaptive Global Max Pooling (AGMP) to extract global energy terms. These pooling operations act as replacements for linear layers, drastically reducing the number of trainable parameters.

The paper also proposes an end-to-end DilatedCRF pipeline that is trainable via GPU. This pipeline integrates the DSConv module and uses adaptive pooling to replace fully connected layers. The unary energy term can be derived from various sources, including pre- or post-softmax features from a classifier, or even predicted affordance maps from conventional classifiers, making the approach versatile. The framework is designed to be differentiable, allowing for end-to-end training with gradient back-propagation.

Datasets

The primary dataset used for training and evaluation is the suction dataset. This dataset contains raw images and corresponding affordance maps, where different colors represent background, negative samples (unsuitable for suction), and valid suction areas. The dataset has a training split of 1470 images and a testing split of 367 images, each with a resolution of 480x640.

Key Insights and Ideas

  • Relaxing Mean Field Approximation: The paper demonstrates that moving away from the strict mean field approximation in CRFs, by allowing for global optimization through DSConv, can lead to improved accuracy without sacrificing efficiency.
  • DSConv for Parameter Reduction: The DSConv module, combined with adaptive pooling, is a highly effective way to drastically reduce the number of parameters in CRF models, making them more computationally feasible.
  • End-to-End Trainability: Designing a CRF refinement module that can be trained end-to-end with deep learning models is crucial for seamless integration and optimal performance. DilatedCRF achieves this by being fully differentiable.
  • Versatility of Unary Terms: The ability to derive unary terms from various sources (e.g., different classifier outputs) makes DilatedCRF adaptable to a wide range of existing segmentation pipelines.

Key Results

DilatedCRF achieves state-of-the-art performance on the suction dataset, outperforming other CRF-based approaches. Specifically:

  • Efficiency: DilatedCRF significantly reduces inference time compared to DenseCRF. For example, the octant-size DilatedCRF has an inference time of 8.91ms, while DenseCRF takes 450.16ms.
  • Accuracy: While DenseCRF shows the highest average precision and accuracy, DilatedCRF models (especially the quarter-size) demonstrate a better trade-off between inference time and metrics like Recall and Dice Coefficient. The octant-size DilatedCRF shows competitive performance in Accuracy and Jaccard Index despite its coarseness.
  • Improved Recall and Dice: All DilatedCRF variants outperform the original LR-ASPP approach on Recall and Dice scores, indicating better object localization.
  • Qualitative Improvements: Visualizations show that DilatedCRF better recognizes relationships between separated regions and locates more objects, even if some are initially misclassified as false negatives.

Limitations and Future Work

Limitations:

  • The paper notes that the massive concatenation of output features and memory transfer can consume a significant portion of runtime, which is identified as a future work area for acceleration.
  • While DilatedCRF improves recall, it can sometimes lead to slightly lower recall and Jaccard index compared to the initial LR-ASPP artifacts, suggesting that the refinement might not always perfectly align with the ground truth in all aspects.

Future Work:

  • Applying more powerful backbone networks to learn better unary terms.
  • Developing network adaptive searching strategies to find optimal strides for the DSConv module.
  • Integrating high-order contextual information into global energy terms to enhance regional encoding.
  • Speeding up inference for full-size and half-size models by reducing concatenation operations.

Summary updated

Abstract

Mean field approximation methodology has laid the foundation of modern Continuous Random Field (CRF) based solutions for the refinement of semantic segmentation. In this paper, we propose to relax the hard constraint of mean field approximation - minimizing the energy term of each node from probabilistic graphical model, by a global optimization with the proposed dilated sparse convolution module (DSConv). In addition, adaptive global average-pooling and adaptive global max-pooling are implemented as replacements of fully connected layers. In order to integrate DSConv, we design an end-to-end, time-efficient DilatedCRF pipeline. The unary energy term is derived either from pre-softmax and post-softmax features, or the predicted affordance map using a conventional classifier, making it easier to implement DilatedCRF for varieties of classifiers. We also present superior experimental results of proposed approach on the suction dataset comparing to other CRF-based approaches.
BibTeX
@inproceedings{mo2022dilated,
  author = {Xi Mo and Xiangyu Chen and Cuncong Zhong and Rui Li and Kaidong Li and Usman Sajid},
  title = {{Dilated Continuous Random Field for Semantic Segmentation}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2022},
  doi = {10.1109/icra46639.2022.9812064},
}
Figure