A Projected Gradient Descent Method for CRF Inference Allowing End-to-End Training of Arbitrary Pairwise Potentials.

Måns Larsson, Anurag Arnab, Fredrik Kahl, Shuai Zheng, Philip H. S. Torr

Abstract

Are we using the right potential functions in the Conditional Random Field models that are popular in the Vision community? Semantic segmentation and other pixel-level labelling tasks have made significant progress recently due to the deep learning paradigm. However, most state-of-the-art structured prediction methods also include a random field model with a hand-crafted Gaussian potential to model spatial priors, label consistencies and feature-based image conditioning. In this paper, we challenge this view by developing a new inference and learning framework which can learn pairwise CRF potentials restricted only by their dependence on the image pixel values and the size of the support. Both standard spatial and high-dimensional bilateral kernels are considered. Our framework is based on the observation that CRF inference can be achieved via projected gradient descent and consequently, can easily be integrated in deep neural networks to allow for end-to-end training. It is empirically demonstrated that such learned potentials can improve segmentation accuracy and that certain label class interactions are indeed better modelled by a non-Gaussian potential. In addition, we compare our inference method to the commonly used mean-field algorithm. Our framework is evaluated on several public benchmarks for semantic segmentation with improved performance compared to previous state-of-the-art CNN+CRF models.

Introduction

In semantic segmentation, Convolutional Neural Networks (CNNs) have become the standard for generating pixel-level predictions. Despite their power, CNN outputs often lack spatial coherence, resulting in "blobby" or noisy segmentations. To address this, researchers typically employ Conditional Random Fields (CRFs) as a structured prediction layer to enforce label consistency. However, traditional CRFs rely heavily on hand-crafted Gaussian potentials, assuming that spatial relationships between classes follow a simple bell curve. This paper introduces CRF-Grad, a framework that challenges this assumption by allowing the model to learn arbitrary pairwise potentials directly from data.

The CRF-Grad Framework

The core innovation of CRF-Grad lies in its approach to inference. While standard methods often use Mean-Field (MF) inference—which minimizes a proxy KL-divergence—CRF-Grad treats inference as a constrained optimization problem solved via Projected Gradient Descent (PGD).

By unrolling the iterative PGD steps, the inference process becomes a Recurrent Neural Network (RNN). This unrolling ensures that every operation, including the projection back onto the probability simplex, is differentiable. Consequently, the entire pipeline—from the initial feature-extracting CNN to the final CRF refinement—can be trained end-to-end.

The end-to-end architecture unrolling PGD steps as a recurrent network.

Learning Arbitrary Potentials

Unlike traditional CRFs that use fixed Gaussian kernels, CRF-Grad implements spatial pairwise terms as standard 2D convolutions. This allows the model to learn the "true" shape of spatial priors. For instance, the relationship between a "sidewalk" and a "road" or a "traffic light" and its surrounding "terrain" may not be symmetric or Gaussian.

One of the most significant findings is the discovery of "oblong" filters. For vertical objects like traffic signs or lights, the model learns filters that extend vertically, reflecting the physical reality that these objects are often supported by poles or situated in specific vertical contexts. This flexibility allows the model to capture complex, non-symmetric spatial priors that Gaussian kernels simply cannot represent.

Learned non-Gaussian spatial potentials for various class interactions, showing asymmetric and oblong shapes.

Experimental Results

The framework was evaluated on several benchmarks, including NYU V2 and Cityscapes.

  • NYU V2: CRF-Grad achieved a mean Intersection over Union (IoU) of 55.0%, outperforming standard CRF-as-RNN models.
  • Cityscapes: The model reached 71.9% IoU, with notable improvements in small, vertically-oriented classes like traffic lights (nearly 2% gain).

The results indicate that PGD-based inference not only allows for more flexible potentials but also avoids the "uncertainty bias" of Mean-Field methods, which tend to favor solutions with high entropy. In contrast, PGD pushes pixel values toward more confident, lower-energy states. Furthermore, the authors found that just 5 to 6 iterations were sufficient to reach peak performance, making it practical for real-time applications.

Conclusion

CRF-Grad demonstrates that moving beyond hand-crafted Gaussian potentials can significantly improve semantic segmentation. By integrating PGD-based inference into a deep learning framework, we can learn complex spatial relationships that were previously ignored. While the increased flexibility introduces a risk of overfitting on small datasets, the performance gains on large-scale benchmarks like Cityscapes highlight the potential of learned structured prediction.

Summary updated

Abstract

Are we using the right potential functions in the Conditional Random Field models that are popular in the Vision community? Semantic segmentation and other pixel-level labelling tasks have made significant progress recently due to the deep learning paradigm. However, most state-of-the-art structured prediction methods also include a random field model with a hand-crafted Gaussian potential to model spatial priors, label consistencies and feature-based image conditioning. In this paper, we challenge this view by developing a new inference and learning framework which can learn pairwise CRF potentials restricted only by their dependence on the image pixel values and the size of the support. Both standard spatial and high-dimensional bilateral kernels are considered. Our framework is based on the observation that CRF inference can be achieved via projected gradient descent and consequently, can easily be integrated in deep neural networks to allow for end-to-end training. It is empirically demonstrated that such learned potentials can improve segmentation accuracy and that certain label class interactions are indeed better modelled by a non-Gaussian potential. In addition, we compare our inference method to the commonly used mean-field algorithm. Our framework is evaluated on several public benchmarks for semantic segmentation with improved performance compared to previous state-of-the-art CNN+CRF models.
BibTeX
@inproceedings{larsson2017projected,
  author = {Måns Larsson and Anurag Arnab and Fredrik Kahl and Shuai Zheng and Philip H. S. Torr},
  title = {{A Projected Gradient Descent Method for CRF Inference Allowing End-to-End Training of Arbitrary Pairwise Potentials.}},
  booktitle = {EMMCVPR},
  year = {2017},
  url = {https://openreview.net/forum?id=W0SxTmbuUqE},
}
Figure