Structured Modeling of Joint Deep Feature and Prediction Refinement for Salient Object Detection

Yingyue Xu, Dan Xu, Xiaopeng Hong, Wanli Ouyang, Rongrong Ji, Min Xu, Guoying Zhao

Abstract

Recent saliency models extensively explore to incorporate multi-scale contextual information from Convolutional Neural Networks (CNNs). Besides direct fusion strategies, many approaches introduce message-passing to enhance CNN features or predictions. However, the messages are mainly transmitted in two ways, by feature-to-feature passing, and by prediction-to-prediction passing. In this paper, we add message-passing between features and predictions and propose a deep unified CRF saliency model. We design a novel cascade CRFs architecture with CNN to jointly refine deep features and predictions at each scale and progressively compute a final refined saliency map. We formulate the CRF graphical model that involves message-passing of feature-feature, feature-prediction, and prediction-prediction, from the coarse scale to the finer scale, to update the features and the corresponding predictions. Also, we formulate the mean-field updates for joint end-to-end model training with CNN through back propagation. The proposed deep unified CRF saliency model is evaluated over six datasets and shows highly competitive performance among the state of the arts.

Introduction

Salient Object Detection (SOD) aims to identify the most visually conspicuous objects in a scene, mimicking the human visual system's ability to prioritize information. While modern Deep Learning models, particularly CNNs, excel at extracting multi-scale features, they often encounter two primary challenges. First, high-level features are semantically rich but spatially blurry, while low-level features are sharp but prone to noise. Second, existing refinement methods typically treat features and predictions in isolation, missing the potential synergy between them.

Methodology

The core innovation of this research is a cascade CRF architecture integrated directly into a CNN backbone. Unlike standard Conditional Random Fields (CRFs) that only model labels (predictions), this model treats both deep features ($h$) and predictions ($o$) as hidden variables in a unified graph.

Joint Graphical Model and Message Passing

The model employs a multi-way message-passing scheme that accounts for three distinct types of interactions:

  1. Feature-to-Feature: Passing context between different layers to enhance semantic consistency.
  2. Prediction-to-Prediction: Ensuring spatial consistency across the output map, similar to standard Dense-CRF approaches.
  3. Feature-to-Prediction: The critical "joint" component where rough prediction maps provide spatial and shape context to clean up noisy features, which in turn produce sharper final predictions.
Message passing schemes: (a) feature-to-feature, (b) prediction-to-prediction, and (c) the proposed joint refinement.

To ensure the model is trainable end-to-end, the mean-field inference updates are formulated as a series of convolutional and deconvolutional layers. This allows the entire CRF block to be optimized via standard backpropagation within the CNN pipeline. The model operates in a cascade, working from coarse scales to fine scales to progressively refine the saliency map.

Experimental Evaluation

The model was trained on the MSRA-B dataset and evaluated across six major benchmarks, including ECSSD, PASCAL-S, DUT-OMRON, HKU-IS, and iCoseg. These datasets represent a wide variety of challenges, from complex scenes to multiple salient objects.

Quantitative Results

The Deep Unified CRF consistently outperformed or matched ten state-of-the-art models. On the ECSSD dataset, the model achieved a Max F-measure of 0.928 and a remarkably low Mean Absolute Error (MAE) of 0.049. Ablation studies confirmed the efficacy of the joint approach: adding joint feature-prediction message passing improved the F-measure from a baseline of 0.884 to 0.928.

Qualitative Analysis

Visual results demonstrate the model's ability to produce sharp, accurate saliency maps even in complex environments. The interaction between features and predictions allows the model to suppress background noise that might otherwise be captured by standard multi-scale fusion.

Visual demonstration of the saliency detection process, showing the input image (i), the resulting saliency map (ii), and intermediate feature representations (iii, iv).

Conclusion

By bridging the gap between feature refinement and prediction refinement, the proposed deep unified CRF model offers a more holistic approach to salient object detection. The use of predictions as context for feature cleaning proves to be a powerful mechanism for improving both overall accuracy and spatial precision. While the model shows some sensitivity to specific data augmentation techniques and carries the complexity of a multi-stage training process, its state-of-the-art performance and computational efficiency (0.48s per image) make it a significant advancement in the field.

Summary updated

Abstract

Recent saliency models extensively explore to incorporate multi-scale contextual information from Convolutional Neural Networks (CNNs). Besides direct fusion strategies, many approaches introduce message-passing to enhance CNN features or predictions. However, the messages are mainly transmitted in two ways, by feature-to-feature passing, and by prediction-to-prediction passing. In this paper, we add message-passing between features and predictions and propose a deep unified CRF saliency model . We design a novel cascade CRFs architecture with CNN to jointly refine deep features and predictions at each scale and progressively compute a final refined saliency map. We formulate the CRF graphical model that involves message-passing of feature-feature, feature-prediction, and prediction-prediction, from the coarse scale to the finer scale, to update the features and the corresponding predictions. Also, we formulate the mean-field updates for joint end-to-end model training with CNN through back propagation. The proposed deep unified CRF saliency model is evaluated over six datasets and shows highly competitive performance among the state of the arts.
BibTeX
@inproceedings{xu2019structured,
  author = {Yingyue Xu and Dan Xu and Xiaopeng Hong and Wanli Ouyang and Rongrong Ji and Min Xu and Guoying Zhao},
  title = {{Structured Modeling of Joint Deep Feature and Prediction Refinement for Salient Object Detection}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00389},
}
Figure