Structured Modeling of Joint Deep Feature and Prediction Refinement for Salient Object Detection

Yingyue Xu, Dan Xu, Xiaopeng Hong, Wanli Ouyang, Rongrong Ji, Min Xu, Guoying Zhao

Introduction

Salient Object Detection (SOD) is a fundamental computer vision task that aims to identify and segment the most conspicuous objects in an image, mimicking the human visual system. While deep Convolutional Neural Networks (CNNs) have significantly advanced the field by extracting powerful multi-scale features, they often struggle to maintain global consistency or capture fine-grained object boundaries.

Existing methods frequently employ "message-passing" techniques, such as Conditional Random Fields (CRFs), to address these issues. However, these refinements typically occur in isolation: they either pass messages between features to improve representation or between predictions to smooth the final output. This paper introduces a novel Deep Unified CRF framework that breaks these silos by enabling joint refinement of both deep features and saliency predictions.

The Deep Unified CRF Framework

The core innovation of this work is the integration of a joint CRF block into a cascaded CNN architecture. Unlike standard post-processing CRFs, this unified model facilitates three distinct types of information sharing:

  1. Feature-to-Feature (f-f): Refines deep representations to ensure feature consistency.
  2. Prediction-to-Prediction (s-s): Encourages spatial smoothness and consistency in the saliency map.
  3. Feature-to-Prediction (f-s): A novel bidirectional interaction where coarse predictions provide shape context to features, and refined features sharpen the final predictions.
The three types of message-passing interactions: (a) feature-to-feature, (b) prediction-to-prediction, and (c) the proposed joint feature-prediction refinement.

By formulating the CRF's mean-field inference as a series of differentiable CNN layers, the entire framework can be trained end-to-end. This allows the model to learn scale-specific kernels—using similarity-based grouping at coarse scales and proximity-based smoothing at fine scales.

Key Insights: Mutual Reinforcement

The authors identify that coarse saliency maps, while lacking detail, contain vital structural information. By passing messages from these rough predictions back to the deep features, the model becomes "aware" of object boundaries earlier in the network. This mutual reinforcement ensures that features are guided by the emerging object shape, while predictions remain grounded in the high-resolution details of the refined features.

Visualizing the refinement process: (i) input image, (ii) initial coarse prediction, and (iii-iv) stages of refinement showing increased detail and focus on the salient object.

Experimental Results

The proposed model was evaluated across six benchmark datasets, including MSRA-B, ECSSD, and DUT-OMRON. It consistently outperformed ten state-of-the-art methods in both accuracy and error reduction:

  • Accuracy: On the ECSSD dataset, the model achieved a Max F-measure of 0.928, a significant jump from the 0.884 baseline.
  • Error Reduction: The Mean Absolute Error (MAE) was notably lower on difficult datasets like DUT-OMRON (0.057 vs. the next best 0.062).
  • Efficiency: The cascaded architecture maintains a competitive inference speed of approximately 0.48s per image, making it practical for real-world applications.

Conclusion

By moving the CRF from a post-processing step to a core, differentiable component of the network, this research demonstrates that joint feature and prediction refinement is key to high-quality salient object detection. The authors suggest that this structured modeling approach could be extended to other pixel-labeling tasks, such as semantic segmentation or depth estimation, where the interplay between features and predictions is equally vital.

Summary updated

Abstract

Recent saliency models extensively explore to incorporate multi-scale contextual information from Convolutional Neural Networks (CNNs). Besides direct fusion strategies, many approaches introduce message-passing to enhance CNN features or predictions. However, the messages are mainly transmitted in two ways, by feature-to-feature passing, and by prediction-to-prediction passing. In this paper, we add message-passing between features and predictions and propose a deep unified CRF saliency model . We design a novel cascade CRFs architecture with CNN to jointly refine deep features and predictions at each scale and progressively compute a final refined saliency map. We formulate the CRF graphical model that involves message-passing of feature-feature, feature-prediction, and prediction-prediction, from the coarse scale to the finer scale, to update the features and the corresponding predictions. Also, we formulate the mean-field updates for joint end-to-end model training with CNN through back propagation. The proposed deep unified CRF saliency model is evaluated over six datasets and shows highly competitive performance among the state of the arts.
BibTeX
@inproceedings{xu2019structured,
  author = {Yingyue Xu and Dan Xu and Xiaopeng Hong and Wanli Ouyang and Rongrong Ji and Min Xu and Guoying Zhao},
  title = {{Structured Modeling of Joint Deep Feature and Prediction Refinement for Salient Object Detection}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00389},
}
Figure