Modeling the Distributional Uncertainty for Salient Object Detection Models

Xinyu Tian, Jing Zhang, Mochu Xiang, Yuchao DaiCVPR 2023 · DOI

Introduction: The Challenge of Distributional Uncertainty in SOD

Salient Object Detection (SOD) is the task of identifying objects that naturally draw human attention. While modern deep learning models have achieved high accuracy on standard benchmarks, their reliability in real-world scenarios—where data may differ from the training distribution—remains a critical concern. This paper shifts the focus from simple performance metrics to understanding distributional uncertainty: the gap between training data and "out-of-distribution" (OOD) test samples.

Unlike standard classification tasks where OOD is often defined by the presence of unseen object categories, SOD is inherently class-agnostic. The authors argue that OOD in SOD is "continuous," occurring when test images violate fundamental saliency priors such as center-bias, high color contrast, or shape compactness. This fluid nature makes modeling uncertainty significantly more complex than in discrete classification settings.

Comparison of training data vs. OOD testing samples in SOD.

Methodology

The researchers adapted and evaluated three main categories of techniques to address distributional uncertainty in SOD:

1. Long-tail Learning

Treating "difficult" saliency samples as "tail classes," the authors tested methods designed to handle skewed distributions:

  • Model-rebalance: Utilizing multiple decoders to handle different data distributions.
  • Loss-reweighting: Increasing the penalty for errors on rare or difficult samples to ensure the model doesn't ignore the "tail" of the distribution.
  • Post-hoc Logit Adjustment: Adjusting model outputs after training to account for distribution shifts.

2. Single-model Uncertainty Estimation

These methods estimate pixel-level confidence during a single forward pass:

  • Post-hoc Scores: Using Energy-based scores or Maximum Class Probability (MCP) to gauge confidence.
  • Training-time Improvements: Implementing True Class Probability (TCP), which adds a branch to predict the model's own accuracy, and ReAct, which truncates extreme internal activations to mitigate noise in OOD samples.

3. Test-time Strategies

  • Test-time Training (TTT): Adapting model parameters to a specific test image via self-supervision.
  • Test-time Augmentation (TTA): Aggregating predictions from multiple transformed versions (flips, scales) of the input image.

Key Insights

The study reveals that traditional categorical solutions for long-tail data perform poorly in SOD because they rely on fixed class counts, which do not exist in the fluid world of saliency. In contrast, Deep Ensembles—training multiple models and averaging their outputs—emerged as the most robust method for capturing distributional uncertainty, significantly outperforming Monte Carlo (MC) Dropout.

Another significant finding is the efficacy of Activation Truncation (ReAct). By clipping high-frequency internal activations during the forward pass, models become less likely to make high-confidence mistakes on strange-looking or OOD images.

Key Results and Evaluation

The authors utilized the Variance of Gradient (VoG) score to identify difficult samples within existing datasets like DUTS, ECSSD, and DUT to serve as proxies for OOD data.

  • Uncertainty Quality: The TCP method proved superior for OOD detection, achieving an AUROC of 0.955 and a low False Positive Rate (FPR95 of 0.126) on the DUTS dataset.
  • Ensemble Robustness: Deep Ensembles provided more calibrated and visually accurate uncertainty maps compared to the "free-lunch" MC Dropout method, which often failed to generate reliable uncertainty in complex regions.
Visual comparison of uncertainty maps generated by Base, MCDropout, and DeepEnsemble methods.

Limitations and Future Directions

A major hurdle identified is the application of Test-time Augmentation (TTA) to dense prediction tasks like SOD. While classification labels are invariant to many distortions, SOD requires pixel-perfect spatial correspondence. Many common augmentations destroy the spatial information the model needs. Furthermore, Test-time Training (TTT) carries the risk of "model drift," where the model over-adapts to its own potentially erroneous predictions, leading to degraded performance.

Future research should focus on developing specialized weight regularization to prevent drift and creating augmentation policies specifically tailored for dense, class-agnostic tasks.

Summary updated

BibTeX
@inproceedings{tian2023modeling,
  author = {Xinyu Tian and Jing Zhang and Mochu Xiang and Yuchao Dai},
  title = {{Modeling the Distributional Uncertainty for Salient Object Detection Models}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.01883},
}
Figure