Visualizing Color-Wise Saliency of Black-Box Image Classification Models

Yuhki Hatakeyama, Hiroki Sakuma, Yoshinori Konishi, Kohei Suenaga

Abstract

As deep learning models are increasingly deployed in safety-critical systems, understanding the rationale behind their decisions is paramount. Existing black-box explanation methods generate saliency maps that highlight important spatial regions but often ignore the critical role of color and suffer from noisy, non-zero baselines. We present MC-RISE (Multi-Color Randomized Input Sampling for Explanation), a novel model-agnostic method that generates color-aware saliency maps. By utilizing randomly sampled colored masks and a debiasing mathematical formulation, MC-RISE distinguishes between texture and color sensitivity while ensuring that irrelevant pixels have a saliency score of zero. Our evaluations on traffic sign recognition and person re-identification demonstrate that MC-RISE provides more interpretable and actionable insights than standard saliency methods.

Introduction

Deep learning models, particularly in domains like traffic sign recognition, rely heavily on color features. However, standard explanation methods like LIME, SHAP, and RISE typically produce heatmaps that only indicate where a model is looking, not what specific features (like color) are driving the decision. For instance, a model might focus on a stop sign, but it remains unclear if it recognizes the octagonal shape or the red color.

Furthermore, standard saliency maps often suffer from "shifting baselines," where background pixels receive non-zero scores, making it difficult for humans to distinguish signal from noise. MC-RISE addresses these issues by introducing color-aware probing and a debiased scoring mechanism.

Comparison of standard RISE and the proposed MC-RISE. (a) Input image. (b) Standard RISE map with a noisy baseline. (c) MC-RISE maps showing sensitivity to specific colors (red, green, blue, white, black).

Methodology

MC-RISE builds upon the RISE framework, which probes models by applying random binary masks and observing changes in confidence. The authors introduce two key innovations:

1. Debiased RISE

The core RISE formula is modified to subtract the expected confidence when a pixel is masked from the expected confidence when it is visible. This mathematical adjustment shifts the baseline such that irrelevant pixels score exactly zero. Features that support the prediction score positively, while those that hinder it score negatively.

2. Multi-Color RISE (MC-RISE)

Instead of binary masks that simply hide pixels (e.g., by turning them black), MC-RISE applies masks of various colors (red, green, blue, white, black). By blending the input image with these colored patches, the method generates a separate saliency map for each color, revealing how the model's confidence would change if a specific region were forced to a specific color.

The MC-RISE pipeline: Input images are processed with random color masks and non-masked pixel maps. The black-box model's outputs are then used to compute weighted sums, resulting in raw saliency maps and a baseline map, which are subtracted to produce the final color-wise saliency maps.

Experiments and Results

The method was evaluated on several datasets, including the German Traffic Sign Recognition Benchmark (GTSRB), ImageNet, and Market-1501.

Quantitative Evaluation

The authors introduced the CA-deletion (Color-Aware deletion) metric, which measures the drop in model confidence when important pixels are masked with their most sensitive color. On the GTSRB dataset using a VGG-16 model, MC-RISE achieved a CA-deletion score of 0.0270, significantly outperforming standard RISE (0.0664) and LIME (0.1324).

Qualitative Insights

  • Color vs. Texture: On ImageNet, saliency maps across different colors were similar, indicating a reliance on texture. On GTSRB, MC-RISE highlighted that the red borders of stop signs are critical.
  • Error Analysis: MC-RISE can diagnose failures. In one instance, a misclassified traffic sign was shown to have an overexposed (white) center. MC-RISE revealed that the model's confidence for the correct class would have increased if that center had been red, pinpointing the exact cause of the error.
  • Person Re-Identification: In metric-learning tasks, MC-RISE successfully visualized which clothing colors the model used to match individuals across different camera views.

Conclusion

MC-RISE provides a significant step forward in model-agnostic explanations by incorporating color sensitivity and providing a cleaner, zero-baselined visualization. While the method is computationally intensive due to the need for thousands of model evaluations per image, its ability to provide actionable error analysis and separate color from texture makes it a valuable tool for auditing safety-critical vision systems. Future work may extend this masking approach to other features, such as image blur, to further decompose model decision-making.

Summary updated

BibTeX
@inproceedings{hatakeyama2020visualizing,
  author = {Yuhki Hatakeyama and Hiroki Sakuma and Yoshinori Konishi and Kohei Suenaga},
  title = {{Visualizing Color-Wise Saliency of Black-Box Image Classification Models}},
  booktitle = {ACCV},
  year = {2020},
  doi = {10.1007/978-3-030-69535-4\_12},
}
Figure