BOREx: Bayesian-Optimization-Based Refinement of Saliency Map for Image- and Video-Classification Models

Atsushi Kikuchi, Kotaro Uchida, Masaki Waga, Kohei Suenaga

Introduction

Deep learning models, despite their state-of-the-art performance, often operate as "black boxes," making their decision-making processes opaque to users. In safety-critical domains such as autonomous driving or medical diagnostics, understanding why a model arrives at a specific classification is as crucial as the classification itself. Saliency maps—visual heatmaps that highlight influential pixels or frames—have emerged as a primary tool for model interpretability. However, existing methods often face a trade-off between computational efficiency and the precision of the resulting explanation.

This paper introduces BOREx (Bayesian Optimization for Refinement of visual model Explanation), a novel black-box method designed to refine existing saliency maps. By treating initial, noisy explanations as a "prior" and employing Gaussian Process Regression (GPR), BOREx iteratively samples the model to produce high-fidelity, localized explanations for both image and video classifiers.

The Interpretability Trade-off

Current saliency map techniques generally fall into two categories: "white-box" and "black-box" methods. White-box methods, such as Grad-CAM, are computationally efficient but require internal access to the model’s gradients. Conversely, black-box methods like RISE are model-agnostic but typically produce blurry, low-resolution maps or require an impractical number of model inferences to achieve clarity.

BOREx addresses this gap by observing that even low-quality "rough draft" maps contain valuable information. Instead of starting from scratch, BOREx uses these existing maps to guide an optimization process, achieving high-resolution results without requiring internal model access.

Methodology: Refining the Prior

The BOREx framework is built upon Bayesian Optimization powered by Gaussian Process Regression. The process follows several key innovations:

1. Leveraging the Prior

Unlike standard optimization approaches that begin with a blank slate, BOREx incorporates an existing saliency map (e.g., from RISE or Grad-CAM) as an initial prior. This guides the search toward likely important regions from the first iteration, significantly accelerating convergence.

2. Iterative Sampling and Acquisition

The algorithm selects sampling locations and mask sizes using an acquisition function. This function balances "exploitation" (sampling areas known to be important) with "exploration" (investigating areas of high uncertainty). To ensure spatial smoothness and interpretability, the authors utilize a Matérn kernel, which assumes that saliency is spatially correlated.

3. The Flipped Mask Strategy

A significant challenge for perturbation-based methods is the presence of multiple identical objects (e.g., a school of fish). If one object is occluded, the model's confidence may remain high due to the presence of others, leading the algorithm to incorrectly label the occluded region as unimportant. BOREx solves this by using "flipped" masks—measuring the model's response when only a specific region is visible.

4. Area-Weighted Averaging

In the final map generation, BOREx applies area-weighted averaging. Observations made with smaller masks are given higher weight, as they provide more precise localization data compared to larger, more general masks.

Extension to Video

BOREx extends its logic to the temporal domain for video classification. By utilizing 3D "saliency tubes"—represented as rectangular cuboid masks—the method tracks importance across both spatial dimensions and time. This allows the model to explain actions in datasets like Kinetics-400 by following moving subjects more accurately than static white-box methods.

Experimental Results

The researchers evaluated BOREx using the PascalVOC (images) and Kinetics-400 (video) datasets, testing explanations on ResNet-152 and i3D models. Performance was measured using three primary metrics:

  • Insertion: The rate at which model confidence increases as "important" pixels are added.
  • Deletion: The rate at which confidence drops as "important" pixels are removed.
  • F-measure: The alignment of the saliency map with human-annotated bounding boxes.

BOREx demonstrated statistically significant improvements ($p < 0.001$) over baseline methods like RISE and Grad-CAM++. Notably, it outperformed previous Bayesian Optimization attempts by successfully integrating the initial prior and the flipped mask strategy. In video tasks, BOREx proved particularly adept at following dynamic objects, such as a skier, where traditional methods often produced overly broad or static highlights.

Limitations and Future Work

While BOREx represents a significant step forward, several challenges remain. The current use of square or rectangular masks limits the method's ability to capture complex, non-rectangular object contours. Additionally, the refinement process takes approximately 9 seconds per image, which, while suitable for auditing and research, is currently too slow for real-time applications. Finally, the optimization can occasionally be misled if the initial prior is extremely inaccurate, though the exploration component of the acquisition function is designed to mitigate this risk.

Future research will focus on implementing more flexible mask geometries to better capture the natural shapes of salient objects.

Summary updated

BibTeX
@inproceedings{kikuchi2022borex,
  author = {Atsushi Kikuchi and Kotaro Uchida and Masaki Waga and Kohei Suenaga},
  title = {{BOREx: Bayesian-Optimization-Based Refinement of Saliency Map for Image- and Video-Classification Models}},
  booktitle = {ACCV},
  year = {2022},
  doi = {10.1007/978-3-031-26293-7\_17},
}
Figure