Allowing humans to interactively guide machines where to look does not always improve human-AI team's classification accuracy

Giang Nguyen, Mohammad Reza Taesiri, Sunnie S. Y. Kim, Anh Nguyen

TL;DR

A common assumption in Explainable AI (XAI) is that letting humans interactively guide a model's attention will improve human-AI team performance. This paper tests that hypothesis by introducing CHM-Corr++, an interactive tool for fine-grained bird classification, and finds a surprising result: giving users the power to edit feature importance maps at test time does not significantly improve decision accuracy compared to providing static explanations.

What Problem Does This Paper Address?

Most current XAI tools provide static, one-way explanations—such as heatmaps or feature importance maps—that show a user what a model looked at after it made a prediction. However, there is a growing push toward interactive XAI, where humans can provide real-time feedback to help the model correct its course.

While researchers often hypothesize that this interactivity inherently boosts human-AI team accuracy, empirical validation is scarce. This paper investigates a fundamental question: Does allowing users to interactively edit a model's attention at test time actually improve the team's final decision accuracy?

Methods

To explore this, the authors built CHM-Corr++, an interactive interface built on top of CHM-Corr, a state-of-the-art explainable image classifier.

Figure 1. Users provide feedback to the explainable AI by editing the feature importance map (like an “attention map” [50]).

Here is how the system works:

  1. Base Model (CHM-Corr): Given an input image, the model finds patch-wise correspondences (on a $7 \times 7$ grid) between the input and the 50 most similar images in the training set, basing its prediction on these similarities.
  2. Interactive Extension (CHM-Corr++): The authors added a custom HTML/Gradio interface that allows users to manually select or deselect patches on the $7 \times 7$ grid, effectively telling the model exactly where to look (e.g., focusing on the beak instead of the background).
  3. The Loop: Users can iteratively adjust the attention grid, observe how the model's top predictions and visual explanations change, and then make a final decision to accept or reject the AI's original prediction.
Figure 2. Our CHM-Corr++ interactive interface workflow.

Datasets

The authors evaluated their approach using the CUB-200 (Caltech-UCSD Birds-200-2011) dataset, a standard benchmark for fine-grained visual classification. They curated a balanced evaluation set of 600 total images from the test set: 300 that the base model classified correctly and 300 that it misclassified, establishing a 50% random-chance baseline for the human-AI team.

Key Insights and Ideas

  • Interactivity is Not a Silver Bullet: Despite participants actively using the interface (averaging nearly two attention adjustments per decision), the dynamic interaction failed to yield statistically significant accuracy gains over static explanations.
  • The Stubbornness of Automation Bias: Humans still struggle immensely with rejecting incorrect AI predictions. Users were highly accurate when the AI was right (~85-86%), but their accuracy plummeted to around ~60% when the AI was wrong—regardless of whether they had access to interactive tools.
  • Utility is Outcome-Dependent: Interactivity only truly helps when the user's intervention successfully flips an incorrect AI prediction to a correct one. If the model is fundamentally limited and remains stubborn in its error despite human guidance, the interaction can inadvertently increase user confusion or false confidence.

Key Results

The authors conducted a user study with 18 machine learning experts who made a total of 1,400 decisions:

  • Static vs. Dynamic Accuracy: The overall decision accuracy was 72.68% for static explanations and 73.57% for dynamic explanations. This minor difference was not statistically significant ($p = 0.749$).
  • Impact of Model Consistency:
  • When the AI was originally correct and remained consistent after human interaction, user accuracy was highest (90.80%).
  • When the AI was originally incorrect and remained incorrect despite the user changing its attention, user accuracy dropped significantly to 52.55%, showing that a stubborn model misleads humans.
  • The interface was most effective when attention control successfully flipped the model from incorrect to correct, driving user accuracy up to 65.43%.

Limitations and Future Work

The authors highlight two main reasons why interactive guidance fell short in this study:

  1. Task Characteristics: In the CUB-200 dataset, the birds are typically well-centered and clearly visible. The base AI's attention is already sufficient in most cases, meaning human adjustments add little value. Testing this on complex, cluttered scenes where AI naturally struggles to focus would be a better indicator of utility.
  2. Classifier Limitations: CHM-Corr relies on a $k$-Nearest Neighbors ($k$NN) retrieval step. If the true bird species isn't captured in the initial pool of 50 candidate images, no amount of human attention-tweaking can force the model to output the correct label.

Future work should focus on developing underlying classifiers that can better capitalize on human feedback, rather than just improving the interface itself.

Summary updated

Abstract

Via thousands of papers in Explainable AI (XAI), attention maps \cite{vaswani2017attention} and feature importance maps \cite{bansal2020sam} have been established as a common means for finding how important each input feature is to an AI's decisions. It is an interesting, unexplored question whether allowing users to edit the feature importance at test time would improve a human-AI team's accuracy on downstream tasks. In this paper, we address this question by leveraging CHM-Corr, a state-of-the-art, ante-hoc explainable classifier \cite{taesiri2022visual} that first predicts patch-wise correspondences between the input and training-set images, and then bases on them to make classification decisions. We build CHM-Corr++, an interactive interface for CHM-Corr, enabling users to edit the feature importance map provided by CHM-Corr and observe updated model decisions. Via CHM-Corr++, users can gain insights into if, when, and how the model changes its outputs, improving their understanding beyond static explanations. However, our study with 18 expert users who performed 1,400 decisions finds no statistical significance that our interactive approach improves user accuracy on CUB-200 bird image classification over static explanations. This challenges the hypothesis that interactivity can boost human-AI team accuracy and raises needs for future research. We open-source CHM-Corr++, an interactive tool for editing image classifier attention (see an interactive demo here: http://137.184.82.109:7080/). We release code and data on github: https://github.com/anguyen8/chm-corr-interactive.
BibTeX
@inproceedings{nguyen2024allowing,
  author = {Giang Nguyen and Mohammad Reza Taesiri and Sunnie S. Y. Kim and Anh Nguyen},
  title = {{Allowing humans to interactively guide machines where to look does not always improve human-AI team's classification accuracy}},
  booktitle = {Xai4cv},
  year = {2024},
}
Figure