TL;DR
A common assumption in Explainable AI (XAI) is that letting humans interactively guide a model's attention will improve human-AI team performance. This paper tests that hypothesis by introducing CHM-Corr++, an interactive tool for fine-grained bird classification, and finds a surprising result: giving users the power to edit feature importance maps at test time does not significantly improve decision accuracy compared to providing static explanations.
What Problem Does This Paper Address?
Most current XAI tools provide static, one-way explanations—such as heatmaps or feature importance maps—that show a user what a model looked at after it made a prediction. However, there is a growing push toward interactive XAI, where humans can provide real-time feedback to help the model correct its course.
While researchers often hypothesize that this interactivity inherently boosts human-AI team accuracy, empirical validation is scarce. This paper investigates a fundamental question: Does allowing users to interactively edit a model's attention at test time actually improve the team's final decision accuracy?
Methods
To explore this, the authors built CHM-Corr++, an interactive interface built on top of CHM-Corr, a state-of-the-art explainable image classifier.
Here is how the system works:
- Base Model (CHM-Corr): Given an input image, the model finds patch-wise correspondences (on a $7 \times 7$ grid) between the input and the 50 most similar images in the training set, basing its prediction on these similarities.
- Interactive Extension (CHM-Corr++): The authors added a custom HTML/Gradio interface that allows users to manually select or deselect patches on the $7 \times 7$ grid, effectively telling the model exactly where to look (e.g., focusing on the beak instead of the background).
- The Loop: Users can iteratively adjust the attention grid, observe how the model's top predictions and visual explanations change, and then make a final decision to accept or reject the AI's original prediction.
Datasets
The authors evaluated their approach using the CUB-200 (Caltech-UCSD Birds-200-2011) dataset, a standard benchmark for fine-grained visual classification. They curated a balanced evaluation set of 600 total images from the test set: 300 that the base model classified correctly and 300 that it misclassified, establishing a 50% random-chance baseline for the human-AI team.
Key Insights and Ideas
- Interactivity is Not a Silver Bullet: Despite participants actively using the interface (averaging nearly two attention adjustments per decision), the dynamic interaction failed to yield statistically significant accuracy gains over static explanations.
- The Stubbornness of Automation Bias: Humans still struggle immensely with rejecting incorrect AI predictions. Users were highly accurate when the AI was right (~85-86%), but their accuracy plummeted to around ~60% when the AI was wrong—regardless of whether they had access to interactive tools.
- Utility is Outcome-Dependent: Interactivity only truly helps when the user's intervention successfully flips an incorrect AI prediction to a correct one. If the model is fundamentally limited and remains stubborn in its error despite human guidance, the interaction can inadvertently increase user confusion or false confidence.
Key Results
The authors conducted a user study with 18 machine learning experts who made a total of 1,400 decisions:
- Static vs. Dynamic Accuracy: The overall decision accuracy was 72.68% for static explanations and 73.57% for dynamic explanations. This minor difference was not statistically significant ($p = 0.749$).
- Impact of Model Consistency:
- When the AI was originally correct and remained consistent after human interaction, user accuracy was highest (90.80%).
- When the AI was originally incorrect and remained incorrect despite the user changing its attention, user accuracy dropped significantly to 52.55%, showing that a stubborn model misleads humans.
- The interface was most effective when attention control successfully flipped the model from incorrect to correct, driving user accuracy up to 65.43%.
Limitations and Future Work
The authors highlight two main reasons why interactive guidance fell short in this study:
- Task Characteristics: In the CUB-200 dataset, the birds are typically well-centered and clearly visible. The base AI's attention is already sufficient in most cases, meaning human adjustments add little value. Testing this on complex, cluttered scenes where AI naturally struggles to focus would be a better indicator of utility.
- Classifier Limitations: CHM-Corr relies on a $k$-Nearest Neighbors ($k$NN) retrieval step. If the true bird species isn't captured in the initial pool of 50 candidate images, no amount of human attention-tweaking can force the model to output the correct label.
Future work should focus on developing underlying classifiers that can better capitalize on human feedback, rather than just improving the interface itself.
Summary updated