Can Explanations Be Useful for Calibrating Black Box Models?

Xi Ye, Greg Durrett

Abstract

Modern NLP models often struggle with reliability when deployed on out-of-domain data, frequently exhibiting high confidence in incorrect predictions. This paper investigates whether model-agnostic explanations, such as LIME and SHAP, can be leveraged to build a secondary "calibrator." By transforming raw word-level attributions into human-interpretable features—such as the model's focus on proper nouns or specific segments—the authors demonstrate significant improvements in identifying model failures in black-box settings without requiring weight updates.

Introduction

In real-world applications, NLP models are frequently treated as "black boxes" due to proprietary restrictions or computational constraints. A critical issue arises when these models encounter out-of-domain (OOD) data, where they tend to be "confident but wrong." Standard calibration techniques, which rely on the model's internal probability scores (e.g., MaxProb), often fail to generalize to these new distributions. This work proposes a method to calibrate black-box models by analyzing why they make specific predictions rather than just what they predict.

Methodology

The proposed pipeline consists of three main stages:

  1. Explanation Generation: Using model-agnostic techniques like LIME or SHAP, the system identifies which input tokens most influence a specific prediction by perturbing the input.
  2. Feature Extraction: Raw attributions are aggregated into intuitive categories. These include segments (Question vs. Context), Part-of-Speech (POS) tags (e.g., proper nouns, verbs), and word overlap in NLI tasks.
  3. The Calibrator: A Random Forest classifier is trained on a small target-domain dataset (approx. 500 examples). It uses the aggregated attribution features to predict the likelihood that the base model's answer is correct.
Example of how feature attributions to proper nouns (NNP) can signal prediction correctness.

Key Insights: Heuristics and Attribution

The core intuition is that human-like heuristics can guide calibration. For instance, in Question Answering (QA), a model should ideally attend to proper nouns (NNP) in the question to identify the subject. As illustrated in the methodology, a lower attribution to these key entities often correlates with incorrect predictions. By learning these patterns, the calibrator acts as a "BS detector."

Notably, this approach is highly data-efficient, requiring only a few hundred examples to achieve robust calibration. In some OOD scenarios, this black-box calibration even outperforms "glass-box" fine-tuning, which can be unstable on small datasets and may degrade the base model's general knowledge.

Experimental Results

The method was evaluated on extractive QA (transferring from SQuAD to HotpotQA/TriviaQA) and NLI (transferring from MNLI to QNLI/MRPC).

Average F1 vs. Coverage for different calibration methods on out-of-domain QA tasks.

On adversarial datasets like SQuAD-ADV, the explanation-based calibrator (LimeCal) significantly outperformed the standard MaxProb baseline. As seen in the performance curves, when the model is allowed to be selective (answering only high-confidence questions), the F1 score remains substantially higher than baselines across various coverage levels. While improvements in NLI were more modest due to the inherent strength of simple word-overlap heuristics in those datasets, the QA results highlight the power of attribution-based features for complex reasoning tasks.

Limitations and Future Work

Despite its success, the approach has limitations. Generating LIME/SHAP explanations is computationally expensive, requiring thousands of model passes per input, which may hinder real-time deployment. Additionally, the current system relies on manually defined POS-tag heuristics. Future research could explore automated rationale discovery and methods to reduce the inference-time overhead of generating explanations.

Summary updated

Abstract

NLP practitioners often want to take existing trained models and apply them to data from new domains. While fine-tuning or few-shot learning can be used to adapt a base model, there is no single recipe for making these techniques work; moreover, one may not have access to the original model weights if it is deployed as a black box. We study how to improve a black box model's performance on a new domain by leveraging explanations of the model's behavior. Our approach first extracts a set of features combining human intuition about the task with model attributions generated by black box interpretation techniques, then uses a simple calibrator, in the form of a classifier, to predict whether the base model was correct or not. We experiment with our method on two tasks, extractive question answering and natural language inference, covering adaptation from several pairs of domains with limited target-domain data. The experimental results across all the domain pairs show that explanations are useful for calibrating these models, boosting accuracy when predictions do not have to be returned on every example. We further show that the calibration model transfers to some extent between tasks.
BibTeX
@inproceedings{ye2022can,
  author = {Xi Ye and Greg Durrett},
  title = {{Can Explanations Be Useful for Calibrating Black Box Models?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=VkHFixN7zR},
}

Loading related content...

Figure