Abstract
NLP practitioners often want to take existing trained models and apply them to data from new domains. While fine-tuning or few-shot learning can be used to adapt a base model, there is no single recipe for making these techniques work; moreover, one may not have access to the original model weights if it is deployed as a black box. We study how to improve a black box model's performance on a new domain by leveraging explanations of the model's behavior. Our approach first extracts a set of features combining human intuition about the task with model attributions generated by black box interpretation techniques, then uses a simple calibrator, in the form of a classifier, to predict whether the base model was correct or not. We experiment with our method on two tasks, extractive question answering and natural language inference, covering adaptation from several pairs of domains with limited target-domain data. The experimental results across all the domain pairs show that explanations are useful for calibrating these models, boosting accuracy when predictions do not have to be returned on every example. We further show that the calibration model transfers to some extent between tasks.
Introduction
In the modern NLP landscape, models are frequently deployed as "black boxes"—either as proprietary APIs or due to hardware constraints that prevent full model inspection. A significant challenge arises when these models encounter data from a new "target" domain that differs from their training "source" domain. In such out-of-distribution (OOD) scenarios, performance typically degrades, and models often become "confidently wrong," providing incorrect answers with high probability scores.
Standard calibration—knowing when to trust a model's probability score—usually requires access to the model's internal weights or a massive amount of data to fine-tune. This research investigates whether model "explanations" (specifically feature attributions that highlight which input tokens influenced a decision) can serve as a reliable signal for detecting errors in a black-box setting.
Methodology: The InterpCalib Pipeline
The authors propose InterpCalib, a framework that uses interpretability tools to calibrate black-box models. The pipeline consists of four main stages:
- Prediction: The primary black-box model (e.g., RoBERTa) generates a prediction for a given input.
- Explanation Generation: Model-agnostic techniques like LIME or SHAP are used to produce feature attributions. These scores quantify the contribution of each input token to the final prediction.
- Feature Extraction: Raw attribution scores are aggregated using human-intuitive heuristics. For instance, in Question Answering, features might include the total attribution assigned to Proper Nouns (NNP) in the question versus verbs in the context.
- Calibration: A secondary "calibrator" model—a simple Random Forest classifier—is trained on a small labeled subset of the target domain. It uses the engineered attribution features to predict whether the primary model's answer is correct.
The core intuition is that models exhibit distinct "reasoning" patterns when they fail. For example, a QA model might be more likely to be correct if it focuses on specific entities in the question. If it instead attends to irrelevant context words, it is likely being misled by a distractor.
Experimental Setup
The method was evaluated across two core NLP tasks involving significant domain shifts:
- Extractive Question Answering (QA): Models trained on SQuAD (Wikipedia) were tested on SQuAD-ADV (adversarial), HotpotQA (multi-hop), and TriviaQA (web snippets).
- Natural Language Inference (NLI): Models trained on MNLI were tested on QNLI and MRPC.
These pairs represent scenarios where linguistic styles and reasoning requirements vary significantly, challenging the robustness of the base models.
Key Results and Insights
The experimental results demonstrate that explanation-based features provide a much stronger signal for calibration than raw confidence scores (MaxProb), especially in OOD settings.
Performance Gains
The proposed methods, LimeCal and ShapCal, consistently outperformed standard baselines. On the adversarial SQuAD-ADV dataset, LimeCal achieved a 7.1-point improvement in Area Under the Coverage-F1 curve (AUC) compared to existing state-of-the-art calibration techniques.
Selective Question Answering
One of the most practical applications of this work is "Selective QA," where a model can choose to abstain from answering if the calibrator predicts an error. As shown in the figure below, as the "coverage" (the percentage of questions answered) decreases, the Average F1 score of the system increases significantly. This allows for highly reliable systems in critical applications where a "don't know" is better than a wrong answer.
Data Efficiency and Transferability
InterpCalib is remarkably data-efficient, requiring only 100–500 target-domain examples to train an effective calibrator. Furthermore, the researchers found that a calibrator trained on one target domain (e.g., TriviaQA) could often generalize to another (e.g., HotpotQA), suggesting that the patterns of "bad reasoning" are somewhat universal across domains.
Conclusion and Future Directions
While InterpCalib offers a powerful way to enhance black-box reliability, it faces a major hurdle: computational overhead. Perturbation-based explanation methods like LIME and SHAP require hundreds of model passes per query, making real-time deployment difficult. Future research should focus on distilling these attribution-based features into faster models or developing more efficient sampling techniques to generate explanations without sacrificing calibration quality.
Summary updated