Abstract
Modern NLP models often struggle with reliability when deployed on out-of-domain data, frequently exhibiting high confidence in incorrect predictions. This paper investigates whether model-agnostic explanations, such as LIME and SHAP, can be leveraged to build a secondary "calibrator." By transforming raw word-level attributions into human-interpretable features—such as the model's focus on proper nouns or specific segments—the authors demonstrate significant improvements in identifying model failures in black-box settings without requiring weight updates.
Introduction
In real-world applications, NLP models are frequently treated as "black boxes" due to proprietary restrictions or computational constraints. A critical issue arises when these models encounter out-of-domain (OOD) data, where they tend to be "confident but wrong." Standard calibration techniques, which rely on the model's internal probability scores (e.g., MaxProb), often fail to generalize to these new distributions. This work proposes a method to calibrate black-box models by analyzing why they make specific predictions rather than just what they predict.
Methodology
The proposed pipeline consists of three main stages:
- Explanation Generation: Using model-agnostic techniques like LIME or SHAP, the system identifies which input tokens most influence a specific prediction by perturbing the input.
- Feature Extraction: Raw attributions are aggregated into intuitive categories. These include segments (Question vs. Context), Part-of-Speech (POS) tags (e.g., proper nouns, verbs), and word overlap in NLI tasks.
- The Calibrator: A Random Forest classifier is trained on a small target-domain dataset (approx. 500 examples). It uses the aggregated attribution features to predict the likelihood that the base model's answer is correct.
Key Insights: Heuristics and Attribution
The core intuition is that human-like heuristics can guide calibration. For instance, in Question Answering (QA), a model should ideally attend to proper nouns (NNP) in the question to identify the subject. As illustrated in the methodology, a lower attribution to these key entities often correlates with incorrect predictions. By learning these patterns, the calibrator acts as a "BS detector."
Notably, this approach is highly data-efficient, requiring only a few hundred examples to achieve robust calibration. In some OOD scenarios, this black-box calibration even outperforms "glass-box" fine-tuning, which can be unstable on small datasets and may degrade the base model's general knowledge.
Experimental Results
The method was evaluated on extractive QA (transferring from SQuAD to HotpotQA/TriviaQA) and NLI (transferring from MNLI to QNLI/MRPC).
On adversarial datasets like SQuAD-ADV, the explanation-based calibrator (LimeCal) significantly outperformed the standard MaxProb baseline. As seen in the performance curves, when the model is allowed to be selective (answering only high-confidence questions), the F1 score remains substantially higher than baselines across various coverage levels. While improvements in NLI were more modest due to the inherent strength of simple word-overlap heuristics in those datasets, the QA results highlight the power of attribution-based features for complex reasoning tasks.
Limitations and Future Work
Despite its success, the approach has limitations. Generating LIME/SHAP explanations is computationally expensive, requiring thousands of model passes per input, which may hinder real-time deployment. Additionally, the current system relies on manually defined POS-tag heuristics. Future research could explore automated rationale discovery and methods to reduce the inference-time overhead of generating explanations.
Summary updated