Abstract
Explaining the predictions of AI models is paramount in safety-critical applications, such as in legal or medical domains. One form of explanation for a prediction is an extractive rationale, i.e., a subset of features of an instance that lead the model to give its prediction on the instance. Previous works on generating extractive rationales usually employ a two-phase model: a selector that selects the most important features (i.e., the rationale) followed by a predictor that makes the prediction based exclusively on the selected features. One disadvantage of these works is that the main signal for learning to select features comes from the comparison of the answers given by the predictor and the ground-truth answers. In this work, we propose to squeeze more information from the predictor via an information calibration method. More precisely, we train two models jointly: one is a typical neural model that solves the task at hand in an accurate but black-box manner, and the other is a selector-predictor model that additionally produces a rationale for its prediction. The first model is used as a guide to the second model. We use an adversarial-based technique to calibrate the information extracted by the two models such that the difference between them is an indicator of the missed or over-selected features. In addition, for natural language tasks, we propose to use a language-model-based regularizer to encourage the extraction of fluent rationales. Experimental results on a sentiment analysis task as well as on three tasks from the legal domain show the effectiveness of our approach to rationale extraction.
Introduction
Deep learning models have achieved remarkable success across various domains, yet they often remain "black boxes," making their deployment in safety-critical fields like law or medicine problematic. To foster trust and accountability, researchers have turned to extractive rationales. This approach involves a model highlighting specific, influential snippets of the input text (e.g., "the defendant stole a phone") to justify its final prediction.
Standard extractive methods typically utilize a "selector-predictor" architecture: one module selects the most relevant words, and a second module makes a prediction based solely on that selection. However, these models face two significant hurdles. First, they often suffer from weak learning signals, as the selector primarily learns from the final prediction's accuracy, which is a noisy proxy for identifying the best features. Second, the resulting rationales often lack coherence, appearing as a disjointed "word salad" that is difficult for humans to interpret.
Methodology
The proposed InfoCal (Information Calibration) framework addresses these limitations through three primary innovations: a guider model, adversarial calibration, and a language model regularizer.
- The Guider Model: During training, InfoCal simultaneously trains a "guider"—a high-capacity, black-box model that has access to the entire input text. This guider serves as a benchmark for the information that should ideally be captured.
- Adversarial Information Calibration: This is the core mechanism of the framework. It employs a discriminator (akin to those in Generative Adversarial Networks) to align the internal feature representation of the selected rationale with the full-text representation from the guider. This process "squeezes" the selector to identify the most informative tokens possible.
- Language Model (LM) Regularizer: To combat the "word salad" problem, a pre-trained language model acts as a coach. Instead of merely penalizing discontinuities, the LM regularizer encourages the selector to choose sequences that form natural, fluent sub-phrases, respecting linguistic boundaries.
- Information Bottleneck: The framework incorporates an information bottleneck to ensure rationales are "least-but-enough," preventing the model from over-selecting and simply highlighting the entire document.
Key Insights
The central philosophy of InfoCal is "Learning from the Best." By allowing an interpretable model to "peek" at the internal logic of a superior black-box counterpart, the selector can converge on high-quality rationales more efficiently. Furthermore, InfoCal distinguishes itself from standard model distillation; while distillation typically matches final output probabilities, InfoCal calibrates the information content of internal latent vectors.
Additionally, InfoCal treats fluency as a first-class feature. By recognizing that language has inherent structure where certain word pairs are semantically "stickier" than others, the model produces explanations that are not only accurate but also human-readable.
Experimental Results
The researchers evaluated InfoCal on two distinct benchmarks:
- BeerAdvocate: A multi-aspect sentiment analysis task where models predict ratings for beer appearance, smell, and palate. This dataset includes human-annotated "gold" rationales for objective evaluation.
- CAIL2018: A large-scale dataset of Chinese legal documents for predicting law articles, charges, and penalties. Due to the lack of gold rationales, the authors conducted a human study with law students to assess quality.
Quantitative Performance
On the BeerAdvocate dataset, InfoCal significantly outperformed state-of-the-art methods like Bernoulli and HardKuma in both rationale precision and recall. For instance, in the "Smell" aspect, the F1-score for rationale extraction improved from 54.5 to 61.7. In the legal domain, InfoCal achieved top-tier performance, often outperforming models that used the entire text while selecting only 14-20% of the tokens.
Qualitative Analysis
The impact of the LM regularizer is most evident in qualitative comparisons. While baseline methods often produce fragmented selections, InfoCal identifies coherent phrases that provide clear logical links to the prediction.
Human evaluations further validated these results. In the legal tasks, InfoCal scored significantly higher across three key metrics: Usefulness (explanatory power), Completeness, and Fluency.
Conclusion
InfoCal represents a significant advancement in making AI explainability a high-performance feature rather than a compromise. By leveraging adversarial calibration and linguistic priors, it produces rationales that are both informative and interpretable. While currently focused on natural language, the principles of information calibration could potentially be extended to other domains, such as computer vision or tabular data analysis, in future research.
Summary updated