Abstract
Explaining the predictions of AI models is paramount in safety-critical applications, such as in legal or medical domains. One form of explanation for a prediction is an extractive rationale, i.e., a subset of features of an instance that lead the model to give its prediction on the instance. Previous works on generating extractive rationales usually employ a two-phase model: a selector that selects the most important features (i.e., the rationale) followed by a predictor that makes the prediction based exclusively on the selected features. One disadvantage of these works is that the main signal for learning to select features comes from the comparison of the final answers given by the predictor and the ground-truth answers. In this work, we propose to squeeze more information from the predictor via an information calibration method. More precisely, we train two models jointly: one is a typical neural model that solves the task at hand in an accurate but black-box manner, and the other is a selector-predictor model that additionally produces a rationale for its prediction. The first model is used as a guide to the second model. We use an adversarial-based technique to calibrate the information extracted by the two models such that the difference between them is an indicator of the missed or over-selected features. In addition, for natural language tasks, we propose to use a language-model-based regularizer to encourage the extraction of fluent rationales. Experimental results on a sentiment analysis task as well as on three tasks from the legal domain show the effectiveness of our approach to rationale extraction.
Introduction
In safety-critical fields like law or medicine, "black-box" AI predictions are insufficient; understanding the reasoning behind a model's conclusion is essential. A common approach to transparency is the use of extractive rationales—selecting a specific subset of input features (such as key sentences in a legal case) that justify a prediction.
Traditional selector-predictor models often struggle with two primary issues: weak learning signals, where the selector only learns from the final output accuracy, and "word salad" explanations, where selected tokens are statistically relevant but semantically disjointed.
The InfoCal Framework
The authors introduce InfoCal, a framework designed to improve interpretability by training a "student" selector-predictor model to mimic the internal information of a high-performing "guider" model.
The architecture consists of three main components:
- The Guider: A high-accuracy, black-box model that processes the entire input and serves as the informational gold standard.
- The Selector-Predictor: The selector identifies the smallest possible subset of features (the rationale) that allows the predictor to match the Guider's performance.
- Adversarial Information Calibration: A discriminator compares the internal feature vectors of the Guider and the Predictor. If the discriminator can distinguish between them, it indicates that the selector has either missed crucial information or included excessive noise, forcing the model to calibrate its selections.
To ensure the rationales are human-readable, InfoCal employs a Language Model (LM) Regularizer. This rewards the selector for picking tokens that form fluent, consecutive sub-phrases, moving beyond simple mathematical constraints.
Experimental Results
InfoCal was evaluated on the BeerAdvocate sentiment analysis dataset and the CAIL2018 Chinese legal judgment dataset. The results demonstrate significant improvements in both rationale quality and predictive performance.
Key findings include:
- Improved Recall: On the BeerAdvocate "Smell" aspect, InfoCal achieved a 45.6% recall, a substantial increase over the previous state-of-the-art (HardKuma) at 31.5%.
- Legal Domain Efficiency: In legal tasks, InfoCal maintained higher prediction accuracy than baselines even when restricted to selecting only 14% of the text.
- Human Evaluation: In assessments of legal rationales, InfoCal consistently scored highest in Usefulness, Completeness, and Fluency, typically ranging between 4.2 and 4.7 out of 5.
Conclusion
InfoCal demonstrates that "squeezing" richer information from a teacher model's internal representations provides a more effective learning signal for rationale extraction. While the LM regularizer is highly effective for text, future research is needed to develop similar regularizers for other domains like image recognition and to validate these rationales with practicing professionals in real-world settings.
Summary updated