Deep learning models are often criticized as "black boxes" that provide answers without justification. To build trust, especially in high-stakes domains like education, fact-checking, and security, we need models that provide rationales—natural language explanations that bridge the gap between input data and the final output.
The fundamental challenge is the "rationale bottleneck." While standard labels (like "True/False" or "Score: 5") are relatively easy to collect, high-quality human explanations are prohibitively expensive and time-consuming to produce. Researchers are now overcoming this by developing methods to generate, distill, and refine explanations without requiring massive human-annotated datasets.
The Rationale Bottleneck
In traditional supervised learning, a model learns a direct mapping from input to output. To make a model explainable, we typically require a third component: the "gold-standard" rationale. As Pepa Atanasova and Isabelle Augenstein note in Diagnostics-Guided Explanation Generation, this data scarcity limits our ability to create interpretable systems.
Whether it is explaining why a student's answer is incorrect Short Answer Feedback (SAF) or why a software bug is a security vulnerability LLMVulExp, the cost of expert annotation remains the primary barrier to transparency.
Distilling Explanations from Large Models
One powerful solution is to use Large Language Models (LLMs) as "teacher" models that generate rationales for smaller, more efficient "student" models.
In the AERA framework, Jiazheng Li and colleagues Distilling ChatGPT for Explainable Automated Student Answer Assessment demonstrate that ChatGPT can generate and refine grading rationales. These are then distilled into a compact Long T5 model. Surprisingly, this student model can outperform the teacher's grading accuracy while providing the necessary justifications for real-world deployment. Similarly, Dominik Stammbach and Elliott Ash used GPT-3 to generate 67,000 explanations for fact-checking in the e-FEVER dataset, proving that machine-generated explanations can bootstrap explainability where human data is missing.
Self-Alignment and Latent Reasoning
If we don't have a teacher model, can a model learn to explain itself? Several researchers have proposed treating explanations as intermediate, "latent" steps in the decision-making process.
- Explanations as Latent Variables: Wangchunshu Zhou introduced the ELV framework Towards Interpretable Natural Language Understanding with Explanations as Latent Variables, which uses a variational framework to jointly learn reasoning and prediction. This allows models to generate explanations using only a tiny fraction of human-annotated data.
- Self-Training: Fangkai Jiao and others A Self-Training Method for Machine Reading Comprehension with Soft Evidence Extraction developed methods where models iteratively generate "pseudo-evidence" labels to supervise their own internal evidence extractors.
- Honesty through Explanations: Yang Deng and Tat-Seng Chua applied this to the problem of hallucinations. In Don't Just Say "I don't know"!, they used a self-augmentation loop to train models to explain why a question is unanswerable, rather than just failing silently.
Human-in-the-Loop Alignment
Rather than asking humans to write explanations from scratch, a more efficient approach is to have humans filter or rank machine-generated ones.
The ILLUME framework by Manuel Brack Rationalizing Vision-Language Models through Human Interactions allows Vision-Language Models to "jabber" various candidate rationales. Humans then select the ones that align with common sense. This iterative process allows models to transfer reasoning capabilities to new tasks with significantly less manual effort than traditional fine-tuning.
Why Explanations Improve Performance
A recurring theme across these papers is that teaching a model to explain its work doesn't just make it more transparent—it often makes it more accurate and robust.
In the Explain-Then-Predict framework Enhancing adversarial robustness in Natural Language Inference using explanations, Alexandros Koulakos and colleagues showed that forcing a model to generate a human-readable rationale before making a judgment makes it much harder to fool with adversarial attacks. By grounding the final decision in a linguistic explanation, the model is less likely to rely on "spurious correlations" or statistical noise in the training data.
As Dennis Wei argued in the TED framework Teaching Meaningful Explanations, when we provide explanations during training, we aren't just adding a feature; we are providing a richer signal that helps the model learn the underlying logic of the task.