Abstract
This study presents a systematic investigation into few-shot self-rationalization, where models generate both a task label and a free-text explanation from minimal examples. We introduce the Few Explanations Benchmark (FEB) and evaluate various prompting strategies across model scales. Our findings reveal that while scaling improves explanation quality, a significant gap remains between AI-generated rationales and human reasoning.
Introduction
Standard NLP models typically function as "black boxes," providing answers without justification. Self-rationalization addresses this by requiring models to jointly predict labels and generate natural language explanations. However, traditional training relies on thousands of human-authored explanations. This work explores the few-shot setting (16–48 examples), investigating how prompt engineering and model scaling can bridge the data gap.
The Few Explanations Benchmark (FEB)
To standardize evaluation, we introduce FEB, comprising four diverse tasks adapted for few-shot learning. Each task is restricted to a small number of training examples per label to simulate real-world scarcity.
The benchmark includes:
- E-SNLI: Natural Language Inference (entailment, neutral, contradiction).
- ECQA: Commonsense question answering with five choices.
- CoMVE: Identifying nonsensical sentences.
- SBIC: Detecting offensive social media content.
Methodology: Prompting Strategies
We evaluated three primary prompting strategies to "prime" models for self-rationalization:
- QA Prompts ($QA_{SIMPLE}$): Formatting tasks as direct questions (e.g., "What is the relation...?").
- Infilling Prompts: Leveraging T5’s span-filling objective.
- Task-Specific Prompts ($\approx$T5): Mimicking T5’s supervised pretraining formats.
Key Findings
1. Prompt Sensitivity and Structure
The choice of prompt significantly impacts performance. Versatile QA-style prompts ($QA_{SIMPLE}$) outperformed span-infilling. Furthermore, adding explicit structural tags (e.g., premise:, hypothesis:) proved essential for multi-input tasks like E-SNLI; without them, models often failed to parse the few-shot request correctly.
2. The Impact of Scaling
Explanation plausibility, as rated by humans, increases monotonically with model size. Upgrading from a 200M parameter "Base" model to a 3B parameter model resulted in a 14.8-point average increase in plausibility scores. While GPT-3 (175B) produced the most "natural" sounding explanations, UnifiedQA-3B remained highly competitive in task accuracy.
3. The Human-AI Plausibility Gap
Despite improvements from scaling, a "grand canyon" exists between AI and human reasoning. Human explanations achieved a 76% plausibility score, whereas the best model (GPT-3) reached only 51%. Models also showed high sensitivity to labels; for instance, they easily explained why a post was not offensive using templates but struggled with the nuances of actual offensive content.
Conclusion
Few-shot self-rationalization remains a challenging frontier. While prompt engineering and model scaling provide significant gains, current models often provide correct answers for the wrong reasons. FEB provides a rigorous framework for developing more sample-efficient, explainable AI. Future work must address the weak correlation between automated metrics like BERTscore and true human plausibility.
Summary updated