Few-Shot Self-Rationalization with Natural Language Prompts

Ana Marasovic, Iz Beltagy, Doug Downey, Matthew E. Peters

Abstract

This study presents a systematic investigation into few-shot self-rationalization, where models generate both a task label and a free-text explanation from minimal examples. We introduce the Few Explanations Benchmark (FEB) and evaluate various prompting strategies across model scales. Our findings reveal that while scaling improves explanation quality, a significant gap remains between AI-generated rationales and human reasoning.

Introduction

Standard NLP models typically function as "black boxes," providing answers without justification. Self-rationalization addresses this by requiring models to jointly predict labels and generate natural language explanations. However, traditional training relies on thousands of human-authored explanations. This work explores the few-shot setting (16–48 examples), investigating how prompt engineering and model scaling can bridge the data gap.

The Few Explanations Benchmark (FEB)

To standardize evaluation, we introduce FEB, comprising four diverse tasks adapted for few-shot learning. Each task is restricted to a small number of training examples per label to simulate real-world scarcity.

Table 1: Overview of the FEB tasks, including task descriptions and the number of shots used for each.

The benchmark includes:

  • E-SNLI: Natural Language Inference (entailment, neutral, contradiction).
  • ECQA: Commonsense question answering with five choices.
  • CoMVE: Identifying nonsensical sentences.
  • SBIC: Detecting offensive social media content.

Methodology: Prompting Strategies

We evaluated three primary prompting strategies to "prime" models for self-rationalization:

  1. QA Prompts ($QA_{SIMPLE}$): Formatting tasks as direct questions (e.g., "What is the relation...?").
  2. Infilling Prompts: Leveraging T5’s span-filling objective.
  3. Task-Specific Prompts ($\approx$T5): Mimicking T5’s supervised pretraining formats.
Table 2: Performance comparison of different prompting strategies across FEB tasks. QA_Simple generally yields the best average performance.

Key Findings

1. Prompt Sensitivity and Structure

The choice of prompt significantly impacts performance. Versatile QA-style prompts ($QA_{SIMPLE}$) outperformed span-infilling. Furthermore, adding explicit structural tags (e.g., premise:, hypothesis:) proved essential for multi-input tasks like E-SNLI; without them, models often failed to parse the few-shot request correctly.

2. The Impact of Scaling

Explanation plausibility, as rated by humans, increases monotonically with model size. Upgrading from a 200M parameter "Base" model to a 3B parameter model resulted in a 14.8-point average increase in plausibility scores. While GPT-3 (175B) produced the most "natural" sounding explanations, UnifiedQA-3B remained highly competitive in task accuracy.

3. The Human-AI Plausibility Gap

Despite improvements from scaling, a "grand canyon" exists between AI and human reasoning. Human explanations achieved a 76% plausibility score, whereas the best model (GPT-3) reached only 51%. Models also showed high sensitivity to labels; for instance, they easily explained why a post was not offensive using templates but struggled with the nuances of actual offensive content.

Conclusion

Few-shot self-rationalization remains a challenging frontier. While prompt engineering and model scaling provide significant gains, current models often provide correct answers for the wrong reasons. FEB provides a rigorous framework for developing more sample-efficient, explainable AI. Future work must address the weak correlation between automated metrics like BERTscore and true human plausibility.

Summary updated

BibTeX
@inproceedings{marasovic2022few,
  author = {Ana Marasovic and Iz Beltagy and Doug Downey and Matthew E. Peters},
  title = {{Few-Shot Self-Rationalization with Natural Language Prompts}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.findings-naacl.31},
}
Figure