A smashed glass cannot be full: Generation of Commonsense Explanations through Prompt-based Few-shot Learning

Andrea Zaninello, Bernardo MagniniACL 2023

TL;DR

This paper introduces a novel methodology for generating commonsense explanations from pairs of sentences, leveraging large language models (LLMs) with prompt-based few-shot learning. Their approach achieves state-of-the-art results on the e-SNLI dataset while significantly reducing the need for labeled data, opening new avenues for eliciting implicit knowledge.

What Problem Does This Paper Address?

In everyday communication, we often omit details that are considered self-evident. This is particularly true for commonsense knowledge, which underpins our reasoning. When we ask for an explanation, we're essentially asking to make this implicit, assumed knowledge explicit. Computational systems struggle with this because this knowledge isn't directly present in the text. The challenge is to automatically generate these commonsense explanations, which requires eliciting implicit knowledge that goes beyond the literal text. Traditional methods would require vast amounts of labeled data (sentence pairs with corresponding explanations), which is often impractical for open-domain scenarios.

Methods

The authors propose a general methodology to elicit implicit knowledge from pre-trained LLMs using a combination of prompting and few-shot learning.

  • Prompting: This involves reframing an NLP task as a language modeling task. An input prompt, written in natural language, describes the desired task and can include examples. The LLM then generates the output as if it were predicting the next word.
  • Few-Shot Learning: Instead of extensive fine-tuning, this approach uses a small number of examples to guide the model. The paper explores three configurations:
  • Unsupervised: The model is evaluated without any parameter updates, using prompts to guide generation at inference time.
  • Fine-Tuning: The model's parameters are updated using a small set of training examples, augmented with synthetically generated data to prevent overfitting.
  • Ensembling: This method combines the strengths of multiple prompts by using an untrained model to score candidate explanations generated by different prompts, then fine-tuning a final model on the best-ranked explanations. This approach aims to mitigate the negative impact of poorly designed prompts.

The core idea is to leverage the vast implicit knowledge already present in LLMs and guide them to express it through carefully crafted prompts and minimal training data.

Datasets

The primary dataset used in this work is the e-SNLI dataset. This dataset extends the Stanford Natural Language Inference (SNLI) dataset by adding crowd-sourced natural language explanations for each premise-hypothesis pair. The e-SNLI dataset is crucial because it provides the necessary "ground truth" explanations for evaluating the model's generation capabilities.

Key Insights and Ideas

  • Implicit knowledge is key: The paper posits that commonsense explanations are largely about making implicit knowledge explicit.
  • LLMs as knowledge repositories: The authors hypothesize that LLMs, due to their extensive pre-training, already contain most of the necessary commonsense knowledge.
  • Prompting + Few-Shot = Efficiency: Combining prompting with few-shot learning offers a way to elicit this knowledge with significantly less labeled data than traditional supervised methods.
  • Ensembling for robustness: The ensembling method proves effective in mitigating the negative impact of "bad" or suboptimal prompts, making the approach more robust.
  • T5 excels: The T5 model, with its text-to-text pre-training format, appears particularly well-suited for this task.

Key Results

  • The proposed methods, particularly the T5-ENSEMBLING configuration, achieve state-of-the-art results on the e-SNLI dataset for explanation generation.
  • A substantial reduction in labeled data is achieved compared to previous methods. For instance, their T5-Ensembling method uses only 10 training examples to achieve a BERT-score of 91.20, significantly outperforming prior work that used thousands or even tens of thousands of examples.
  • The ensembling method is particularly effective in mitigating the negative effects of shallow prompt design across all tested models (BART, Pegasus, and T5).
  • Interestingly, models show very low scores in zero-shot configurations, highlighting the importance of even minimal fine-tuning or prompt-based learning for this task.

Limitations and Future Work

The authors acknowledge several limitations:

  • Dataset Specificity: The experiments were conducted solely on the e-SNLI dataset. Future work should explore other datasets with natural language explanations.
  • Prompt Optimization: The paper did not focus on automatic prompt optimization, which could potentially yield further improvements. This is suggested as a future research direction.
  • Evaluation Metric Consensus: There is still no clear consensus on the best metric for evaluating explanation generation, making direct comparisons with other state-of-the-art results challenging. The authors hope for a consolidated benchmark in the future.
  • Assumption of Implicit Explanations: The study assumes that all sentence pairs in the dataset have an implicit explanation. Future work could investigate methods to detect when an explanation is already explicit in the text.

Summary updated

BibTeX
@inproceedings{zaninello2023smashed,
  author = {Andrea Zaninello and Bernardo Magnini},
  title = {{A smashed glass cannot be full: Generation of Commonsense Explanations through Prompt-based Few-shot Learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure