Can In-context Learners Learn a Reasoning Concept from Demonstrations?

Michal Štefánik, Marek KadlčíkACL 2023

TL;DR

This paper introduces "Conceptual Few-shot Learning" to better assess if Large Language Models (LLMs) can truly learn reasoning concepts from demonstrations, rather than just relying on pre-trained knowledge. They found that while some models benefit from concept-specific demonstrations, no single model consistently learns all reasoning concepts, highlighting an open challenge in LLM development.

What Problem Does This Paper Address?

Large Language Models (LLMs) exhibit impressive "in-context learning" (ICL) abilities, meaning they can learn new tasks from just a few examples provided in the prompt. However, recent research suggests that this learning might be superficial. Instead of truly grasping the underlying reasoning, LLMs might be exploiting biases in the demonstrations or relying heavily on their pre-existing knowledge, like the sentiment of labels. This is partly due to how ICL is typically evaluated: using randomly selected demonstrations. Such random selections often don't highlight the specific reasoning patterns needed for a task, making it hard to distinguish genuine learning from memorization or pattern matching. This paper aims to disentangle a model's ability to learn a new reasoning concept from its ability to simply perform a task based on superficial cues.

Methods

The core contribution is the introduction of Conceptual Few-shot Learning (CFSL). Instead of random demonstrations, CFSL uses demonstrations that share a specific, informative reasoning concept relevant to the prediction task.

Here's how it works:

  1. Concept Extraction: The authors extract reasoning concepts from human-written explanations for existing datasets. These explanations often answer "why" a particular input-output pair is correct. They focus on semi-structured explanations to ensure consistency and clarity.
  2. Conceptual Demonstrations: For a given prediction task, they select demonstrations that exhibit the same reasoning concept as the one needed for the target input.
  3. Evaluation: They then compare the performance of LLMs using these concept-sharing demonstrations against their performance using randomly selected demonstrations. This allows them to measure the gain in performance specifically attributable to learning the shared reasoning concept.

Datasets

The paper evaluates several LLMs on a variety of datasets, from which they extract reasoning concepts:

  • WorldTree: A dataset of science exam questions with explanations in the form of shared facts. Concepts are extracted as these shared facts.
  • OpenBookQA: Elementary-grade science questions requiring common sense. Concepts are extracted from the syntactic analysis of explanations, focusing on the "relation" part of object-relation-object triples.
  • HotpotQA: A multi-document QA dataset with human-annotated explanations. Concepts are extracted as relation-entity pairs from these explanations.
  • GLUE Diagnostic: A Natural Language Inference dataset specifically designed to fool simple statistical models. This dataset directly annotates 30 distinct logical concepts (e.g., double negation, conjunction), which are used directly as the reasoning concepts.

Key Insights and Ideas

  • ICL is not always deep reasoning: The study strongly suggests that current ICL methods don't always lead to genuine understanding of reasoning concepts. Models can perform well on tasks without truly learning the underlying logic.
  • Concept-specific demonstrations matter: Providing demonstrations that share a relevant reasoning concept can significantly improve model performance, confirming the informativeness of these extracted concepts.
  • Model sensitivity varies: Different models react differently to concept-sharing demonstrations. Some models benefit greatly from certain concepts but fail with others, indicating a lack of consistent concept learning.
  • Smaller models might be more adaptable: Smaller models (≤3B parameters) seem to benefit more often from concept-sharing demonstrations compared to larger models. This could be because larger models might rely more on their vast memorized knowledge, making them less sensitive to specific in-context examples.
  • No universal concept learner: Crucially, no single model was found to consistently benefit from all the assessed reasoning concepts. This points to a significant gap in current LLM capabilities.

Key Results

  • For each assessed concept, at least one model showed a statistically significant improvement when using concept-sharing demonstrations compared to random ones.
  • The gain from concept-sharing demonstrations varied significantly across models and concepts. For instance, instruction-tuned TK-INSTRUCT models excelled with GLUE concepts but degraded with WorldTree concepts.
  • Figure 3 shows that on average, models like T0 and smaller FLAN models benefited more from concept-sharing demonstrations.
  • The paper presents detailed performance changes in Figures 2 and 3, showing relative performance changes for various models across different datasets and concepts. For example, Figure 2 shows that for some models, the performance change can be as high as +0.4 (e.g., T0-3B on GLUE Diagnostic) or as low as -0.4.

Limitations and Future Work

Limitations: * Concept Extraction: The concepts are extracted from semi-structured explanations, limiting the variety of concepts that can be evaluated. The selection of datasets and concepts is constrained by data availability. * Model Variance: Only a limited set of models were evaluated due to computational constraints and API access (e.g., only two GPT models were tested). Models over 20B parameters were generally excluded. * Dataset Size: The sizes of the sensitivity evaluation datasets are relatively small, which might limit the ability to compare concept sensitivity between models.

Future Work: * Broader Concept Taxonomy: Develop a more comprehensive taxonomy of reasoning concepts that better reflects the wide range of concepts models are expected to understand. This could guide more targeted concept collection and annotation. * More Diverse Concepts: Explore a wider variety of concepts, including those requiring complex reasoning steps, different granularities, or specific types of knowledge (e.g., entity properties vs. analogical reasoning). * Scalability: Investigate how to scale conceptual evaluation to a wider range of concepts and models to more realistically assess the abilities of emerging LLMs. * Understanding Model Reliance: Further investigate why larger models might be less sensitive to concept-sharing demonstrations, potentially due to their reliance on memorized knowledge.

Summary updated

BibTeX
@inproceedings{stefanik2023can,
  author = {Michal Štefánik and Marek Kadlčík},
  title = {{Can In-context Learners Learn a Reasoning Concept from Demonstrations?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}

Loading related content...

Figure