Abstract
In-Context Learning (ICL) is an important paradigm for adapting Large Language Models (LLMs) to downstream tasks through a few demonstrations. Despite the great success of ICL, the limitation of the demonstration number may lead to demonstration bias, i.e. the input-label mapping induced by LLMs misunderstands the task's essence. Inspired by human experience, we attempt to mitigate such bias through the perspective of the inter-demonstration relationship. Specifically, we construct Comparable Demonstrations (CDs) by minimally editing the texts to flip the corresponding labels, in order to highlight the task's essence and eliminate potential spurious correlations through the inter-demonstration comparison. Through a series of experiments on CDs, we find that (1) demonstration bias does exist in LLMs, and CDs can significantly reduce such bias; (2) CDs exhibit good performance in ICL, especially in out-of-distribution scenarios. In summary, this study explores the ICL mechanisms from a novel perspective, providing a deeper insight into the demonstration selection strategy for ICL.
Introduction
In-Context Learning (ICL) has emerged as a primary method for adapting Large Language Models (LLMs) to new tasks without the need for parameter updates. By providing a small set of demonstrations, models can infer the intended task. However, the limited sample size often leads to demonstration bias, where the model latches onto task-irrelevant features or spurious correlations. For instance, if a sentiment analysis prompt contains only reviews of romantic movies, the model might incorrectly conclude that the task is to identify movie genres rather than sentiment.
Methodology: Comparable Demonstrations (CDs)
To address this bias, we propose a novel selection strategy: Comparable Demonstrations (CDs). Inspired by contrastive learning in humans, CDs are pairs of examples where the text is minimally edited to flip the label. By presenting these "near-miss" pairs, the model is forced to observe exactly which semantic changes trigger a label flip, effectively highlighting the task's essence.
Instruction Induction
To measure demonstration bias, we employ Instruction Induction. We ask the LLM to generate a natural language description of the task based on the provided examples. If the model generates a biased or overly specific instruction (e.g., "Read the review of comedy" instead of "Determine sentiment"), it indicates a failure to grasp the true input-label mapping.
Experimental Results
We evaluated our approach across Sentiment Analysis (IMDb, Amazon, Yelp) and Natural Language Inference (SNLI, MNLI) tasks. We compared standard random selection and semantic similarity (nearest neighbor) against our CD-based strategies.
Reducing Bias through Comparison
Our manual evaluation of induced instructions reveals that CDs significantly outperform traditional methods in fostering task understanding. While random and nearest-neighbor selection often lead to invalid or imperfect instructions, CDs consistently produce "correct and satisfying" task descriptions, proving their efficacy in mitigating demonstration bias.
Performance and Robustness
CDs demonstrate superior performance, particularly in Out-of-Distribution (OOD) scenarios. On the Yelp sentiment dataset, the "CDs nearest" strategy—which selects comparable pairs semantically close to the test query—outperformed the standard nearest-neighbor approach by 7.6% in an 8-shot setting. Furthermore, CDs exhibit high robustness to the number of shots (K=4, 8, 12), maintaining stable performance where standard ICL often fluctuates.
Conclusion
This research highlights that the relationship between demonstrations is a critical factor in ICL success. By utilizing Comparable Demonstrations, we can effectively guide LLMs to ignore spurious correlations and focus on the core task. Future work will explore the automated generation of these pairs to reduce annotation costs and extend the framework to complex reasoning tasks.
Summary updated