Abstract
In-context learning (ICL) enables large language models (LLMs) to perform new tasks by using sample-label pairs as demonstrations. However, variations in demonstrations can lead to significantly different performances. Current research mainly focuses on selecting demonstration samples, preassuming the class name to be the label word when creating sample-label pairs. However, the choice of label words is crucial for ICL performance. Besides, we observe that using a single class name in demonstration may not yield optimal results while using multiple label words in one sample-label pair can enhance ICL performance. In this paper, we propose a comprehensive approach that organizes both samples and labels in demonstrations based on LLMs' output space distribution. This approach uses multiple label words in one sample-label pair to enhance label instruction. Evaluation results from seven classification datasets show that this demonstration organization method, which incorporates multiple label words to provide diverse label information, improves ICL performance.
Introduction
Large Language Models (LLMs) have demonstrated a remarkable ability to perform new tasks via In-Context Learning (ICL) by observing a few examples in a prompt. However, ICL is notoriously brittle; the specific choice of examples and label words can dramatically impact performance. While most research focuses on sample selection, the choice of "verbalizers" (label words) is often overlooked, with researchers typically defaulting to human-intuitive class names like "positive" or "negative."
This paper addresses this gap by introducing a unified framework that selects both samples and labels based on the model's internal "understanding." We find that human-preferred labels are not always model-optimal and that providing multiple descriptive label words can significantly reduce ambiguity and improve accuracy.
Methodology: Logit Separability-based ICL (LICL)
The core of our approach is a new criterion called Logit Separability. This metric measures how clearly a model’s internal raw prediction scores (logits) distinguish the correct class from others for a given sample or label word.
The LICL framework consists of three main steps:
- Pool Refinement: We begin with a large pool of class-related words and filter them using point-biserial correlation to identify words that statistically distinguish between classes in the model's output space.
- Sample Selection & Ordering: Training samples are scored based on their logit separability. Samples that yield a high logit for the correct class and low logits for others are prioritized and interleaved in the prompt.
- Multiple Class-Related Words Insertion: Instead of a single label, we use a sequence of words (e.g., "Sentiment: negative unhealthy unjust"). We employ a Sequential Forward Search to iteratively add words to the label based on which combination maximizes validation performance.
Key Insights and Results
Our experiments reveal that "model-centric" labels often outperform human-centric ones. For example, the word "bad" frequently provides a clearer signal to the model than the formal class name "negative." By using multiple descriptive words, we provide a broader range of label information, helping the model align the demonstration with its internal representations.
We evaluated LICL on seven diverse classification datasets, including sentiment analysis (SST-2, CR), emotion classification (ISEAR, AMAN), and question classification (TREC-6). Key results include:
- Significant Accuracy Gains: In 1-shot ICL, LICL improved average accuracy by 3.11% on LLaMA2-7b and 11.58% on GPT2-xl.
- Multi-class Performance: The gains were most dramatic in complex multi-class tasks, such as AGNews, where accuracy jumped by 23.68% using GPT2-xl.
- Efficiency: LICL packs more information into a single sample-label pair, making it more computationally efficient during inference compared to methods requiring numerous augmented samples.
Conclusion
By organizing demonstrations based on the model's output space distribution, LICL significantly enhances the robustness and performance of in-context learning. The use of multiple label words provides a richer instructional signal that better aligns with the model's internal logic. Future work will explore the application of logit separability to generative tasks such as summarization and translation.
Summary updated