Logit Separability-Driven Samples and Multiple Class-Related Words Selection for Advancing In-Context Learning

Zhu Zixiao, Zijian Feng, Zhou Hanzhang, Qian Junlang, Kezhi Mao

Abstract

Effective organization of in-context learning (ICL) demonstrations is key to improving the quality of large language model (LLM) responses. To create better sample-label pairs that instruct LLM understanding, we introduce logit separability, a criterion to assess the clarity of both samples and class-related words at the logit level. This facilitates the optimization of sample and label selection, enhancing the precision of information provided in ICL demonstrations. Additionally, we find that incorporating multiple class-related words for each sample, rather than relying on a single class name, improves performance by offering a broader range of label information. Building on these insights, we propose LICL, a logit separability-based method that jointly organizes samples and integrates multiple class-related words into each sample-label pair. Evaluations across seven classification datasets show that this approach significantly improves ICL performance by providing clearer instructions and richer label information.

Introduction

In-Context Learning (ICL) has emerged as a powerful paradigm for Large Language Models (LLMs), enabling them to perform new tasks using only a few demonstrations. However, ICL performance is notoriously sensitive to the choice of demonstration samples, their order, and the specific words used as labels. Small variations in these factors can lead to significant fluctuations in accuracy, a phenomenon often described as "brittleness."

While previous research has addressed sample selection or label optimization independently, few methods provide a unified approach. Moreover, many existing techniques rely on external metrics (such as S-BERT similarity) that may not align with the target LLM's internal representations. This paper introduces LICL (Logit separability-based In-Context Learning), a framework that leverages the model's own internal "confidence"—represented by logits—to optimize both sample selection and label descriptions.

The LICL Framework

The core of the LICL approach is the concept of Logit Separability. A sample or label word is considered to have high logit separability if the LLM assigns a significantly higher logit value to the correct class-related words compared to incorrect ones. This internal signal serves as a more reliable guide for selecting demonstrations than external similarity metrics.

The LICL framework operates in three distinct stages:

  1. Pool Refinement: A broad set of class-related words is filtered using point-biserial testing to retain only those that the LLM consistently associates with the correct class across the training set.
  2. Sample Selection and Ordering: Training samples are ranked based on their logit separability. Samples that produce the clearest signals (where the correct label is ranked highest by a wide margin) are prioritized as demonstrations.
  3. Multiple Class-Related Words Insertion: Instead of a single class name (e.g., "Sentiment: negative"), LICL constructs a sequence of descriptive words (e.g., "Sentiment: negative unhealthy unjust"). A Sequential Forward Search is employed to find the optimal combination of words that maximizes validation performance.
Overview of the LICL framework

Experimental Evaluation

The authors evaluated LICL across seven diverse classification datasets: SST-2, CR, and IMDB (Sentiment Analysis); ISEAR and AMAN (Emotion Classification); TREC-6 (Question Classification); and AGNews (Topic Classification). These datasets encompass both binary and multi-class tasks, providing a robust testing ground for the method's generalizability.

Key Results and Insights

LICL demonstrated substantial performance gains over strong baselines like Vanilla ICL and TopK selection. On LLaMA2-7b, the average accuracy improved by 3.11%, while the smaller GPT2-xl model saw a significant boost of 11.58%. In some complex tasks like AGNews, accuracy soared by over 23%.

A key finding is the "Label Bottleneck" effect: relying on a single class name often limits the information transfer to the model. By using multiple related words, LICL provides a richer semantic signal that aligns better with the model's internal state, reducing ambiguity.

Performance comparison across datasets

Beyond accuracy, LICL also improves the stability of ICL. The use of multiple label words reduces the standard deviation of results across different random seeds, making the model's performance more predictable and robust. Interestingly, the researchers found that descriptive words like "bad" often have higher logit separability than official class names like "negative," suggesting that LLMs benefit from more naturalistic descriptors.

Conclusion

LICL offers a model-aware solution to the brittleness of in-context learning. By using logit separability as a compass, it identifies the most effective demonstrations and labels. While the method introduces some computational overhead during the initial organization phase and slightly increases prompt length, the resulting gains in accuracy and stability across various model sizes and tasks make it a compelling approach for optimizing LLM performance.

Summary updated

Abstract

Effective organization of in-context learning (ICL) demonstrations is key to improving the quality of large language model (LLM) responses. To create better sample-label pairs that instruct LLM understanding, we introduce logit separability, a criterion to assess the clarity of both samples and class-related words at the logit level. This facilitates the optimization of sample and label selection, enhancing the precision of information provided in ICL demonstrations. Additionally, we find that incorporating multiple class-related words for each sample, rather than relying on a single class name, improves performance by offering a broader range of label information. Building on these insights, we propose LICL, a logit separability-based method that jointly organizes samples and integrates multiple class-related words into each sample-label pair. Evaluations across seven classification datasets show that this approach significantly improves ICL performance by providing clearer instructions and richer label information.
BibTeX
@inproceedings{zixiao2025logit,
  author = {Zhu Zixiao and Zijian Feng and Zhou Hanzhang and Qian Junlang and Kezhi Mao},
  title = {{Logit Separability-Driven Samples and Multiple Class-Related Words Selection for Advancing In-Context Learning}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.343},
}
Figure