Can Jev Improve Demonstration Retrieval for In-Context Learning?

Rui Song, Yingji Li, Hao XuarXiv 2026 · arXiv

TL;DR

This paper introduces DisJEV, a framework that leverages decision-oriented language models (like Jev) to select high-utility demonstrations for In-Context Learning (ICL). By distilling Jev’s probabilistic judgment capabilities into a lightweight MLP, the authors achieve significant accuracy gains across diverse tasks and models while maintaining high computational efficiency.

What Problem Does This Paper Address?

In-Context Learning (ICL) allows Large Language Models (LLMs) to perform new tasks simply by providing a few examples (demonstrations) in the prompt. However, the effectiveness of ICL is highly sensitive to which examples are chosen.

Standard retrieval methods usually rely on semantic similarity (e.g., kNN), assuming that examples most similar to the query are the most helpful. The authors argue this is a flawed heuristic: a semantically similar example isn't necessarily the most "useful" for helping a model make a correct prediction. While specialized decision models like Jev can directly estimate the utility of an example, they are too computationally expensive to use for real-time retrieval at scale.

Methods

The authors propose a two-stage retrieval and distillation pipeline:

  1. Jev-Guided Reranking: They first use a standard encoder to retrieve a candidate pool ($M=20$). Then, they use Jev as a "teacher" to score these candidates based on their likelihood of helping the LLM classify the query correctly.
  2. Multi-objective Distillation: To make this efficient, they distill Jev’s knowledge into a lightweight Multilayer Perceptron (MLP) student. The distillation uses two complementary losses:
  3. Score Distillation: The student learns to match Jev’s absolute utility scores.
  4. Ranking Distillation: A pairwise loss (based on RankNet) that teaches the student the relative preference between different candidates.
  5. Efficient Selection via Beam Search: During inference, instead of just picking the top $K$ individual examples, the distilled student uses a beam search strategy to find the best combination of demonstrations, accounting for how examples interact with each other in a prompt.
Figure 4: Overview of the proposed framework.

Datasets

The framework was evaluated on five widely used text classification benchmarks:

  • TREC: Question classification.
  • SST-2 & SST-5: Binary and fine-grained sentiment analysis.
  • AG News: Topic classification.
  • MNLI: Natural language inference.

Training for the distilled student was performed on a subset (15%) of the training data, while evaluation was conducted on up to 2,000 test samples per dataset.

Key Insights and Ideas

  • Similarity $\neq$ Utility: The authors found a notable discrepancy between semantic similarity and actual utility. As shown in Figure 2, Jev-guided reranking replaces a substantial proportion (often over 80% for 1-shot) of the examples initially chosen by kNN.
  • Decision Quality Over Model Size: Interestingly, the smaller NeoJEV-4B model often outperformed the larger JEV-9B in identifying informative demonstrations. This suggests that the specific discriminative tuning of a decision model is more important than raw parameter count for this task.
  • Distillation is Highly Effective: The lightweight MLP student achieved approximately 70% pairwise agreement with its teacher (Jev), proving that complex utility judgments can be compressed into a very simple model for efficient inference.
  • Cross-Task Generalization: A student trained on four datasets could successfully generalize to a fifth "unseen" dataset, outperforming kNN baselines even without task-specific tuning.
Figure 2: Distribution of demonstration replacements.

Key Results

  • Accuracy Gains: DisJEV consistently outperformed kNN across various backbone LLMs. For example, on Qwen3-1.7B, it improved average accuracy by 3.68 percentage points.
  • Efficiency: Compared to other training-based retrievers like DiSP, DisJEV is significantly faster. It reduced training time by roughly 3.6x (137.2s vs 497.7s per 100 samples) and maintained lower inference latency.
  • Robustness: The method showed consistent improvements even when scaling up to larger backbone models like Llama-3.1-8B and DeepSeek-V4-Flash.
Figure 1: Average results over five datasets with the increasing of K. More detailed results are in App. B.
Figure 5: Time cost comparison between DisJEV and DiSP during training and inference.

Limitations and Future Work

  • Fixed Candidate Pool: The current two-stage approach relies on an initial kNN retrieval to create a pool of 20 candidates. If the "perfect" demonstration isn't in that top 20, the reranker can't find it.
  • Task Specificity: While the model generalizes well, the authors noted that performance gains vary by task (e.g., smaller gains on MNLI compared to TREC).
  • Future Directions: The authors suggest exploring more sophisticated ways to represent demonstration combinations and investigating how this approach scales to non-classification tasks like code generation or long-form reasoning.

Summary updated

Abstract

Demonstration selection plays a crucial role in the effectiveness of in-context learning (ICL) for large language models (LLMs). Extensive research has explored various retrieval strategies to identify informative demonstrations and improve downstream performance. Recently, the emergence of Jev, a decision-oriented language model capable of producing probabilistic judgments, offers a new perspective on evaluating demonstration utility. This motivates us to investigate a fundamental question: Can Jev improve demonstration retrieval for in-context learning? In this paper, we demonstrate that Jev can effectively enhance demonstration retrieval by directly assessing the utility of candidate demonstrations for a given query. However, applying Jev to large-scale retrieval introduces substantial computational overhead, as each candidate requires an additional decision-making inference. To address this limitation, we propose a simple yet effective knowledge distillation approach that transfers Jev's utility estimation capability to a lightweight multilayer perceptron (MLP). By learning to approximate Jev's probabilistic judgments, the distilled model enables efficient demonstration ranking while maintaining competitive retrieval performance. Our study highlights the potential of decision-oriented language models for demonstration selection and provides an efficient framework for integrating their decision-making capabilities into ICL.
BibTeX
@misc{song2026can,
  author = {Rui Song and Yingji Li and Hao Xu},
  title = {{Can Jev Improve Demonstration Retrieval for In-Context Learning?}},
  year = {2026},
  eprint = {2611.00737},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.00737},
}
Figure