TL;DR
This paper introduces DisJEV, a framework that leverages decision-oriented language models (like Jev) to select high-utility demonstrations for In-Context Learning (ICL). By distilling Jev’s probabilistic judgment capabilities into a lightweight MLP, the authors achieve significant accuracy gains across diverse tasks and models while maintaining high computational efficiency.
What Problem Does This Paper Address?
In-Context Learning (ICL) allows Large Language Models (LLMs) to perform new tasks simply by providing a few examples (demonstrations) in the prompt. However, the effectiveness of ICL is highly sensitive to which examples are chosen.
Standard retrieval methods usually rely on semantic similarity (e.g., kNN), assuming that examples most similar to the query are the most helpful. The authors argue this is a flawed heuristic: a semantically similar example isn't necessarily the most "useful" for helping a model make a correct prediction. While specialized decision models like Jev can directly estimate the utility of an example, they are too computationally expensive to use for real-time retrieval at scale.
Methods
The authors propose a two-stage retrieval and distillation pipeline:
- Jev-Guided Reranking: They first use a standard encoder to retrieve a candidate pool ($M=20$). Then, they use Jev as a "teacher" to score these candidates based on their likelihood of helping the LLM classify the query correctly.
- Multi-objective Distillation: To make this efficient, they distill Jev’s knowledge into a lightweight Multilayer Perceptron (MLP) student. The distillation uses two complementary losses:
- Score Distillation: The student learns to match Jev’s absolute utility scores.
- Ranking Distillation: A pairwise loss (based on RankNet) that teaches the student the relative preference between different candidates.
- Efficient Selection via Beam Search: During inference, instead of just picking the top $K$ individual examples, the distilled student uses a beam search strategy to find the best combination of demonstrations, accounting for how examples interact with each other in a prompt.
Datasets
The framework was evaluated on five widely used text classification benchmarks:
- TREC: Question classification.
- SST-2 & SST-5: Binary and fine-grained sentiment analysis.
- AG News: Topic classification.
- MNLI: Natural language inference.
Training for the distilled student was performed on a subset (15%) of the training data, while evaluation was conducted on up to 2,000 test samples per dataset.
Key Insights and Ideas
- Similarity $\neq$ Utility: The authors found a notable discrepancy between semantic similarity and actual utility. As shown in Figure 2, Jev-guided reranking replaces a substantial proportion (often over 80% for 1-shot) of the examples initially chosen by kNN.
- Decision Quality Over Model Size: Interestingly, the smaller NeoJEV-4B model often outperformed the larger JEV-9B in identifying informative demonstrations. This suggests that the specific discriminative tuning of a decision model is more important than raw parameter count for this task.
- Distillation is Highly Effective: The lightweight MLP student achieved approximately 70% pairwise agreement with its teacher (Jev), proving that complex utility judgments can be compressed into a very simple model for efficient inference.
- Cross-Task Generalization: A student trained on four datasets could successfully generalize to a fifth "unseen" dataset, outperforming kNN baselines even without task-specific tuning.
Key Results
- Accuracy Gains: DisJEV consistently outperformed kNN across various backbone LLMs. For example, on Qwen3-1.7B, it improved average accuracy by 3.68 percentage points.
- Efficiency: Compared to other training-based retrievers like DiSP, DisJEV is significantly faster. It reduced training time by roughly 3.6x (137.2s vs 497.7s per 100 samples) and maintained lower inference latency.
- Robustness: The method showed consistent improvements even when scaling up to larger backbone models like Llama-3.1-8B and DeepSeek-V4-Flash.
Limitations and Future Work
- Fixed Candidate Pool: The current two-stage approach relies on an initial kNN retrieval to create a pool of 20 candidates. If the "perfect" demonstration isn't in that top 20, the reranker can't find it.
- Task Specificity: While the model generalizes well, the authors noted that performance gains vary by task (e.g., smaller gains on MNLI compared to TREC).
- Future Directions: The authors suggest exploring more sophisticated ways to represent demonstration combinations and investigating how this approach scales to non-classification tasks like code generation or long-form reasoning.
Summary updated