Introduction
In natural conversation, human efficiency often leads to linguistic shortcuts such as ellipsis (omitting words) and co-reference (using pronouns like "it" to refer to previous topics). For an AI to maintain coherence, it must perform Incomplete Utterance Rewriting (IUR)—the task of transforming fragments like "How about Mediterranean food?" into self-contained sentences based on previous context (e.g., "I'm looking for an expensive restaurant").
While Large Language Models (LLMs) excel at this via In-Context Learning (ICL), their performance is highly sensitive to the specific "few-shot" examples provided. Traditional retrieval methods, such as BM25 or simple vector similarity, often select examples that are textually similar to the query but fail to provide the necessary "analogy" for the LLM to rewrite the text correctly. This paper introduces RLS (Reinforcement Learning for example Selection), a framework that optimizes example selection by incorporating direct feedback from the LLM.
The RLS Framework
The authors propose a "Selector-Generator" framework designed to bridge the gap between example retrieval and LLM requirements. The RLS architecture consists of three primary components:
- The LM Selector: A small-scale language model (e.g., BERT) that acts as the policy. It encodes candidate examples and the test case into dense vectors to calculate selection probabilities.
- The LLM Generator: A frozen LLM (such as ChatGLM or GPT-3.5) that receives the top-$k$ selected examples as a prompt and generates the rewritten utterance.
- Policy Iteration: The framework uses the Policy Gradient algorithm to update the LM Selector. The quality of the LLM's output, measured by ROUGE or BLEU scores, serves as the reward signal, training the selector to favor demonstrations that maximize LLM performance.
Experimental Evaluation
The RLS framework was evaluated across three diverse benchmarks: CANARD (English conversational QA), TASK (task-oriented English dialogues), and REWRITE (open-domain Chinese dialogues).
Results indicate that RLS provides significant performance boosts over traditional retrievers. On the TASK dataset, RLS outperformed existing methods like BM25 and EPR by approximately 6.3–7.5 ROUGE points and 7.6–8.2 BLEU points. The framework demonstrated consistent stability across both English and Chinese datasets, proving its cross-lingual robustness.
Key Insights
- Beyond Similarity: The study reveals that the most effective examples for LLMs are not necessarily the most similar ones. Instead, LLMs benefit from "linguistically abundant" examples—those with complex structures and diverse parts-of-speech that help the model bridge the "analogy gap."
- ICL vs. SFT: In few-shot scenarios, RL-optimized example selection actually outperformed Supervised Fine-Tuning (SFT). This suggests that for niche tasks with limited data, optimizing the prompt demonstrations is more effective than updating the model's weights.
- Model Agnostic Benefits: The framework is not tied to a specific architecture; when tested with GPT-3.5, RLS still provided a significant boost (approx. 4.4 ROUGE points) over random selection.
Limitations and Future Work
While effective, the current RLS framework faces challenges regarding the computational cost of repeated LLM calls during training. Future research may explore scaling the candidate pool beyond the 500 examples used in these experiments and incorporating more nuanced semantic rewards beyond standard ROUGE/BLEU metrics. Additionally, testing the selection policies across a wider variety of LLM families, such as Llama or Claude, remains a key area for future exploration.
Summary updated