In-Context Learning with Reinforcement Learning for Incomplete Utterance Rewriting

Haowei Du, Dongyan ZhaoarXiv 2024 · DOI · arXiv

Introduction

In natural conversation, human efficiency often leads to linguistic shortcuts such as ellipsis (omitting words) and co-reference (using pronouns like "it" to refer to previous topics). For an AI to maintain coherence, it must perform Incomplete Utterance Rewriting (IUR)—the task of transforming fragments like "How about Mediterranean food?" into self-contained sentences based on previous context (e.g., "I'm looking for an expensive restaurant").

While Large Language Models (LLMs) excel at this via In-Context Learning (ICL), their performance is highly sensitive to the specific "few-shot" examples provided. Traditional retrieval methods, such as BM25 or simple vector similarity, often select examples that are textually similar to the query but fail to provide the necessary "analogy" for the LLM to rewrite the text correctly. This paper introduces RLS (Reinforcement Learning for example Selection), a framework that optimizes example selection by incorporating direct feedback from the LLM.

The RLS Framework

The authors propose a "Selector-Generator" framework designed to bridge the gap between example retrieval and LLM requirements. The RLS architecture consists of three primary components:

  1. The LM Selector: A small-scale language model (e.g., BERT) that acts as the policy. It encodes candidate examples and the test case into dense vectors to calculate selection probabilities.
  2. The LLM Generator: A frozen LLM (such as ChatGLM or GPT-3.5) that receives the top-$k$ selected examples as a prompt and generates the rewritten utterance.
  3. Policy Iteration: The framework uses the Policy Gradient algorithm to update the LM Selector. The quality of the LLM's output, measured by ROUGE or BLEU scores, serves as the reward signal, training the selector to favor demonstrations that maximize LLM performance.
The RLS framework architecture, illustrating the iterative loop between the LM Selector and the LLM Generator via Policy Gradient.

Experimental Evaluation

The RLS framework was evaluated across three diverse benchmarks: CANARD (English conversational QA), TASK (task-oriented English dialogues), and REWRITE (open-domain Chinese dialogues).

Results indicate that RLS provides significant performance boosts over traditional retrievers. On the TASK dataset, RLS outperformed existing methods like BM25 and EPR by approximately 6.3–7.5 ROUGE points and 7.6–8.2 BLEU points. The framework demonstrated consistent stability across both English and Chinese datasets, proving its cross-lingual robustness.

Comparative performance of RLS against baseline methods on the TASK and REWRITE benchmarks, showing significant gains in ROUGE and BLEU metrics.

Key Insights

  • Beyond Similarity: The study reveals that the most effective examples for LLMs are not necessarily the most similar ones. Instead, LLMs benefit from "linguistically abundant" examples—those with complex structures and diverse parts-of-speech that help the model bridge the "analogy gap."
  • ICL vs. SFT: In few-shot scenarios, RL-optimized example selection actually outperformed Supervised Fine-Tuning (SFT). This suggests that for niche tasks with limited data, optimizing the prompt demonstrations is more effective than updating the model's weights.
  • Model Agnostic Benefits: The framework is not tied to a specific architecture; when tested with GPT-3.5, RLS still provided a significant boost (approx. 4.4 ROUGE points) over random selection.

Limitations and Future Work

While effective, the current RLS framework faces challenges regarding the computational cost of repeated LLM calls during training. Future research may explore scaling the candidate pool beyond the 500 examples used in these experiments and incorporating more nuanced semantic rewards beyond standard ROUGE/BLEU metrics. Additionally, testing the selection policies across a wider variety of LLM families, such as Llama or Claude, remains a key area for future exploration.

Summary updated

Abstract

In-context learning (ICL) of large language models (LLMs) has attracted increasing attention in the community where LLMs make predictions only based on instructions augmented with a few examples. Existing example selection methods for ICL utilize sparse or dense retrievers and derive effective performance. However, these methods do not utilize direct feedback of LLM to train the retriever and the examples selected can not necessarily improve the analogy ability of LLM. To tackle this, we propose our policy-based reinforcement learning framework for example selection (RLS), which consists of a language model (LM) selector and an LLM generator. The LM selector encodes the candidate examples into dense representations and selects the top-k examples into the demonstration for LLM. The outputs of LLM are adopted to compute the reward and policy gradient to optimize the LM selector. We conduct experiments on different datasets and significantly outperform existing example selection methods. Moreover, our approach shows advantages over supervised finetuning (SFT) models in few shot setting. Further experiments show the balance of abundance and the similarity with the test case of examples is important for ICL performance of LLM.
BibTeX
@misc{du2024context,
  author = {Haowei Du and Dongyan Zhao},
  title = {{In-Context Learning with Reinforcement Learning for Incomplete Utterance Rewriting}},
  year = {2024},
  eprint = {2408.13028},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.13028},
}
Figure