When building a context window for a language model—whether you are selecting sentences for an extractive summary, gathering evidence for relation extraction, or picking few-shot exemplars for in-context learning—the standard approach is to rank candidates independently. A retriever computes a similarity score between a query and each candidate text unit, sorting them to take the top $K$ items.
This independent ranking strategy suffers from a fundamental flaw: it ignores inter-candidate redundancy and synergy. If the two highest-scoring sentences contain identical information, an independent ranker will select both, wasting valuable context budget. Conversely, a candidate that appears moderately relevant on its own might be the perfect complement when paired with a previously selected sentence.
To solve this, a growing body of research models text selection not as a single-step ranking problem, but as a sequential decision-making process optimized via Reinforcement Learning (RL). By framing selection as a Markov Decision Process (MDP), an agent can evaluate each candidate conditioned on the history of what has already been chosen.
The Core Intuition: Selection as an MDP
To understand why this approach works, consider how a human builds an argument or a summary. We do not grab a handful of facts all at once; we select one, see what gaps remain, and then look for another fact that fills those specific gaps.
In an MDP framework, this process is formalized through states, actions, and transitions:
- State ($s_t$): The state represents the current composition of the context window. It typically encodes the original query or document cluster along with a representation of the text units selected up to step $t$.
- Action ($a_t$): The action space consists of the remaining unselected text units. The agent evaluates these candidates and selects one to append to the context.
- Transition: Adding the chosen unit updates the state, altering the context embedding for the next step.
Early foundations for this framing were laid by Ryang et al. in their work on Framework of Automatic Text Summarization Using Reinforcement Learning, which demonstrated that sequential selection allows agents to optimize complex, non-decomposable scoring functions that account for coverage and length constraints simultaneously. By treating selection as an evolving history, the model naturally learns to down-regulate the scores of candidates that overlap with the current state, effectively eliminating redundancy.
Managing Synergy and Redundancy
The power of sequential selection becomes clear when dealing with multi-step reasoning or composition. When candidates are selected one by one, the query for the next item is dynamically conditioned on the features of the previously selected items.
[Query] ──> [Agent Policy] ──> Selects Item 1 ──┐
│
┌───────────────────────────────────────────────┘
└──> [Updated State: Query + Item 1] ──> [Agent Policy] ──> Selects Item 2 (Complementary)
This mechanism is highly effective across several distinct domains:
- In-Context Learning (ICL): Traditional retrievers pick a batch of few-shot examples simultaneously. As Chen et al. show in Learning to Retrieve Iteratively for In-Context Learning, selecting exemplars sequentially allows the system to account for inter-example dependencies, maximizing the collective utility of the prompt for a specific downstream LLM.
- Compositional Retrieval: In retrieval-augmented generation (RAG), Long et al. demonstrate in Reinforcing Compositional Retrieval: Retrieving Step-by-Step for Composing Informative Contexts that step-by-step selection allows a retriever to actively identify and fill specific information gaps, creating a diverse and non-redundant context chain that independent ranking cannot replicate.
- Knowledge-Grounded Dialogue: For conversational agents, Liu et al. show in Knowledge-Grounded Dialogue with Reward-Driven Knowledge Selection that retrieving external knowledge snippets sequentially, conditioned on both dialogue history and prior selections, yields a significantly more coherent and contextually relevant integration of facts.
Why Reinforcement Learning?
If sequential selection is so powerful, why not just use a standard supervised sequence-to-sequence model? The answer lies in the nature of the training signal.
In text selection, we rarely have token-level or sentence-level "ground truth" labels showing the single best sequence of choices. A human-written summary or a successful downstream model prediction is a holistic outcome. The final utility metric (such as ROUGE, accuracy, or answerability) is non-differentiable and depends entirely on the final combination of selected texts.
Reinforcement learning bridges this gap by using the downstream task's performance as a reward signal to optimize the discrete selection policy.
Task-Driven and Evaluator-Led Rewards
Rather than forcing a model to match a static human reference, RL allows the selection agent to be trained directly by the module that consumes its output. For example, Chen et al. in Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting use the performance of a downstream abstraction/rewriting module as the reward signal for an extraction agent. The extractor learns to select sentences not based on a heuristic formula, but based on what makes the abstractor succeed.
Answerability-Driven Feedback
A particularly elegant application of this paradigm uses question-answering (QA) models to calculate rewards. In Guiding Extractive Summarization with Question-Answering Rewards, Arumae et al. train a selection agent by generating cloze-style questions from human abstracts. The agent receives a high reward if its selected source sentences contain enough information for a separate QA model to answer those questions correctly.
This loop ensures that the selection policy optimizes for semantic utility and information density, rather than surface-level string matching. It forces the agent to prioritize information that is functionally necessary to resolve downstream queries.
Key Foundational Work
Framework of Automatic Text Summarization Using Reinforcement Learning (2012) — contributed 3 concept elements to this direction.
Finding middle ground? Multi-objective Natural Language Generation from time-series data (2014) — contributed 1 concept elements to this direction.
Fear the REAPER: A System for Automatic Multi-Document Summarization with Reinforcement Learning (2014) — contributed 1 concept elements to this direction.