Introduction
Document summarization is a highly subjective task. A "good" summary often depends on specific user intent or domain requirements, making it difficult to establish a universal ground truth. While Human-in-the-Loop (HITL) systems offer a path forward by allowing models to learn from real-time feedback, traditional Reinforcement Learning (RL) approaches are notoriously data-hungry. They typically require thousands of human interactions to converge, a requirement that is often impractical for real-world deployment.
This paper addresses the challenge of sample efficiency in interactive summarization. The authors propose a framework that leverages "prior data"—the massive datasets used for initial model training—to accelerate learning from a sparse stream of online human preference feedback.
System Architecture
The proposed framework integrates offline data with online human feedback through a sophisticated feedback loop. As shown in Figure 1, the system utilizes selective sampling to pull relevant examples from offline datasets, which are then used alongside online data to fine-tune the backbone model.
The architecture consists of three primary components:
- The Backbone: The system employs BERTSUM, a BERT-based model designed for extractive summarization. This model selects the most salient sentences from a source document to form a summary.
- ROMSR (Respective-Order Mapping Score Reward): A novel reward model that treats preference learning as a distance-mapping problem in a shared vector space.
- Selective Offline Sampling: A mechanism to strategically "recycle" prior data to prevent distribution shifts and maximize learning efficiency.
Reward Modeling with ROMSR
Instead of using a standard binary classifier to predict preferences, the authors developed the Respective-Order Mapping Score Reward (ROMSR). This model maps documents and summaries into a shared vector space where the "reward" is inversely proportional to the Euclidean distance between a document and a candidate summary.
To train ROMSR, a triplet loss function was employed with three specific objectives:
- Topic Alignment: Ensuring the model can distinguish between summaries of the correct document versus a random one.
- Informativeness (Length): Learning that humans generally prefer longer, more informative summaries over overly brief ones.
- Quality Ranking: Training the model to rank human-authored "gold" summaries higher than machine-generated alternatives.
Strategic Data Recycling
A key finding of the research is that naively mixing old data with new feedback can actually degrade performance due to distribution shifts. To solve this, the authors introduced two selective sampling strategies:
- Low-Reward Sampling (LRS): This method identifies offline samples where the current model performs poorly, forcing the model to improve on its weaknesses.
- Document-Similarity Sampling (DSS): This strategy selects offline documents that are semantically similar to the ones currently being reviewed by the human oracle, ensuring the "recycled" data is highly relevant to the current learning task.
Evaluation and Results
The framework was evaluated across three diverse benchmarks: BillSum (legal), Reddit TIFU (social media), and Livedoor (Japanese news).
The ROMSR reward model demonstrated superior performance in preference prediction accuracy compared to established baselines. As illustrated in Figure 2, ROMSR significantly outperformed the reward models used by OpenAI and the APRIL framework, maintaining higher accuracy even with limited feedback.
In active learning scenarios, the method reached near-peak ROUGE-1 scores after only 64 interactions. On the Reddit TIFU dataset, the DSS method achieved a ROUGE-1 score of 28.6, a significant improvement over the baseline score of 26.0 within the same timeframe. Furthermore, in "few-shot" settings involving only four documents, the model showed immediate improvement, whereas standard baselines tended to overfit and decline in performance.
Conclusion
By combining distance-based reward modeling with strategic offline data sampling, this framework offers a highly sample-efficient solution for interactive text summarization. While currently focused on extractive methods, the principles of "recycling" prior data and using distance as a proxy for preference provide a strong foundation for future work in abstractive summarization and other human-in-the-loop AI applications.
Summary updated