Make The Most of Prior Data: A Solution for Interactive Text Summarization with Preference Feedback

Duy-Hung Nguyen, Nguyen-Viet-Dung Nghiem, Bao-Sinh Nguyen, Dung Tien Le, Shahab Sabahi, Minh-Tien Nguyen, Hung Le

Introduction

Document summarization is a highly subjective task. A "good" summary often depends on specific user intent or domain requirements, making it difficult to establish a universal ground truth. While Human-in-the-Loop (HITL) systems offer a path forward by allowing models to learn from real-time feedback, traditional Reinforcement Learning (RL) approaches are notoriously data-hungry. They typically require thousands of human interactions to converge, a requirement that is often impractical for real-world deployment.

This paper addresses the challenge of sample efficiency in interactive summarization. The authors propose a framework that leverages "prior data"—the massive datasets used for initial model training—to accelerate learning from a sparse stream of online human preference feedback.

System Architecture

The proposed framework integrates offline data with online human feedback through a sophisticated feedback loop. As shown in Figure 1, the system utilizes selective sampling to pull relevant examples from offline datasets, which are then used alongside online data to fine-tune the backbone model.

Overview of the interactive summarization framework featuring selective offline sampling and preference-based reward updates.

The architecture consists of three primary components:

  1. The Backbone: The system employs BERTSUM, a BERT-based model designed for extractive summarization. This model selects the most salient sentences from a source document to form a summary.
  2. ROMSR (Respective-Order Mapping Score Reward): A novel reward model that treats preference learning as a distance-mapping problem in a shared vector space.
  3. Selective Offline Sampling: A mechanism to strategically "recycle" prior data to prevent distribution shifts and maximize learning efficiency.

Reward Modeling with ROMSR

Instead of using a standard binary classifier to predict preferences, the authors developed the Respective-Order Mapping Score Reward (ROMSR). This model maps documents and summaries into a shared vector space where the "reward" is inversely proportional to the Euclidean distance between a document and a candidate summary.

To train ROMSR, a triplet loss function was employed with three specific objectives:

  • Topic Alignment: Ensuring the model can distinguish between summaries of the correct document versus a random one.
  • Informativeness (Length): Learning that humans generally prefer longer, more informative summaries over overly brief ones.
  • Quality Ranking: Training the model to rank human-authored "gold" summaries higher than machine-generated alternatives.

Strategic Data Recycling

A key finding of the research is that naively mixing old data with new feedback can actually degrade performance due to distribution shifts. To solve this, the authors introduced two selective sampling strategies:

  • Low-Reward Sampling (LRS): This method identifies offline samples where the current model performs poorly, forcing the model to improve on its weaknesses.
  • Document-Similarity Sampling (DSS): This strategy selects offline documents that are semantically similar to the ones currently being reviewed by the human oracle, ensuring the "recycled" data is highly relevant to the current learning task.

Evaluation and Results

The framework was evaluated across three diverse benchmarks: BillSum (legal), Reddit TIFU (social media), and Livedoor (Japanese news).

The ROMSR reward model demonstrated superior performance in preference prediction accuracy compared to established baselines. As illustrated in Figure 2, ROMSR significantly outperformed the reward models used by OpenAI and the APRIL framework, maintaining higher accuracy even with limited feedback.

Comparison of preference prediction accuracy between ROMSR and existing reward models from OpenAI and APRIL.

In active learning scenarios, the method reached near-peak ROUGE-1 scores after only 64 interactions. On the Reddit TIFU dataset, the DSS method achieved a ROUGE-1 score of 28.6, a significant improvement over the baseline score of 26.0 within the same timeframe. Furthermore, in "few-shot" settings involving only four documents, the model showed immediate improvement, whereas standard baselines tended to overfit and decline in performance.

Conclusion

By combining distance-based reward modeling with strategic offline data sampling, this framework offers a highly sample-efficient solution for interactive text summarization. While currently focused on extractive methods, the principles of "recycling" prior data and using distance as a proxy for preference provide a strong foundation for future work in abstractive summarization and other human-in-the-loop AI applications.

Summary updated

Abstract

For summarization, human preference is critical to tame outputs of the summarizer in favor of human interests, as ground-truth summaries are scarce and ambiguous. Practical settings require dynamic exchanges between human and AI agent wherein feedback is provided in an online manner, a few at a time. In this paper, we introduce a new framework to train summarization models with preference feedback interactively. By properly leveraging offline data and a novel reward model, we improve the performance regarding ROUGE scores and sample-efficiency. Our experiments on three various datasets confirm the benefit of the proposed framework in active, few-shot and online settings of preference learning.
BibTeX
@inproceedings{nguyen2022make,
  author = {Duy-Hung Nguyen and Nguyen-Viet-Dung Nghiem and Bao-Sinh Nguyen and Dung Tien Le and Shahab Sabahi and Minh-Tien Nguyen and Hung Le},
  title = {{Make The Most of Prior Data: A Solution for Interactive Text Summarization with Preference Feedback}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=skzUDpmI0aj},
}
Figure