Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization

Artidoro Pagnoni, Alexander R. Fabbri, Wojciech Kryscinski, Chien-Sheng Wu

Abstract

In long document controllable summarization, where labeled data is scarce, pretrained models struggle to adapt to the task and effectively respond to user queries. In this paper, we introduce Socratic pretraining, a question-driven, unsupervised pretraining objective specifically designed to improve controllability in summarization tasks. By training a model to generate and answer relevant questions in a given context, Socratic pretraining enables the model to more effectively adhere to user-provided queries and identify relevant content to be summarized. We demonstrate the effectiveness of this approach through extensive experimentation on two summarization domains, short stories and dialogue, and multiple control strategies: keywords, questions, and factoid QA pairs. Our pretraining method relies only on unlabeled documents and a question generation system and outperforms pre-finetuning approaches that use additional supervised data. Furthermore, our results show that Socratic pretraining cuts task-specific labeled data requirements in half, is more faithful to user-provided queries, and achieves state-of-the-art performance on QMSum and SQuALITY.

Introduction

Standard summarization models typically follow a "one-size-fits-all" approach, generating a generic overview of a document. However, real-world applications often require controllable or query-focused summarization, where a model must respond to specific user needs (e.g., "Summarize what Sarah said about the budget").

The primary bottleneck in developing these models is the scarcity of high-quality labeled data. Creating datasets that pair long documents with specific queries and accurate summaries is both expensive and time-consuming. Consequently, models pretrained on standard "denoising" tasks—such as filling in missing words—often fail to adapt to the nuanced requirement of following a user's specific intent or plan.

Methodology: The Socratic Objective

To address this, we introduce SOCRATIC pretraining, an unsupervised method that transforms any unlabeled text into a training environment for controllable summarization. This approach mimics the Socratic method by teaching the model to ask and answer questions before summarizing.

Comparison of standard denoising pretraining versus Socratic pretraining

The process consists of two primary stages:

  1. Content Selection: Utilizing a technique inspired by PEGASUS, the model identifies "important" sentences within a document—those with the highest lexical overlap with the remaining text—to serve as a pseudo-summary. These sentences are then masked in the source document.
  2. Question Augmentation: A separate Question Generation (QG) system creates questions for which the masked sentences are the answers.

The model is trained in Ask&Answer mode: given the masked document, it must first generate the questions (the "plan") and then generate the pseudo-summary (the "answer"). This sequence forces the model to learn the fundamental relationship between a query and the relevant content it must extract.

Experimental Setup

We implemented this objective using a BART-large backbone, integrated with the SegEnc architecture to manage the computational challenges of long-document processing. The models were pretrained on three diverse corpora: OpenWebText2 (web content), Books3 (fiction and non-fiction), and UnDial (dialogue data). Evaluation was conducted on two major benchmarks: QMSum (meeting summarization) and SQuALITY (short story summarization).

Results and Performance

SOCRATIC pretraining achieved new state-of-the-art (SOTA) performance on both benchmarks, reaching 38.06 R-1 on QMSum and 46.31 R-1 on SQuALITY. Beyond raw metrics, the method proved to be a significant "data multiplier." A model utilizing Socratic pretraining and only 50% of the task-specific labeled data performed as well as a standard model using the full dataset.

ROUGE-1 scores on QMSum and SQuALITY datasets across different pretraining corpora

Human evaluation further validated these results, with annotators preferring Socratic-pretrained summaries over standard baselines in 59% to 65% of cases. The model also demonstrated versatility across various control strategies, including keywords and natural language questions.

Key Insights

  • Learning to Plan: Generating questions acts as a fine-grained plan. Predicting questions first teaches the model to identify salient information relative to a specific intent.
  • Sentence-Level Complexity: Unlike previous entity-level approaches, Socratic pretraining focuses on full-sentence answers, providing a better balance of complexity and context.
  • Domain Sensitivity: Pretraining on the Books3 corpus was particularly effective for story and dialogue summarization, suggesting that the stylistic alignment of pretraining data and downstream tasks is crucial.

Limitations and Future Work

While effective, the Question Generation phase is computationally intensive, roughly doubling the total pretraining resource requirements. Additionally, while the method improves adherence to queries, automated metrics for factuality in long-document domains remain a challenge. Future research will explore integrating Socratic objectives into native long-context architectures and expanding the method to cross-lingual datasets.

Summary updated

Abstract

In long document controllable summarization, where labeled data is scarce, pretrained models struggle to adapt to the task and effectively respond to user queries. In this paper, we introduce Socratic pretraining, a question-driven, unsupervised pretraining objective specifically designed to improve controllability in summarization tasks. By training a model to generate and answer relevant questions in a given context, Socratic pretraining enables the model to more effectively adhere to user-provided queries and identify relevant content to be summarized. We demonstrate the effectiveness of this approach through extensive experimentation on two summarization domains, short stories and dialogue, and multiple control strategies: keywords, questions, and factoid QA pairs. Our pretraining method relies only on unlabeled documents and a question generation system and outperforms pre-finetuning approaches that use additional supervised data. Furthermore, our results show that Socratic pretraining cuts task-specific labeled data requirements in half, is more faithful to user-provided queries, and achieves state-of-the-art performance on QMSum and SQuALITY.
BibTeX
@inproceedings{pagnoni2023socratic,
  author = {Artidoro Pagnoni and Alexander R. Fabbri and Wojciech Kryscinski and Chien-Sheng Wu},
  title = {{Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=I-ptgxQhZmY},
}
Figure