Abstract
Controllable summarization requires models to generate summaries that satisfy specific user queries. However, the lack of large-scale labeled datasets for this task often leads models to ignore control signals. We present SOCRATIC pretraining, an unsupervised method that mimics the Socratic method by training models to ask and answer questions about a document. Our approach achieves state-of-the-art results on complex benchmarks like QMSum and SQuALITY and demonstrates significant data efficiency, matching baseline performance with 50% less labeled data.
Introduction
Summarization is inherently subjective; a single document can be summarized in multiple ways depending on a user's specific information needs. This challenge, known as controllable or query-focused summarization, is particularly acute in domains like meeting transcripts or long-form narratives. While general-purpose language models excel at syntax and basic denoising, they often struggle to prioritize information based on external queries. The primary bottleneck is the scarcity of high-quality, human-annotated [Document, Query, Summary] triplets.
To address this, we propose SOCRATIC pretraining. This method shifts the pretraining focus from simple text reconstruction to intent-driven generation. By synthetically generating questions and using them as intermediate targets, we teach the model the underlying relationship between a query and its corresponding salient content.
Methodology
SOCRATIC pretraining transforms raw, unlabeled text into a structured training ground for query-following through a three-step pipeline.
Content Selection and Question Augmentation
The process begins with Content Selection, where the most informative sentences in a document are identified using a technique similar to Gap Sentence Generation (GSG). These sentences represent the "answers" or key points of a potential summary. Subsequently, in the Question Augmentation phase, an off-the-shelf Question Generation (QG) model (e.g., MixQG) is employed to create a question for each selected sentence. This pair effectively simulates a user query and its relevant summary segment.
The "Ask & Answer" Objective
The core innovation is the "Ask & Answer" pretraining objective. In this mode, the model is presented with a masked document and tasked with generating a sequence that includes both the synthetic questions and the original salient sentences.
This objective forces the model to develop a "fine-grained plan." By generating the question before the summary content, the model practices identifying salient information relative to a specific information need. At inference time, when provided with a real user query, the model leverages this learned planning capability to produce more relevant summaries.
Experimental Setup
We evaluated SOCRATIC pretraining on two major query-focused summarization benchmarks:
- QMSum: A multi-domain meeting summarization dataset covering product, academic, and parliamentary discussions.
- SQuALITY: A dataset focused on query-based summaries of short stories.
For the unsupervised pretraining phase, we compared three large-scale corpora: OpenWebText2 (web content), Books3 (fiction and non-fiction), and UnDial (dialogue).
Results
State-of-the-Art Performance
SOCRATIC-pretrained models established new state-of-the-art (SOTA) benchmarks on both QMSum and SQuALITY. Notably, our model outperformed baselines that had been "pre-finetuned" on significantly larger supervised datasets, such as WikiSum.
Impact of Pretraining Corpus
The choice of pretraining data proved critical. As shown in the figure below, the Books3 corpus consistently provided the strongest results across both benchmarks. This suggests that the narrative complexity and diverse information needs found in books are particularly beneficial for learning complex summarization tasks.
Data Efficiency and Instruction Adherence
One of the most significant findings is the model's efficiency in low-resource environments. The SOCRATIC model achieved the same performance level with 250 labeled examples that a standard BART model required 500 examples to reach. Furthermore, human annotators preferred SOCRATIC summaries in 60-65% of cases, citing better adherence to the provided queries.
Conclusion and Future Work
SOCRATIC pretraining demonstrates that teaching models to "ask and answer" is a powerful unsupervised proxy for controllable summarization. While the computational cost of question generation remains a challenge, the gains in query adherence and data efficiency are substantial. Future research will focus on distilling the question generation process and extending the method to multi-lingual and extremely long-document contexts.
Summary updated