Plan ahead: Self-Supervised Text Planning for Paragraph Completion Task

Dongyeop Kang, Eduard H. Hovy

Abstract

Despite the recent success of contextualized language models on various NLP tasks, language models themselves often fail to capture the textual coherence of long, multi-sentence documents. Humans typically make structural decisions about what to say and how to say it before making utterances. This work proposes a self-supervised text planner, SSPlanner, designed for the new paragraph completion task (PARCOM). SSPlanner predicts content keywords first and then guides a pretrained language model to generate coherent text. Our results show that SSPlanner significantly outperforms baseline models in both automatic and human evaluations.

Introduction: The Coherence Gap

While modern Large Language Models (LLMs) like GPT-2 and BERT are remarkably fluent at a local level, they frequently struggle with macro-level coherence—maintaining a consistent logical flow across multiple sentences. Humans do not generate words in a vacuum; we rely on a mental blueprint of key points. Existing tasks like Next Sentence Prediction are often too simplistic to capture this planning process.

To address this, the authors introduce PARCOM (Paragraph Completion). In this task, a model must predict masked sentences within a paragraph, requiring it to bridge the gap between preceding and following contexts. As shown in Figure 1, PARCOM is a comprehensive task that necessitates content selection, planning, ordering, and surface realization, distinguishing it from simpler tasks like summarization or paraphrasing.

Comparison of NLP tasks across planning dimensions.

SSPlanner: Planning Before Speaking

The proposed SSPlanner (Self-Supervised Planner) framework decomposes text generation into two distinct stages:

  1. Content Prediction (The Plan): Given the surrounding context, the model predicts a "bag-of-words" plan—a set of key nouns and verbs likely to appear in the missing sentences.
  2. Surface Realization (The Execution): A pretrained language model (e.g., GPT-2) generates the actual sentences. It utilizes a pointer-generator mechanism to incorporate the predicted plan keywords into the final output.

The architecture, illustrated in Figure 2, demonstrates how the Plan Predictor and Transformer LM work in tandem to produce the predicted target by combining word distributions from both the planner and the language model.

Architecture of SSPlanner, showing the interaction between the Plan Predictor and the Transformer LM.

To train this system without manual labels, the authors employed self-supervision techniques such as permutation masking (creating diverse training examples by masking various sentence combinations) and plan extraction (automatically extracting keywords from ground-truth sentences during training to teach the model what constitutes an effective plan).

Experimental Results and Insights

The model was evaluated across diverse domains, including narrative (BookCorpus), factual (WikiText-103), and news (CNN/DailyMail) datasets.

Key Findings:

  • The Importance of Nouns and Verbs: Testing various keyword types revealed that a combination of nouns and verbs is most effective. This aligns with cognitive science theories suggesting human knowledge is organized around entities (nouns) and actions (verbs).
  • Performance Gains: SSPlanner demonstrated substantial improvements over baselines. On the Fantasy dataset, SSPlanner (GPT-2) achieved a METEOR score of 9.2, nearly doubling the 5.0 score of a standard fine-tuned GPT-2.
  • Human Evaluation: In blind tests, humans rated SSPlanner significantly higher in coherence (3.8 vs. 2.1) and overall quality (3.9 vs. 3.6) compared to standard models.

Conclusion and Future Work

SSPlanner proves that self-supervised planning is a viable path toward more coherent AI writing. However, limitations remain. The current model predicts approximately 43% of the "correct" keywords, and the "bag-of-words" approach does not yet capture the chronological or causal order of events. Future research may explore more complex structures, such as discourse trees or causality chains, to provide even more precise guidance for text generation.

Summary updated

Abstract

Despite the recent success of contextualized language models on various NLP tasks, language model itself cannot capture textual coherence of a long, multi-sentence document (e.g., a paragraph). Humans often make structural decisions on what and how to say about before making utterances. Guiding surface realization with such high-level decisions and structuring text in a coherent way is essentially called a planning process. Where can the model learn such high-level coherence? A paragraph itself contains various forms of inductive coherence signals called self-supervision in this work, such as sentence orders, topical keywords, rhetorical structures, and so on. Motivated by that, this work proposes a new paragraph completion task PARCOM; predicting masked sentences in a paragraph. However, the task suffers from predicting and selecting appropriate topical content with respect to the given context. To address that, we propose a self-supervised text planner SSPlanner that predicts what to say first (content prediction), then guides the pretrained language model (surface realization) using the predicted content. SSPlanner outperforms the baseline generation models on the paragraph completion task in both automatic and human evaluation. We also find that a combination of noun and verb types of keywords is the most effective for content selection. As more number of content keywords are provided, overall generation quality also increases.
BibTeX
@inproceedings{kang2020plan,
  author = {Dongyeop Kang and Eduard H. Hovy},
  title = {{Plan ahead: Self-Supervised Text Planning for Paragraph Completion Task}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=0OCzXQZ1ET4},
}
Figure