Sentence-Level Content Planning and Style Specification for Neural Text Generation

Xinyu Hua, Lu Wang

Abstract

Building effective text generation systems requires three critical components: content selection, text planning, and surface realization. Traditionally, these are tackled as separate problems. Recent all-in-one style neural generation models have made impressive progress, yet they often produce outputs that are incoherent and unfaithful to the input. To address these issues, we present an end-to-end trained two-step generation model, where a sentence-level content planner first decides on the keyphrases to cover as well as a desired language style, followed by a surface realization decoder that generates relevant and coherent text. Automatic evaluation shows that our system can significantly outperform competitive comparisons. Human judges further rate our system generated text as more fluent and correct, compared to the generations by its variants that do not consider language style.

Introduction

While neural text generation has made massive strides in fluency, it often fails at the "macro" level. Standard sequence-to-sequence (seq2seq) models frequently suffer from two major issues: incoherence (losing the thread of the argument) and unfaithfulness (hallucinating facts or ignoring input data).

The authors argue that this happens because these models try to handle content selection, text planning, and surface realization simultaneously in a single "black box." For complex tasks like writing a persuasive argument or a scientific abstract, a model needs to decide which points to hit, in what order, and what tone to use for each specific sentence. This paper aims to restore that structural control without losing the benefits of neural learning.

Methodology: A Two-Stage Framework

The authors propose a two-stage architecture that separates "what to say" from "how to say it":

  1. Sentence-Level Content Planner: This module acts as the "architect." Given a topic and a "memory bank" of keyphrases, it decides for each sentence which keyphrases should be covered and what the functional role (style) of the sentence is (e.g., Claim, Premise, or Functional transition). It uses a history tracking vector to prevent repetitive loops.
  2. Style-Controlled Surface Realizer: This is the "writer." It takes the hidden states from the planner and the chosen style to generate the actual words. It uses a copying mechanism to ensure that technical keyphrases from the input are accurately reflected in the output.
Figure 1: The two-stage generation framework illustrating the interplay between style selection, content planning, and surface realization.

The entire system is trained jointly, optimizing for word generation, keyphrase selection accuracy, and style prediction accuracy simultaneously.

Datasets and Evaluation

The model was evaluated on three distinct domains with varying linguistic requirements:

  • Reddit /r/ChangeMyView: Used for persuasive argument generation. Sentences are annotated as Claim, Premise, or Functional.
  • Wikipedia (Normal vs. Simple): A dataset of topically aligned paragraphs where "style" is approximated by sentence complexity and length, allowing the model to choose between sophisticated or simplified explanations.
  • AGENDA: A dataset of scientific paper abstracts where entities and their relations serve as the keyphrase bank.

Results and Insights

Automated evaluation shows that the model significantly outperformed standard seq2seq baselines and previous state-of-the-art models across BLEU, ROUGE, and METEOR scores. For instance, on the Wikipedia task, the model achieved a BLEU score of 33.76 compared to 22.62 for a standard seq2seq model.

Figure 4: Performance metrics (BLEU-2 and ROUGE-L) plotted against keyphrase selection accuracy, demonstrating the potential gains from improved planning (the "Oracle" gap).

Key insights from the experiments include:

  • Style as a Functional Tool: Explicitly modeling style helps the model stay on track, as different sentence types require different linguistic patterns and content density.
  • Content-Style Interplay: The authors found a strong correlation between content and style. In Wikipedia, more complex sentence styles were naturally associated with a higher number of keyphrases, suggesting the model learns to "pack" information based on the chosen style.
  • The "Oracle" Gap: When the model was given a "Gold Standard" plan (an Oracle Plan), performance jumped even higher. This indicates that the surface realizer is highly effective at following instructions, and the main bottleneck for future improvement lies in making the planner even smarter.

Conclusion and Future Work

By separating content planning from surface realization, this framework produces more coherent and faithful long-form content than traditional all-in-one models. While the system is excellent at rearranging and paraphrasing input, future work will explore Reinforcement Learning to better navigate the space of possible plans and address limitations like stylistic overfitting and the generation of entirely novel concepts.

Summary updated

Abstract

Building effective text generation systems requires three critical components: content selection, text planning, and surface realization, and traditionally they are tackled as separate problems. Recent all-in-one style neural generation models have made impressive progress, yet they often produce outputs that are incoherent and unfaithful to the input. To address these issues, we present an end-to-end trained two-step generation model, where a sentence-level content planner first decides on the keyphrases to cover as well as a desired language style, followed by a surface realization decoder that generates relevant and coherent text. For experiments, we consider three tasks from domains with diverse topics and varying language styles: persuasive argument construction from Reddit, paragraph generation for normal and simple versions of Wikipedia, and abstract generation for scientific articles. Automatic evaluation shows that our system can significantly outperform competitive comparisons. Human judges further rate our system generated text as more fluent and correct, compared to the generations by its variants that do not consider language style.
BibTeX
@inproceedings{hua2019sentence,
  author = {Xinyu Hua and Lu Wang},
  title = {{Sentence-Level Content Planning and Style Specification for Neural Text Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=YUj6MNCOCOA},
}
Figure