Sentence-Level Content Planning and Style Specification for Neural Text Generation

Xinyu Hua, Lu Wang

Abstract

Modern neural text generation models often struggle with long-form coherence and faithfulness. This paper introduces a framework that explicitly separates content planning from surface realization at the sentence level. By selecting keyphrases and specifying linguistic styles (e.g., "Claim" vs. "Premise") before generating text, the model achieves superior performance in complex domains like argumentation and Wikipedia generation.

Introduction: The Challenge of Long-Form Generation

While sequence-to-sequence (seq2seq) models excel at local fluency, they frequently falter when producing multi-sentence paragraphs. These "all-in-one" models attempt to handle content selection, logical ordering, and lexical choice simultaneously, often leading to two major issues:

  1. Incoherence: Weak logical transitions between sentences.
  2. Unfaithfulness: "Hallucinations" or omission of critical input facts.

To address these, Hua and Wang propose a structural discipline inspired by traditional pipelined systems, adapted for end-to-end neural training.

Proposed Framework: Planning Before Writing

The authors propose a two-step neural framework that operates at the sentence level to maintain structural integrity.

The proposed model architecture featuring sentence-level content planning and style selection.

1. Sentence-Level Content Planner

The planner determines "what to say" by:

  • Keyphrase Selection: Maintaining a memory bank of input keyphrases and using an LSTM-based planner to select relevant ones for the current sentence.
  • Selection History: Tracking previously used phrases to prevent repetitive output.
  • Style Specification: Predicting a categorical style label for the sentence (e.g., "CLAIM", "PREMISE", or "FUNCTIONAL").

2. Style-Controlled Surface Realizer

The decoder turns the plan into fluent text. It utilizes a dual-attention mechanism over the input and the keyphrase bank, alongside a "copy mechanism" to ensure technical terms are accurately rendered. The predicted style label acts as a control signal, influencing the tone and structure of the generated sentence.

Experimental Setup

The model was evaluated across three diverse datasets:

  • Argument Generation: Based on Reddit’s /r/ChangeMyView, annotated with argumentative roles.
  • Wikipedia Paragraph Generation: A new dataset comparing "Normal" and "Simple" Wikipedia, where style is tied to linguistic complexity.
  • Scientific Abstract Generation: Using the AGENDA dataset to generate abstracts from paper titles and entities.

Key Results and Insights

The framework significantly outperformed competitive baselines. On the Reddit argument task, the model's BLEU score reached 13.19, more than double the previous state-of-the-art (5.73).

Generation quality (BLEU-2 and ROUGE-L) as a function of keyphrase selection accuracy.

The Synergy of Content and Style

A critical finding was that style is not just "flavor"—it is structural. Predicting the style of a sentence (e.g., "Simple Wikipedia") helps the model select the appropriate amount and type of content. Furthermore, as shown in the evaluation metrics, there is a strong correlation between the accuracy of the content plan (Keyphrase Selection F1) and the final generation quality (BLEU/ROUGE), particularly for Wikipedia data.

Reducing Hallucination

By grounding the surface realizer in a pre-selected set of keyphrases, the model is much less likely to invent facts. Human evaluations confirmed that the system's outputs were more fluent and faithful to the intended stance than models without explicit planning.

Conclusion

This work demonstrates that decoupling content planning from realization at the sentence level is a powerful strategy for neural text generation. By treating style as a local, rhetorical choice rather than a global document setting, the model produces more coherent and purposeful text. Future work may involve using Reinforcement Learning to further optimize the diversity and logic of the generated plans.

*Reference: Xinyu Hua and Lu Wang.

  1. Sentence-Level Content Planning and Style Specification for Neural Text Generation. In Proceedings of EMNLP-IJCNLP.*

Summary updated

BibTeX
@inproceedings{hua2019sentence,
  author = {Xinyu Hua and Lu Wang},
  title = {{Sentence-Level Content Planning and Style Specification for Neural Text Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=YUj6MNCOCOA},
}
Figure