DYPLOC: Dynamic Planning of Content Using Mixed Language Models for Text Generation

Xinyu Hua, Ashwin Sreevatsa, Lu Wang

TL;DR

This paper introduces DYPLOC, a novel framework for generating long-form opinion texts that dynamically plans content as it writes. By using a "mixed language model" approach and augmenting input content with predicted concepts and claims, DYPLOC significantly improves coherence and richness compared to existing methods.

What Problem Does This Paper Address?

Generating coherent and informative long-form opinion pieces is challenging for current AI models. Existing neural generation models often struggle with maintaining a consistent flow of ideas, leading to a lack of coherence. Furthermore, these models need to incorporate diverse information, including both subjective opinions and objective facts, to produce high-quality content. While large pre-trained language models have shown promise, they still fall short in controlling the content and structure of generated text, especially for longer outputs. The paper aims to bridge this gap by developing a system that can effectively plan and generate opinionated text with rich and diverse content.

Methods

The core of DYPLOC is its dynamic content planning mechanism. Instead of a rigid, pre-defined plan, DYPLOC plans content on-the-fly during generation. This is achieved through a novel design of mixed language models.

Key components of the DYPLOC framework include:

  • Content Item Augmentation: The input to the model is a set of "content items," each containing a title, entities, and core concepts. DYPLOC enhances these items by:
  • Concept Expansion: Using a fine-tuned BART model to predict additional relevant concepts based on the title, entities, and core concepts.
  • Claim Generation: Employing another fine-tuned BART model to generate a pertinent claim, which acts as a central proposition for the opinion text.
  • Mixed Conditional Language Model: At each step of text generation, DYPLOC doesn't rely on a single language model. Instead, it uses a weighted sum of multiple language models, each conditioned on a specific augmented content item.
  • Plan Scoring Network: A crucial element is the "plan scoring network" that dynamically learns to select and order content items based on what has already been generated. This network outputs a distribution over content items, indicating their importance for the next token prediction. This allows the model to adapt its content selection as the text unfolds.
  • End-to-End Training: The entire framework, including the content planning and generation modules, is trained end-to-end.

Datasets

The authors created and utilized two datasets for their experiments:

  • Reddit ChangeMyView (CMV): This dataset consists of argumentative discussions from the Reddit community "ChangeMyView." The goal is to generate counter-arguments to an original post's title.
  • New York Times (NYT) Opinion Section: This dataset comprises opinion articles from the New York Times. The task here is to generate longer opinion pieces.

The paper also describes the construction of training data for the concept expansion and claim generation modules, leveraging sources like Wikipedia and Kialo for concepts and claims, respectively.

Key Insights and Ideas

  • Dynamic Planning is Crucial: The paper strongly suggests that static, pre-defined content plans are insufficient for generating coherent long-form text. Dynamic planning, which adapts to the generated text, is key to improving quality.
  • Mixed Models for Richness: Using a weighted combination of language models, each focused on a specific content item, allows the generator to draw from diverse information sources and create richer, more informative outputs.
  • Augmenting Input Matters: Simply providing raw entities and concepts is not enough. Expanding these with predicted concepts and generating explicit claims significantly boosts the quality and informativeness of the generated text.
  • Interpretability through Plan Scores: The learned plan scores provide a degree of interpretability, showing which content items the model prioritizes at different generation stages.

Key Results

  • Superior Performance: DYPLOC significantly outperformed competitive baselines, including fine-tuned BART models, on both CMV and NYT datasets across automatic metrics like BLEU, ROUGE, and METEOR.
  • Human Evaluation Wins: Human judges rated DYPLOC's generations as more coherent and content-rich compared to other systems. DYPLOC was ranked as the best system in a substantial percentage of cases (44.7% on CMV, 45.9% on NYT).
  • Impact of Components: Ablation studies showed that removing concepts led to the biggest performance drop, highlighting their importance. Claims also contributed to argumentative quality, as perceived by human readers.
  • Robustness to Noisy Inputs: Even when using generated concepts (which are inherently less perfect than gold-standard ones), DYPLOC maintained a significant performance lead, demonstrating its robustness.

Limitations and Future Work

  • Claim Realization Nuances: While claims improve argumentative quality, the paper notes that the model doesn't always perfectly capture nuanced sentiments or argumentative structures, sometimes leading to disagreements. Future work could focus on better representations for stances and argumentative structures.
  • Concept Accuracy: The performance drop when using generated concepts suggests that improving the accuracy and comprehensiveness of concept expansion is a promising direction.
  • Potential for Misuse: As with many powerful language generation models, there's a recognition that DYPLOC could be misused to create fabricated or offensive content. The authors advocate for responsible deployment.

Summary updated

BibTeX
@inproceedings{hua2021dyploc,
  author = {Xinyu Hua and Ashwin Sreevatsa and Lu Wang},
  title = {{DYPLOC: Dynamic Planning of Content Using Mixed Language Models for Text Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=VEG-T3hYgML},
}
Figure