Unsupervised Hierarchical Story Infilling

Daphne Ippolito, David Grangier, Chris Callison-Burch, Douglas Eck2019 · DOI

TL;DR

This paper tackles the challenge of "story infilling" – filling in missing text spans in a story – by proposing a novel two-stage hierarchical model. This approach aims to generate text that is not only fluent and coherent but also diverse and interesting, overcoming a key limitation of existing language models.

What Problem Does This Paper Address?

Creative writing tools often struggle to assist users in a truly helpful way, especially when it comes to generating novel and engaging text. Story infilling, the task of predicting words to replace a missing section of a story, is a prime example. While current language models can produce fluent text, they often fall into a trap: prioritizing fluency and coherence leads to generic, predictable outputs, while aiming for novelty can sacrifice grammatical correctness and logical flow. This paper addresses this delicate balancing act, seeking to generate text that is both high-quality and creatively inspiring.

Methods

The authors introduce a hierarchical, two-stage approach to story infilling.

  1. Word Prediction (High Entropy): The first stage uses a word-sampling model to predict a set of rare words that should appear in the missing span. This is framed as a high-entropy task, where diversity is prioritized. By focusing on less common words, the model injects novelty.
  2. Word-Conditioned Generation (Low Entropy): The second stage employs a standard Transformer model (similar to those used in machine translation) to generate the actual text for the missing span. Crucially, this generation is conditioned on the rare words predicted in the first stage, along with the surrounding text context and the desired length of the span. This stage focuses on fluency and coherence, leveraging the rare words as anchors for more predictable sentence structures.

This decomposition allows the model to tackle the difficult task of balancing novelty and quality by separating the "what" (rare words) from the "how" (fluent generation). They also condition the generation on length bins to ensure fair comparisons.

Datasets

  • Training: The authors trained their models on the Toronto Book Corpus (TBC) concatenated with Project Gutenberg, totaling over 1.2 billion words.
  • Evaluation: For human evaluation, they selected paragraphs from three children's books (Scott, 1921; Barrow, 1863; Vandercook, 1912) to leverage their concise prose. For automatic evaluation, they used a validation set of 10,000 spans.

Key Insights and Ideas

  • Decomposition is Key: The core insight is that separating the task of selecting novel words from the task of generating fluent text is more effective than trying to do both simultaneously.
  • Rare Words as Structure: Rare words, often carrying more information density, can act as effective "seeds" for generating coherent and interesting text, guiding the model without over-constraining it.
  • Hierarchical Control: The hierarchical structure allows for a more nuanced control over the generation process, enabling a better balance between diversity and quality.
  • Human Preference for Partial Conditioning: Surprisingly, human evaluators preferred the model that was conditioned on only the first three predicted rare words (HIER-3) over the one conditioned on all predicted words (HIER-max). This suggests that giving the second-stage model more "headroom" to generate its own common words is beneficial.

Key Results

  • Increased Diversity: The hierarchical model (HIER) significantly increased text diversity (measured by dist-k metrics) compared to baseline models, especially when compared to standard beam search decoding.
  • Balanced Quality: Despite the increased diversity, the hierarchical model did not sacrifice fluency or coherence. Human evaluations showed that their method achieved comparable quality to strong baselines while being more diverse.
  • Human Preference: Human evaluators strongly preferred the HIER-3 model, rating its outputs as comparable to a strong baseline (BASE/sampling10) but with much higher diversity.

Limitations and Future Work

  • Human Evaluation Nuances: While children's books were chosen for their conciseness, the authors acknowledge that the task was still challenging for human annotators, leading to some issues with honeypot questions in initial experiments.
  • Interactive Editing: The authors envision future work where writers could interactively edit the list of predicted rare words, offering a more direct form of control.
  • Alternative Architectures: They suggest exploring insertion-based architectures, which might be more naturally suited to the infilling task, and using n-gram phrases instead of individual subwords for conditioning.

Summary updated

Abstract

Story infilling involves predicting words to go into a missing span from a story. This challenging task has the potential to transform interactive tools for creative writing. However, state-of-the-art conditional language models have trouble balancing fluency and coherence with novelty and diversity. We address this limitation with a hierarchical model which first selects a set of rare words and then generates text conditioned on that set. By relegating the high entropy task of picking rare words to a word-sampling model, the second-stage model conditioned on those words can achieve high fluency and coherence by searching for likely sentences, without sacrificing diversity.
BibTeX
@misc{ippolito2019unsupervised,
  author = {Daphne Ippolito and David Grangier and Chris Callison-Burch and Douglas Eck},
  title = {{Unsupervised Hierarchical Story Infilling}},
  year = {2019},
  doi = {10.18653/v1/w19-2405},
}
Figure