Identifying Sensible Lexical Relations in Generated Stories

Melissa RoemmeleNAACL 2019

TL;DR

This paper introduces a novel method for evaluating the "narrative sense" of generated stories by analyzing the statistical relationships between words. By comparing these relationships in machine-generated stories to those found in human-authored texts, the authors provide a more nuanced understanding of story coherence beyond simple fluency.

What Problem Does This Paper Address?

Current text generation systems, particularly those for stories, often produce text that is grammatically correct and fluent but lacks a deeper sense of coherence or "narrative sense." While human readers can intuitively judge a story's sensibility, automated metrics for this are scarce. Existing metrics like BLEU are ill-suited for open-ended generation tasks like story writing, where there's no single "correct" output. This paper aims to bridge this gap by developing a method to quantify lexical relations that contribute to a story's coherence, moving beyond superficial linguistic quality.

Methods

The core of the paper's methodology lies in analyzing "narrative sense relations," which are defined as statistical word co-occurrences within stories. Specifically, the authors use Pointwise Mutual Information (PMI) to quantify the strength of these associations. They build a "narrative sense relations dataset" by calculating PMI scores for word pairs across four large, diverse corpora of human-authored stories (ROCStories, VIST, CMU Plots, and Children's Book Test). To ensure they capture relations that span across a story rather than just within short phrases, they exclude co-occurrences within the same trigram.

For evaluation, they process generated stories by lemmatizing words and removing punctuation, symbols, numbers, and common English stopwords, focusing on content words. They then calculate the PMI scores for word pairs within each generated story using their pre-computed narrative sense relations dataset. A key innovation is a prediction task designed to distinguish genuine narrative sense relations from those that might arise by chance. This involves comparing the PMI scores of word pairs in a generated story against those in a randomly generated story of the same length using the same vocabulary. A story is deemed to exceed a "narrative sense threshold" if its word pair scores are statistically significantly higher than those in the random story, indicating that the relations are unlikely to be due to chance alone.

Datasets

  • Narrative Sense Relations Dataset: Constructed from four diverse corpora:
  • ROCStories: 97,027 five-sentence narratives.
  • Visual Information Storytelling (VIST): 50,200 five-sentence stories based on photo sequences.
  • CMU Plots: 58,862 book/movie plot summaries (truncated to 150 tokens).
  • Children's Book Test: Passages from 98 children's novels (segmented into 150-token passages).
  • Story Generation Seed Inputs: 15,138 items from the test set of the English-language Reddit WritingPrompts dataset.

Key Insights and Ideas

  • Lexical relations as a proxy for narrative sense: The paper proposes that the statistical co-occurrence of words within a story can serve as a quantifiable indicator of its coherence and "narrative sense."
  • Beyond fluency: This approach moves beyond traditional metrics that focus on surface-level linguistic correctness, aiming to capture a deeper semantic and narrative quality.
  • Distinguishing chance from intentionality: The prediction task using random stories is a clever way to isolate genuine narrative sense relations from those that might emerge simply due to word frequency or random chance.
  • Model-specific strengths and weaknesses: Different generation models exhibit distinct patterns in their lexical relations, offering insights into their underlying generation strategies.

Key Results

  • Human stories exhibit strong narrative sense: Human-authored stories consistently exceed the narrative sense threshold, indicating their lexical relations are less likely to be predicted by chance.
  • Model differences:
    • HIERARCHICAL and L2W models tend to generate moderately strong relations consistently, with narrower score distributions and fewer overall word pairs.
  • GPT-2 and HUMAN stories contain a wider range of word relations, including many pairs not observed in the narrative sense dataset, leading to lower median scores for the full distribution but also a greater likelihood of very strong relations.
  • CREATIVEHELP stories show the least distinct narrative sense relations, suggesting their relations might be more influenced by the number of word combinations rather than genuine narrative coherence.
  • GPT-2 vs. HUMAN: While GPT-2 generates many strong relations, individual GPT-2 stories tend to have less narrative sense than HUMAN stories when considering overall word distributions.

Limitations and Future Work

  • Dataset bias: The authors acknowledge that their narrative sense relations dataset cannot have full coverage of all sensible word pairs and that the choice of corpora might influence results. Future work could explore using different datasets.
  • Focus on positive relations: The current approach rewards models for producing relations that appear frequently in human stories. Future work could explore penalizing models for relations that violate commonsense or detract from coherence.
  • Evaluating creativity: The paper notes that an ideal system would model human creativity by inducing novel lexical relations that support coherence, which is a direction for future research.
  • Reproducibility: The authors suggest reproducing the analysis with different narrative sense relation datasets to better understand the impact of the dataset choice.

Summary updated

BibTeX
@inproceedings{roemmele2019identifying,
  author = {Melissa Roemmele},
  title = {{Identifying Sensible Lexical Relations in Generated Stories}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure