Semantic Frame Forecast

Chieh-Yang Huang, Ting-Hao 'Kenneth' Huang

TL;DR

This paper introduces semantic frame forecast, a new NLP task that predicts the high-level plot arc of a story by looking hundreds or even thousands of sentences ahead. By representing long "story blocks" as fixed-length vectors of semantic frames, the authors demonstrate that traditional machine learning models can outperform massive transformers like BERT at long-range plot forecasting.

What Problem Does This Paper Address?

Most current story generation and understanding models operate at a "micro" level, predicting the next word or sentence. While this works for short snippets, it fails to capture the "macro" structure—the high-level plot points and thematic shifts—that novelists use to build long-form narratives.

When a writer gets stuck, they don't just need help with the next sentence; they need insight into the next chapter. The challenge is that as stories get longer, the amount of information becomes overwhelming for standard models. This paper seeks a way to "zoom out" and represent long stretches of text (up to 1,000 sentences) in a way that allows a model to predict the thematic "flavor" of what comes next.

Figure 1: The semantic frame forecast is a task that predicts the semantic frames that will occur in the next part of a story based on the texts written so far.

Methods

The authors propose a pipeline that moves from raw text to a structured, predictable representation:

  1. Story Blocks: The text is divided into fixed-length chunks called "story blocks," ranging from 5 to 1,000 sentences.
  2. Semantic Frame Parsing: Using a tool called Open-SESAME, the model identifies "semantic frames" (based on the FrameNet project). These are conceptual structures like Rebellion, Firefighting, or Kinship that describe the underlying meaning of a sentence regardless of the specific words used.
  3. Frame Representation (TF-IDF): Each block is converted into a 1,221-dimensional vector. The values are the Term Frequency-Inverse Document Frequency (TF-IDF) of the semantic frames within that block. This creates a fixed-length "fingerprint" for the block, no matter how many sentences it contains.
  4. Forecasting: The task is to predict the vector for block $n+1$ given the vector(s) for block $n$.

The authors compared several models, including simple baselines (Replay, Prior), traditional ML (Random Forest, LGBM), and deep learning approaches (Denoising Autoencoders, BERT, and GPT-2).

The pipeline: parsing text into frames, converting to TF-IDF vectors, and visualizing as word clouds.

Datasets

The study utilized two primary datasets to test the approach across different types of "creative" writing:

  • Bookcorpus: A collection of 4,794 fiction books, used to test narrative flow.
  • CODA-19: A set of 7,962 human-annotated scientific abstracts, used to see if the model could predict the logical progression of research (e.g., from Background to Method to Findings).

Key Insights and Ideas

  • The Power of Fixed-Length Representations: Unlike sentence-based representations (like SVO tuples), which grow linearly with story length, the frame-based TF-IDF vector stays the same size. This prevents "information overload" and allows models to handle massive context windows.
  • BERT’s "Short-Sightedness": While BERT is the gold standard for many NLP tasks, its performance drops significantly as block sizes increase beyond 100 sentences. This is likely due to its internal token limits and the difficulty of compressing 1,000 sentences into a single [CLS] token.
  • The "Replay" Decay: The authors found that themes occurring in the current block are very likely to appear in the immediate next block, but this "replay" probability decays quickly as you move further into the future.
  • Human-in-the-Loop Potential: By converting the predicted frame vectors back into word clouds (categorized by Nouns, Verbs, and Adjectives), the authors created a tool that humans can actually understand and use for brainstorming.
Visualizing the frame representation as word clouds helps humans perceive the conceptual meaning of a story block.

Key Results

  • LGBM Wins at Scale: For story blocks larger than 150 sentences, the LGBM (Light Gradient Boosting Machine) model using frame features outperformed all other models, including BERT.
  • BERT Wins at the Micro-Level: For smaller blocks (under 100 sentences), BERT remained the top performer, suggesting that raw text features are more useful when the context is tight.
  • GPT-2 Struggles with Plot: Interestingly, GPT-2 performed poorly at this specific forecasting task, even falling behind the simple "Replay" baseline. This suggests that while GPT-2 is great at generating fluent sentences, it struggles to maintain long-term thematic coherence without external guidance.
  • Human Validation: In a crowdsourced evaluation, humans were able to correctly match a story block to its corresponding word cloud with 74% accuracy (via majority vote), proving the representation is specific and representative.

Limitations and Future Work

  • Context Window: Most of the study focused on using just one previous block to predict the next. While increasing the context window helped the LGBM model, it actually hurt BERT's performance.
  • From Forecast to Generation: This paper focuses on predicting the frames. The logical next step is generating the actual story text based on those predicted frames—essentially using the forecast as a "plot roadmap."
  • Frame Granularity: Some semantic frames are very generic (e.g., "State_continue"). The authors suggest that filtering for more "important" or "eventful" frames might improve forecasting accuracy.

Summary updated

Abstract

This paper introduces semantic frame forecast, a task that predicts the semantic frames that will occur in the next 10, 100, or even 1,000 sentences in a running story. Prior work focused on predicting the immediate future of a story, such as one to a few sentences ahead. However, when novelists write long stories, generating a few sentences is not enough to help them gain high-level insight to develop the follow-up story. In this paper, we formulate a long story as a sequence of "story blocks," where each block contains a fixed number of sentences (e.g., 10, 100, or 200). This formulation allows us to predict the follow-up story arc beyond the scope of a few sentences. We represent a story block using the term frequencies (TF) of semantic frames in it, normalized by each frame's inverse document frequency (IDF). We conduct semantic frame forecast experiments on 4,794 books from the Bookcorpus and 7,962 scientific abstracts from CODA-19, with block sizes ranging from 5 to 1,000 sentences. The results show that automated models can forecast the follow-up story blocks better than the random, prior, and replay baselines, indicating the task's feasibility. We also learn that the models using the frame representation as features outperform all the existing approaches when the block size is over 150 sentences. The human evaluation also shows that the proposed frame representation, when visualized as word clouds, is comprehensible, representative, and specific to humans. Our code is available at https://github.com/appleternity/FrameForecasting.
BibTeX
@inproceedings{huang2021semantic,
  author = {Chieh-Yang Huang and Ting-Hao 'Kenneth' Huang},
  title = {{Semantic Frame Forecast}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.215},
}
Figure