TL;DR
This paper introduces semantic frame forecast, a new NLP task that predicts the high-level plot arc of a story by looking hundreds or even thousands of sentences ahead. By representing long "story blocks" as fixed-length vectors of semantic frames, the authors demonstrate that traditional machine learning models can outperform massive transformers like BERT at long-range plot forecasting.
What Problem Does This Paper Address?
Most current story generation and understanding models operate at a "micro" level, predicting the next word or sentence. While this works for short snippets, it fails to capture the "macro" structure—the high-level plot points and thematic shifts—that novelists use to build long-form narratives.
When a writer gets stuck, they don't just need help with the next sentence; they need insight into the next chapter. The challenge is that as stories get longer, the amount of information becomes overwhelming for standard models. This paper seeks a way to "zoom out" and represent long stretches of text (up to 1,000 sentences) in a way that allows a model to predict the thematic "flavor" of what comes next.
Methods
The authors propose a pipeline that moves from raw text to a structured, predictable representation:
- Story Blocks: The text is divided into fixed-length chunks called "story blocks," ranging from 5 to 1,000 sentences.
- Semantic Frame Parsing: Using a tool called Open-SESAME, the model identifies "semantic frames" (based on the FrameNet project). These are conceptual structures like Rebellion, Firefighting, or Kinship that describe the underlying meaning of a sentence regardless of the specific words used.
- Frame Representation (TF-IDF): Each block is converted into a 1,221-dimensional vector. The values are the Term Frequency-Inverse Document Frequency (TF-IDF) of the semantic frames within that block. This creates a fixed-length "fingerprint" for the block, no matter how many sentences it contains.
- Forecasting: The task is to predict the vector for block $n+1$ given the vector(s) for block $n$.
The authors compared several models, including simple baselines (Replay, Prior), traditional ML (Random Forest, LGBM), and deep learning approaches (Denoising Autoencoders, BERT, and GPT-2).
Datasets
The study utilized two primary datasets to test the approach across different types of "creative" writing:
- Bookcorpus: A collection of 4,794 fiction books, used to test narrative flow.
- CODA-19: A set of 7,962 human-annotated scientific abstracts, used to see if the model could predict the logical progression of research (e.g., from Background to Method to Findings).
Key Insights and Ideas
- The Power of Fixed-Length Representations: Unlike sentence-based representations (like SVO tuples), which grow linearly with story length, the frame-based TF-IDF vector stays the same size. This prevents "information overload" and allows models to handle massive context windows.
- BERT’s "Short-Sightedness": While BERT is the gold standard for many NLP tasks, its performance drops significantly as block sizes increase beyond 100 sentences. This is likely due to its internal token limits and the difficulty of compressing 1,000 sentences into a single [CLS] token.
- The "Replay" Decay: The authors found that themes occurring in the current block are very likely to appear in the immediate next block, but this "replay" probability decays quickly as you move further into the future.
- Human-in-the-Loop Potential: By converting the predicted frame vectors back into word clouds (categorized by Nouns, Verbs, and Adjectives), the authors created a tool that humans can actually understand and use for brainstorming.
Key Results
- LGBM Wins at Scale: For story blocks larger than 150 sentences, the LGBM (Light Gradient Boosting Machine) model using frame features outperformed all other models, including BERT.
- BERT Wins at the Micro-Level: For smaller blocks (under 100 sentences), BERT remained the top performer, suggesting that raw text features are more useful when the context is tight.
- GPT-2 Struggles with Plot: Interestingly, GPT-2 performed poorly at this specific forecasting task, even falling behind the simple "Replay" baseline. This suggests that while GPT-2 is great at generating fluent sentences, it struggles to maintain long-term thematic coherence without external guidance.
- Human Validation: In a crowdsourced evaluation, humans were able to correctly match a story block to its corresponding word cloud with 74% accuracy (via majority vote), proving the representation is specific and representative.
Limitations and Future Work
- Context Window: Most of the study focused on using just one previous block to predict the next. While increasing the context window helped the LGBM model, it actually hurt BERT's performance.
- From Forecast to Generation: This paper focuses on predicting the frames. The logical next step is generating the actual story text based on those predicted frames—essentially using the forecast as a "plot roadmap."
- Frame Granularity: Some semantic frames are very generic (e.g., "State_continue"). The authors suggest that filtering for more "important" or "eventful" frames might improve forecasting accuracy.
Summary updated