This research presents a novel, unsupervised approach to teaching computers "narrative common sense"—specifically, identifying which events are likely to follow one another in a story. By combining the structured storytelling of film scripts with the massive scale of web search data, the researchers achieved an impressive 85.64% accuracy in identifying contingent event pairs, significantly outperforming standard statistical methods.
The Challenge of Narrative Contingency
When humans read a story, we constantly perform "predictive processing." If a character "reaches for a clock," we aren't surprised when they "shut it off." This is because we understand contingency: the discourse relation where one event causes, enables, or provides the context for another.
For AI, this is incredibly difficult. Most NLP models struggle with common-sense causal reasoning. While frameworks like the Penn Discourse Treebank (PDTB) exist, building supervised models requires massive amounts of human-annotated data, which is expensive and slow to produce. This study explores whether these narrative patterns can be learned automatically from raw text.
Methodology: From Scripts to the Web
The authors proposed a four-step unsupervised pipeline to extract and validate event pairs:
- Text Processing: Scene descriptions were extracted from film scripts (IMSDb). Scripts are ideal because they are written in the "historical present," and events are typically narrated in chronological order.
- Event Representation: Events were simplified into (Subject, Verb, Object) triples. To reduce data sparsity, specific entities were generalized (e.g., "Douglas Quail" becomes "person").
- Initial Contingency Scoring: Four statistical measures were tested to find likely pairs, including Pointwise Mutual Information (PMI) and Causal Potential (CP), which accounts for both frequency and temporal order.
- Web Search Refinement: This critical step "fact-checked" top-ranked pairs against the entire internet. Using Google search patterns like
"he [verb1] * [verb2]"(e.g.,"he slams * shuts"), the system validated whether these pairs appeared frequently in natural language.
Key Insights
- Genre Nuance: Romance scripts often provided better initial estimates for contingency than Action scripts, suggesting that Romance narratives may rely on more structured or predictable social causal chains.
- The Web as a Filter: The most significant finding was that while a specialized corpus (film scripts) is excellent for generating hypotheses, the massive, noisy web is ideal for validating them. The web search refinement step acted as a powerful filter for "narrative noise."
- Abstraction vs. Specificity: Surprisingly, providing human evaluators with full event arguments (e.g., "person SHAKE head") sometimes made it harder for them to judge contingency compared to just seeing the verbs. Specificity can occasionally distract from core common-sense reasoning.
Results and Performance
The system was evaluated using a "forced choice" task where human judges (via Amazon Mechanical Turk) picked the more likely of two event pairs. While the best initial statistical method (Causal Potential) achieved approximately 75% accuracy, the addition of web search refinement pushed the performance to 85.64%.
Limitations and Future Directions
Despite the high accuracy, the system faces challenges with automated tools; coreference resolution (e.g., knowing "he" refers to a specific character) was only about 60% accurate. Furthermore, the current model only examines pairs of events. Future work aims to induce larger, "plot-like" structures—multi-event chains that represent complex real-world scripts, such as the sequence of actions involved in visiting a restaurant.
Reference: Hu, Z., Rahimtoroghi, E., Munishkina, L., Swanson, R., & Walker, M. A. (2013). Unsupervised Induction of Contingent Event Pairs from Film Scenes. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP), 369–379.
Summary updated