Literary Event Detection

Matthew Sims, Jong Ho Park, David BammanACL 2019 · OpenReview

Introduction

While Natural Language Processing (NLP) has achieved significant success in identifying events within factual reporting like news articles, literary fiction presents a unique set of challenges. In a novel, an "event" is not always a concrete action; it can be a metaphor, a dream, or a summary of years passing. Researchers at UC Berkeley have addressed this by developing a new dataset and model specifically for detecting "realis" events—actions that actually occur within the story world—in classic literature.

Methodology

The task is framed as sequence labeling, where each word is classified as either triggering an event or not. The researchers evaluated two primary approaches:

  1. Featurized Model: A traditional logistic regression model utilizing hand-engineered features such as Part-of-Speech (POS) tags and syntactic dependency relations.
  2. Neural Models: Architectures including LSTMs and BiLSTMs. The highest performance was achieved by using BERT (Bidirectional Encoder Representations from Transformers) as a feature extractor, concatenating the final four layers to represent each token.

The focus remained strictly on realis events—those depicted as actually happening in the past or present of the narrative—excluding hypothetical or desired actions.

Dataset: The LitBank Project

The study utilizes a new gold-standard dataset from the LitBank project, consisting of 210,532 tokens from the first 2,000 words of 100 public-domain novels. This corpus includes a diverse range of works, from canonical authors like Jane Austen and James Joyce to popular genre fiction like Tarzan of the Apes. A total of 7,849 events were manually annotated using an "open" approach, tagging any word that triggers a realis event rather than restricting labels to specific categories.

Results and Analysis

The BiLSTM + BERT model demonstrated superior performance, achieving an F1 score of 73.9. This represents a substantial improvement over simpler baselines, such as a "Verbs only" approach which only reached an F1 of 28.7.

Performance comparison of various event detection methods, highlighting the superiority of the BiLSTM + BERT model.

Beyond model performance, the study revealed a significant stylistic trend: the "Prestige Gap." Analysis showed an inverse relationship between literary prestige and event density. "High prestige" novels, characterized by inclusion in elite reviews or biographical dictionaries, tend to be less "eventful," focusing more on internal states and description. In contrast, "low prestige" or popular novels are approximately 20% more likely to depict concrete physical actions.

Comparison of event ratios and distances across different literary classes, showing lower event density in high prestige works.

Discussion and Future Work

The findings provide quantitative evidence for qualitative theories in literary criticism regarding the nature of "high" vs. "popular" art. However, the current scope is limited to realis events and the initial sections of novels. Future research aims to incorporate "irrealis" thoughts (hopes, fears) and "event coreference" to more fully map the chronological story, or fabula, of a narrative.

Summary updated

BibTeX
@inproceedings{sims2019literary,
  author = {Matthew Sims and Jong Ho Park and David Bamman},
  title = {{Literary Event Detection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=lHYV88_PGR},
}
Figure