Introduction
While Natural Language Processing (NLP) has achieved significant success in identifying events within factual reporting like news articles, literary fiction presents a unique set of challenges. In a novel, an "event" is not always a concrete action; it can be a metaphor, a dream, or a summary of years passing. Researchers at UC Berkeley have addressed this by developing a new dataset and model specifically for detecting "realis" events—actions that actually occur within the story world—in classic literature.
Methodology
The task is framed as sequence labeling, where each word is classified as either triggering an event or not. The researchers evaluated two primary approaches:
- Featurized Model: A traditional logistic regression model utilizing hand-engineered features such as Part-of-Speech (POS) tags and syntactic dependency relations.
- Neural Models: Architectures including LSTMs and BiLSTMs. The highest performance was achieved by using BERT (Bidirectional Encoder Representations from Transformers) as a feature extractor, concatenating the final four layers to represent each token.
The focus remained strictly on realis events—those depicted as actually happening in the past or present of the narrative—excluding hypothetical or desired actions.
Dataset: The LitBank Project
The study utilizes a new gold-standard dataset from the LitBank project, consisting of 210,532 tokens from the first 2,000 words of 100 public-domain novels. This corpus includes a diverse range of works, from canonical authors like Jane Austen and James Joyce to popular genre fiction like Tarzan of the Apes. A total of 7,849 events were manually annotated using an "open" approach, tagging any word that triggers a realis event rather than restricting labels to specific categories.
Results and Analysis
The BiLSTM + BERT model demonstrated superior performance, achieving an F1 score of 73.9. This represents a substantial improvement over simpler baselines, such as a "Verbs only" approach which only reached an F1 of 28.7.
Beyond model performance, the study revealed a significant stylistic trend: the "Prestige Gap." Analysis showed an inverse relationship between literary prestige and event density. "High prestige" novels, characterized by inclusion in elite reviews or biographical dictionaries, tend to be less "eventful," focusing more on internal states and description. In contrast, "low prestige" or popular novels are approximately 20% more likely to depict concrete physical actions.
Discussion and Future Work
The findings provide quantitative evidence for qualitative theories in literary criticism regarding the nature of "high" vs. "popular" art. However, the current scope is limited to realis events and the initial sections of novels. Future research aims to incorporate "irrealis" thoughts (hopes, fears) and "event coreference" to more fully map the chronological story, or fabula, of a narrative.
Summary updated