TL;DR
This paper introduces a contrastive learning framework to model narrative salience by training story embeddings on "narrative twins"—stories with identical plots but different surface wording. By evaluating four classic narratological theories, the authors demonstrate that a sentence's ability to summarize the overall narrative is the most reliable computational indicator of its importance to the plot.
What Problem Does This Paper Address?
When we read a story, some events are critical to the plot's progression (like a key revelation in a mystery novel), while others provide background details or minor character color. In Natural Language Processing (NLP), identifying these highly important or "salient" events is crucial for tasks like automatic story summarization and narrative understanding.
However, evaluating narrative salience computationally remains a challenge. Current approaches lack a systematic understanding of how to build story embeddings that capture plot structure rather than surface-level text features. Furthermore, it is unclear which theoretical definition of salience—such as how much a sentence alters a story if deleted, or how resistant it is to being moved around—is most effective when translated into an algorithm.
Methods
The authors propose a contrastive learning framework to learn robust story embeddings. The model is trained using triplets:
- Anchor narrative ($x$): The original story.
- Narrative twin ($x^+$): A positive sample that shares the exact same plot but uses different words.
- Distractor ($x^-$): A hard negative sample that features the same characters and setting but a completely different plot.
By optimizing an InfoNCE loss, a shared bidirectional encoder ($f_\theta$) learns to map twins close together in the embedding space while pushing distractors away.
If explicit narrative twins are unavailable, the authors introduce dropout twins—approximating a twin by passing the same story through the encoder twice with different random dropout masks. For long narratives, they use in-story negatives (different sections/windows of the same text) as distractors.
At inference time, they operationalize four distinct narratological theories to score sentence salience:
- Deletion: Measures how much the story embedding changes when a specific sentence is removed.
- Shifting: Measures the embedding change when a sentence is moved to every other possible position in the text.
- Disruption: Evaluates the level of "surprise" or narrative shift introduced by a sentence given its prior context.
- Summarization: Computes the direct cosine similarity between an individual sentence embedding and the global embedding of the entire story.
Datasets
- ROCStories: A collection of short, human-written 5-sentence stories. The authors used an LLM (GPT-4.1 mini) to generate verbose versions, twins, and distractors for 20,000 stories to create a training corpus. They also collected human annotations on 500 stories to establish a ground-truth salience ranking for evaluation.
- Wikipedia Plot Summaries: A dataset of 99 longer movie plot summaries (averaging ~34 sentences) annotated with human-identified "turning points" (Papalampidi et al., 2019). Aligned narrative twins were sourced from the cross-lingual Tell me again! dataset.
Key Insights and Ideas
- Summarization Wins the Day: Across both short and long stories, the summarization operation consistently outperformed deletion, shifting, and disruption. This reveals that the most narratively salient sentence is simply the one that aligns closest with the global meaning of the entire narrative.
- Dropout is a Powerful Substitute: When parallel plot data isn't available, using simple random network dropout works nearly as well as actual textual narrative twins. This makes the approach highly practical for fully self-supervised narrative modeling.
- Surface Features are Deceptive Shortcuts: Language models easily exploit shallow lexical overlaps (like character names and settings) to match stories. Including hard distractors—negatives with identical characters but altered plots—is absolutely vital to force the model to learn true plot structure.
Key Results
- ROCStories: Contrastively trained narrative twins combined with the summarization operation achieved a Spearman's correlation ($\rho$) of 0.44 and an Area Under the Curve (AUC) of 0.76, vastly outperforming the standard Masked Language Model baseline ($\rho = 0.21$, AUC = 0.62).
- Wikipedia Narratives: The summarization operation achieved the highest average AUC (0.65–0.66), significantly beating out deletion (0.56–0.60) and disruption (0.53–0.54). Dropout twins trained purely with in-story negatives achieved joint-best results, proving the viability of a fully self-supervised setup.
Limitations and Future Work
- Weak LLM Supervision: For short stories, generating hard negative distractors relies on explicitly prompting LLMs to alter the climax of a story, which injects a subtle form of weak LLM supervision into the pipeline.
- Coarse Resolution: The current evaluation operates strictly at the sentence level. However, narrative salience is inherently hierarchical—an important event could be a single clause within a sentence, or span an entire chapter. The authors note that future work should explore training models contrastively across multiple structural levels simultaneously.
Summary updated