Rethinking Scene Segmentation. Advancing Automated Detection of Scene Changes in Literary Texts

Svenja Guhr, Huijun Mao, Fengyi LinNAACL 2025

TL;DR

This paper tackles the challenge of automatically identifying scene changes in literary texts, a crucial step for computational literary studies. They propose a novel approach using a transformer-based classifier, fine-tuned on manually annotated romance novels, to predict scene boundaries with promising results.

What Problem Does This Paper Address?

Segmenting literary texts into meaningful units like scenes is essential for comparative analysis in computational literary studies (CLS). However, automating this process is difficult. Existing methods often rely on coarse divisions like chapter boundaries, which don't always align with narrative structure, or require pre-segmented texts due to computational limitations. This paper aims to develop a robust method for automatically detecting scene changes in English-language fiction, enabling more granular analysis of literary works.

Methods

The authors' approach involves two main stages: manual annotation and automated classification.

  1. Manual Annotation: They manually annotated 20% of a corpus of romance novels, following established guidelines for scene segmentation. This process involved multiple annotators to ensure a "balanced compromise" for gold annotations, acknowledging the subjective nature of scene definition. They found that genre fiction, like romance novels, was easier to annotate than high-brow literature due to its more structured writing style.
  2. Automated Classification:
    • Initial LLM Exploration: They first experimented with large language models (LLMs) like ChatGPT 4-0 and Claude 3.5 for direct scene annotation. However, these models struggled with context and often misinterpreted conversations or over-annotated.
  3. Fine-tuning Transformer Models: Based on these limitations, they shifted to fine-tuning pre-trained transformer models. They adopted a strategy of predicting scene changes within six-sentence segments. This segment size was chosen because scene transitions can be gradual and human annotators often have a tolerance of about three sentences for precise boundary placement.
  4. Model Choice: They fine-tuned both BERT and the Universal Sentence Encoder (USE). While BERT is widely used, they found USE to perform better for this specific task, hypothesizing that its sentence-level embeddings are more suitable for analyzing larger textual segments.

Datasets

  • Training Data:
    • 20% of a corpus of 50 thematically cohesive romance novels from the Harlequin series "Men Made in America" (1982-2002), manually annotated for scene changes.
  • An automatically generated translation of 20 novels from the German scene segmentation project (Zehe et al., 2021b, 2025).
  • Test Data: The final five manually annotated scenes from each of the ten romance novels in the original corpus (totaling 50 scenes).

Key Insights and Ideas

  • Genre Fiction as a Good Starting Point: Romance novels, with their structured narrative, proved to be a more tractable domain for scene segmentation compared to high-brow literature, offering a practical entry point for developing automated methods.
  • The Subjectivity of Scene Definition: The low inter-annotator agreement among human annotators highlights the inherent subjectivity in defining scene boundaries, underscoring the need for consensus-building and the absence of a definitive "ground truth."
  • Six-Sentence Segments as a Sweet Spot: The decision to segment texts into six-sentence chunks for prediction is a novel heuristic that balances the gradual nature of scene changes with computational feasibility.
  • LLMs Aren't Quite There Yet for Complex Segmentation: Despite their power, current LLMs struggle with long-context sequences and nuanced narrative structures, making them less effective for tasks like scene segmentation compared to fine-tuned specialized models.
  • Character Density as a Double-Edged Sword: While character introductions can signal scene changes, a high density of characters can also lead to false positives, as the model might misinterpret new character introductions as breaks. Conversely, recurring character names can lead to false negatives by suggesting continuity where a scene change has occurred.

Key Results

  • The fine-tuned USE model achieved promising results, with the authors noting that it is more prone to false negatives (missing scene changes) than false positives (incorrectly identifying scene changes).
  • On the test set, the model made 403 correct predictions and 90 incorrect ones, with 74 false negatives and 16 false positives.
  • Incorrect predictions (both false positives and false negatives) tended to occur in longer segments.
  • Scenes with a higher number of character mentions were more likely to be misclassified.

Limitations and Future Work

  • Generalizability: The current approach is likely to perform best on genres similar to the annotated corpus (romance novels) and may struggle with more complex, highbrow literary texts due to structural and stylistic differences.
  • Boundary Detection vs. Full Segmentation: The study focuses on detecting scene boundaries rather than fully segmenting and classifying scenes. It does not distinguish between scenes and non-scenes or account for nested scene structures.
  • Comparability: Due to differences in language and test sets, direct comparison with prior work (e.g., Zehe et al.) is challenging.
  • Future Work: The authors suggest developing a sentence-wise prediction model to enhance precision and are reopening the discussion on scene segmentation to add a new perspective for CLS. They also mention the potential for a more hierarchical segmentation approach to address nested structures.

Summary updated

BibTeX
@inproceedings{guhr2025rethinking,
  author = {Svenja Guhr and Huijun Mao and Fengyi Lin},
  title = {{Rethinking Scene Segmentation. Advancing Automated Detection of Scene Changes in Literary Texts}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure