Automatic Annotation of Direct Speech in Written French Narratives

Noé Durandard, Viet-Anh Tran, Gaspard Michel, Elena V. Epure

TL;DR

This paper introduces a comprehensive framework for automatically identifying direct speech (DS) in French narratives, a crucial step for computational narrative understanding. They present the largest French DS-annotated dataset to date and benchmark various rule-based and deep learning models, revealing that while deep learning excels, accurately identifying full DS spans remains a challenge.

What Problem Does This Paper Address?

Understanding written narratives often hinges on distinguishing between the narrator's voice and the characters' direct speech (DS). This is vital for tasks like character analysis, dialogue attribution, and even generating audiobooks from text. While automatic direct speech annotation (AADS) has seen progress in English and German, French narratives present unique challenges due to less strict formatting conventions and the prevalence of "incises" (narrative breaks within DS). Existing French AADS efforts are sparse and lack comparability, hindering further research. This paper aims to bridge this gap by creating a unified framework for AADS in French, focusing on building a robust dataset and evaluating models that generalize well across different writing styles.

Methods

The authors tackle AADS by framing it as a token classification task, which they argue is better suited for identifying incises than traditional span-based methods. They adapt and benchmark two main categories of AADS approaches:

  • Rule-based Baselines: They adapt existing regex-based systems (like those from BookNLP and Byszuk et al.) that rely on quotation marks and dialogue dashes. These are modified to better handle French punctuation and text segmentation.
  • Deep Learning Baselines: They employ two deep learning models:
  • A BiLSTM-CRF model, a standard architecture for sequence labeling, adapted from prior work on German AADS.
  • A fine-tuned CamemBERT model, a powerful French language model, to perform sequence labeling.

A key aspect of their deep learning approach is extensive input preprocessing to handle long narrative texts, segmenting them into manageable chunks while preserving contextual coherence. They also explore a clause-consistent post-processing step to ensure that all words within a narrative clause receive the same DS label, aiming to improve overall consistency.

Datasets

The core contribution is a newly consolidated and expanded French narrative dataset for AADS. This dataset comprises:

  • Straight Talk! (ST!): Re-annotated to a finer word-level granularity.
  • SynPaFlex (SPF): Extended from chapter excerpts to full chapters.
  • fr-LitBank: A French variant of the Multilingual BookNLP project.
  • SPF extension: Provided by the authors of SynPaFlex.

In total, the dataset consists of 86 full chapters (approximately 680,000 tokens) from 19th and 20th-century French novels. They also distinguish between a clean test set (Testc), comprising well-formatted texts, and a noisy test set (TestN), which includes texts with significant formatting issues (like those from the original ST! corpus), to evaluate model generalization.

Key Insights and Ideas

  • French AADS is Underserved: The paper highlights the significant gap in AADS research for French compared to English and German, motivating their work.
  • Token Classification for Incises: Framing AADS as token classification is crucial for accurately handling "incises," which are common in French literature and can break up direct speech segments.
  • Deep Learning Outperforms Rules on Noisy Data: While rule-based methods (Regex) perform well on clean, well-formatted texts, deep learning models (especially F.CamemBERT) show much greater robustness and generalization capabilities on texts with poor formatting.
  • Span Identification is the Bottleneck: Even with strong token-level performance, accurately identifying the entire span of direct speech remains a significant challenge for all models, particularly on noisy data.
  • Clause-Consistency Matters: A simple post-processing step to ensure all words within a narrative clause share the same DS label significantly boosts performance, especially for sequence-level metrics.

Key Results

  • F.CamemBERT Dominates: The fine-tuned CamemBERT model consistently achieved the best results across most metrics, particularly on span-level (SSM F1) and generalization (TestN) tasks.
  • Regex is Good for Clean Data: On the clean test set (Testc), the Regex baseline achieved a token-level F1-score of 90%, demonstrating its effectiveness for well-structured texts.
  • Deep Learning Gains on Noisy Data: While Regex's performance dropped drastically on the noisy test set (TestN), F.CamemBERT maintained strong token-level F1-scores (93%) and BiLSTM-CRF also performed well (88%).
  • SSM F1 Improvements: The clause-consistent post-processing step led to substantial gains in SSM F1 scores, with F.CamemBERT improving by up to 8 points on TestN.
  • Homodiegetic Novels are Tricky: The BiLSTM-CRF model struggled with homodiegetic novels (first-person narratives) where the narrator is a character, often misclassifying narration as DS.

Limitations and Future Work

  • Binary Annotation Limitation: The current word-level binary annotation (DS vs. non-DS) doesn't explicitly capture speaker turns or handle imbricated narration (incises) as distinct entities, requiring more complex annotation for future work.
  • Copyrighted Corpus Period: The dataset is limited to novels from a specific period (1830-1937), which might affect model generalization to modern or contemporary narratives.
  • Parameter Tuning: The authors acknowledge that a more extensive parameter search for the deep learning models could potentially yield further improvements.
  • Future Directions:
    • Investigating methods to identify full DS turns and their speakers.
  • Leveraging multilingual language models and existing AADS corpora.
  • Exploring more sophisticated annotation schemes that explicitly handle incises and speaker turns.
  • Further analysis of model performance on specific narrative styles and genres.

Summary updated

BibTeX
@inproceedings{durandard2023automatic,
  author = {Noé Durandard and Viet-Anh Tran and Gaspard Michel and Elena V. Epure},
  title = {{Automatic Annotation of Direct Speech in Written French Narratives}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.393},
}
Figure