TL;DR
This paper introduces the first publicly available, annotated dataset of French emails for anaphora resolution. By creating this resource and detailing their annotation process, the authors aim to foster research into the unique challenges of anaphora in email communication, which differs significantly from other text genres.
What Problem Does This Paper Address?
Natural Language Processing (NLP) tasks, like understanding and summarizing text, heavily rely on anaphora resolution – identifying when different words or phrases refer to the same entity. While extensive research and resources exist for English, the French language, particularly in the context of emails, has been underserved. Emails, with their conversational nature, segmented structure, and use of metadata, present unique challenges for anaphora resolution that existing corpora and methods don't fully capture. This lack of resources hinders the development and evaluation of NLP systems for French emails.
Methods
The authors developed a novel annotation scheme and protocol specifically for French emails. Key aspects include:
- Corpus Creation: They collected 100 anonymized professional email threads (314 emails total) from a company that wished to remain anonymous.
- Anonymization: A state-of-the-art named entity recognizer was used to identify and replace personal information, ensuring consistency in replacements.
- Annotation Protocol: A rigorous, multi-stage process involving three MSc/PhD annotators with linguistics backgrounds was employed. This included:
- Developing detailed annotation guidelines.
- Conducting agreement studies (using Cohen's Kappa and Fleiss' Kappa) to refine the guidelines and ensure annotator consistency.
- A link-based strategy for representing anaphoric relations, focusing on directed dependencies between anaphors and antecedents.
- Typology: They defined specific types of anaphoric relations (Identity, Association, Synthetic, etc.) and anaphoric expression types (Pronouns, Nouns, Adjectival pointers).
- Handling Special Cases: The scheme accommodates split antecedents, dual antecedents, chains of references, and uncertainty in antecedent identification.
Datasets
- Newly Introduced Dataset: A corpus of 100 French email threads, comprising 314 emails and approximately 7163 words. This corpus is annotated with anaphoric links and is made freely available for scientific exploitation.
- Comparison Datasets: The paper references and compares its methodology to existing corpora like ANCOR, ARRAU, and OntoNotes, highlighting differences in markable delimitation, annotation scope, and handling of specific phenomena.
Key Insights and Ideas
- Email Specificities: Emails have unique structural and stylistic features (e.g., metadata like sender/recipient, segmented message structure) that significantly impact anaphora resolution, making it more akin to cross-document coreference than within-document coreference.
- Pronominal Anaphora Dominance: Pronominal anaphora is particularly frequent in emails, often with antecedents found in the email's metadata (e.g., sender/recipient pronouns).
- Challenges in Anonymization: Maintaining referential consistency during anonymization is crucial, especially for coreferent entities with multiple mentions.
- Need for Genre-Specific Resources: Existing French corpora, while valuable, do not adequately cover the email genre, necessitating the creation of specialized datasets like the one presented.
Key Results
- The corpus contains 1856 annotations across six relation types, with Identity being the most represented (1367 occurrences), followed by Association (252) and Synthetic (114).
- Pronominal anaphora constitutes a high percentage (74.8%) of the annotated links in their corpus, significantly higher than in the ANCOR corpus (41.1%).
- The annotation process, after refinement through agreement studies, achieved a good agreement score (Cohen's Kappa and Fleiss' Kappa >= 0.70), indicating the robustness of their guidelines and typology.
- The study provides a quantitative overview of anaphora distribution in French emails, offering insights into the most common and challenging aspects for future NLP system development.
Limitations and Future Work
- Dataset Size: The authors acknowledge that their dataset is relatively small (100 threads), which they consider a starting point.
- Future Work: The authors plan to use this corpus to tackle anaphora resolution directly, evaluating various machine learning and symbolic systems. They aim to identify the most significant challenges and frequently encountered anaphora types in emails to focus future research efforts. They also intend to explore the complexities of referential identity annotation between named entities across email threads and the ambiguity arising from first and second-person pronouns referring to speakers and organizations.
Summary updated