TL;DR
This paper adapts a state-of-the-art neural coreference resolution system (E2EC) to better handle oral French corpora, which often present challenges like disfluencies and a higher prevalence of singleton mentions. By introducing explicit mention detection and limiting context to immediate surroundings, the adapted system achieves competitive results on the ANCOR corpus, demonstrating the applicability of knowledge-poor, end-to-end neural models to new domains and languages.
What Problem Does This Paper Address?
Coreference resolution, the task of identifying mentions in text that refer to the same real-world entity, has seen significant advancements with neural network models. However, these models, particularly those performing well on benchmarks like OntoNotes (e.g., the E2EC system), are often trained on specific types of data (written text, formal language) and may not generalize well to other domains or languages. Oral corpora, like the ANCOR dataset of transcribed French speech, introduce unique challenges:
- Disfluencies and Noisy Input: Spoken language is inherently less structured than written text, featuring hesitations, repetitions, and incomplete sentences.
- Singleton Mentions: Oral corpora often include a higher proportion of mentions that refer to entities mentioned only once, which can complicate evaluation and model training if not handled properly.
- Computational Demands: Existing end-to-end models can be computationally expensive, requiring entire documents to be processed at once, making training on large oral corpora difficult.
- Domain Adaptation: The specific characteristics of oral French might require adaptations to models trained on English or other languages.
This paper aims to bridge this gap by adapting a successful neural coreference resolution system to effectively handle the specific characteristics of oral French, providing a new baseline and evaluation for this challenging domain.
Methods
The core of this work is the adaptation of the End-to-End Neural Coreference Resolution (E2EC) system, originally developed by K. Lee et al. The authors introduce two main modifications to make E2EC more suitable for oral corpora like ANCOR:
- Explicit Mention Detection: Unlike the original E2EC, which implicitly handles mentions as part of span detection, this adapted system includes a distinct mention detection layer. This layer explicitly classifies text spans as either "None" (not a mention) or a specific mention type (e.g., "NP" for noun phrases, or distinct classes for noun phrases and pronouns). This is crucial for handling corpora with singleton mentions and for enabling evaluation of mention detection capabilities independently.
- Limited Lexical Context: To address the computational demands of processing entire documents and the potential noise from distant parts of oral transcripts, the system restricts the input representation to the immediate left and right context of a text span. This is achieved by using a bidirectional LSTM and a self-attention mechanism on a window of words around the span.
Additionally, the authors incorporate:
- Word Representations: A combination of pretrained word embeddings (FastText) and character-level embeddings derived from a bidirectional GRU to handle disfluencies and unknown words.
- Span Embeddings: A combination of recurrent (bidirectional LSTM) and self-attentional mechanisms to create dense representations of text spans, incorporating both the span's internal states and its immediate context.
- Antecedent Scoring: A coarse-to-fine inference mechanism similar to the original E2EC, but with a specific "mention-new" score to account for the higher number of non-anaphoric mentions in ANCOR.
Datasets
- ANCOR: This is the primary corpus used for training and evaluation. ANCOR is described as a large, publicly available corpus of transcribed oral French, containing approximately 418,000 words. It includes coreference annotations and morphosyntactic annotations for noun phrases and pronouns, importantly including singleton mentions.
- CoNLL-2012/OntoNotes: While not directly used for training or evaluation in this paper, the authors reference the CoNLL-2012 shared task and OntoNotes dataset as the benchmark on which the original E2EC system achieved state-of-the-art performance. This provides a point of comparison for the general approach.
Key Insights and Ideas
- Generalizability of Knowledge-Poor Models: The paper demonstrates that end-to-end neural architectures, which rely less on hand-crafted linguistic features, can be effectively adapted to new domains and languages like oral French, even with limited domain-specific knowledge.
- Importance of Explicit Mention Detection: For corpora with a high prevalence of singleton mentions, an explicit mention detection step is beneficial. It allows for a more nuanced evaluation and can help the model better distinguish between actual mentions and other text spans.
- Context Window Effectiveness: Limiting the context to immediate surroundings, rather than processing entire documents, proves to be a viable strategy for managing computational resources and mitigating the impact of noise in oral transcripts. This suggests that for certain tasks, local context is sufficient.
- Oral French Challenges: The work highlights the specific difficulties posed by oral French, such as disfluencies and a higher rate of singleton mentions, underscoring the need for specialized adaptations of existing NLP models.
Key Results
- Mention Detection: The adapted system achieved a mention detection F-score of 85.87 on the ANCOR corpus, outperforming the baseline of Grobol et al. (2017) (65.72) and showing competitive results against Godbert and Favre (2017) (88.94). The authors note that their precision could be improved.
- Coreference Resolution: The system achieved competitive coreference resolution scores on ANCOR, with MUC (47.7), CEAFE (71.0), CONLL (72.8), and BLANC (72.0) metrics. These results are compared to previous works on ANCOR, such as Désoyer et al. (2015) and Godbert and Favre (2017). The authors emphasize that direct comparison is challenging due to differing methodologies and datasets. They note that their system performs well despite not using gold features, unlike some previous approaches.
Limitations and Future Work
- Single Corpus Evaluation: The authors acknowledge that their system has only been tested on the ANCOR corpus, and its architecture is optimized for it. Further assessment on other comparable corpora (e.g., ARRAU, Polish Coreference Corpus, DEMOCRAT) is needed.
- Comparability to CoNLL-2012: While the system is inspired by E2EC, direct comparison on the CoNLL-2012 dataset is difficult due to the need for gold mentions and the handling of singleton mentions, which are treated differently. Future work could involve evaluating on CoNLL-2012 with "gold mention boundaries" for better comparability.
- Mention Detection Improvement: The authors note that the precision of their mention detection component has room for improvement.
- Formal Assessment of Mention-New Score: The specific "mention-new" score introduced to handle non-anaphoric mentions has not yet undergone a formal assessment of its impact.
- Future Directions: The paper suggests further testing on diverse corpora and potentially refining the architecture for even better performance on oral language.
Summary updated