TL;DR
This paper introduces qxoRef 1.0, the first-ever coreference corpus for Conchucos Quechua, a low-resource indigenous language. They also present a baseline mention-pair coreference resolution system trained on this corpus, demonstrating that even with limited linguistic resources, progress can be made, though syntactic parsing remains a crucial missing piece for robust performance.
What Problem Does This Paper Address?
Coreference resolution, the task of identifying and grouping mentions (phrases) that refer to the same real-world entity, is a fundamental NLP problem. While significant progress has been made in high-resource languages like English, low-resource languages, particularly those with complex morphology like Quechua, remain underserved. Existing coreference resolution systems and corpora are often designed for morphologically simpler languages, and the lack of annotated data and linguistic tools for Quechua presents a significant barrier. This paper aims to bridge this gap by creating a new resource and a baseline system for Conchucos Quechua, a variety of Quechua spoken in the Peruvian Highlands.
Methods
The authors introduce qxoRef 1.0, a coreference corpus built from transcribed stories. They then develop a baseline mention-pair coreference resolution system. This approach involves training a binary classifier on pairs of mentions to predict whether they are coreferential. The system utilizes a random forest classifier, a common choice for its interpretability. They explore three different heuristics for creating training data, focusing on how to balance positive and negative coreference instances. Features used for classification include string-based (e.g., Levenshtein distance, longest common substring), grammatical (e.g., plurality, mention type), and discourse features (e.g., sentence distance, speaker).
Datasets
The primary dataset introduced is qxoRef 1.0, a new coreference corpus for Conchucos Quechua. It consists of twelve documents, which are transcriptions of stories told by native speakers. The data was originally collected for prosody and phonology research and has been adapted for coreference annotation. The corpus is made freely available under a CC-BY-NC-SA 4.0 license. The paper also references existing coreference corpora for other pro-drop languages (Czech, Spanish, Catalan) for comparative context.
Key Insights and Ideas
- The first of its kind:
qxoRef 1.0is a significant contribution as the first coreference corpus for any Quechuan language, opening doors for research in this under-resourced language family. - Morphological complexity is a challenge: Quechua's agglutinative nature means words are morphologically complex. While this can be leveraged, it also means standard NLP tools designed for simpler languages may not directly apply.
- Pro-drop languages and null arguments: Quechua is a pro-drop language, meaning subjects and objects can be omitted. A major limitation of
qxoRef 1.0is the inability to annotate these null arguments as mentions due to the lack of a suitable syntactic parser. - Syntactic parsing is crucial: The paper strongly emphasizes that the performance of the coreference system is hindered by the absence of syntactic parsing. This missing step prevents the annotation of null arguments and limits the richness of grammatical features.
- String similarity is a strong, but limited, signal: For Conchucos Quechua, the most important features for coreference resolution were string-based. This is attributed to the high proportion of nominal mentions and the lack of null arguments, making coreferential mentions often lexically similar. However, this also leads to poor performance on mentions with divergent surface forms.
Key Results
The baseline mention-pair coreference resolution system achieved an average F1 score of 68.51% across three evaluation metrics (MUC, B³, CEAF). Notably, the heuristic proposed by Ng and Cardie (2002), which excluded pronoun-antecedent pairs, performed best despite producing the smallest training set. This suggests that focusing on noun-noun coreference, which is more prevalent in the data, was more effective. The error analysis revealed that the system struggled with coreferential mentions that had significantly different surface forms and with demonstrative pronouns.
Limitations and Future Work
The authors acknowledge several limitations:
- Lack of null argument annotation: Due to the absence of a syntactic parser for Conchucos Quechua, null arguments, a common feature of pro-drop languages, could not be annotated as mentions.
- Single annotator bias: The corpus was annotated by a single author, potentially leading to idiosyncratic decisions. Future work should involve multiple annotators to ensure reliability and compute inter-annotator agreement.
- Limited feature set: The lack of syntactic and semantic features (like word embeddings) due to resource constraints limits the system's ability to capture deeper linguistic relationships.
Future work directions include:
- Improving the corpus: Adding more annotators and performing syntactic parsing to enable null argument annotation.
- Developing syntactic parsers: Creating parsers for Conchucos Quechua is a critical step for advancing coreference resolution and other NLP tasks.
- Leveraging embeddings: Exploring sub-word embeddings for related Quechua varieties (like Ancash Quechua) to enrich feature representations.
- Focusing on foundational tools: The paper advocates for prioritizing the development of basic NLP tools for low-resource languages, as these are essential for tackling more complex tasks like coreference resolution.
Summary updated