TL;DR
Ensuring consistent annotation across diverse linguistic corpora is crucial for NLP. This paper adapts and evaluates a method for detecting inconsistencies in Universal Dependencies (UD) treebanks, finding it effective and developing a tool to streamline manual validation. They also explore using lemmas instead of wordforms to improve recall, with mixed success across languages.
What Problem Does This Paper Address?
The Universal Dependencies (UD) project aims to create a consistent framework for annotating dependency structures across many languages. With a rapidly growing number of UD corpora, ensuring that similar linguistic phenomena are annotated consistently within and across these corpora is a significant challenge. While methods exist for detecting inconsistencies in part-of-speech tagging, robust and clear methods for dependency annotations are less established. This paper addresses the need for a reliable way to assess and improve the consistency of dependency annotations in UD treebanks.
Methods
The core of the paper's methodology is an adaptation of the "variation nuclei" method proposed by Boyd et al. (2008). This method identifies pairs of words that appear in similar contexts but are linked by different dependency relations. The authors implement two heuristics:
- "Non-fringe" heuristic: Requires the words in the nucleus to share the same left and right context (one word on each side).
- "Dependency context" heuristic: A more stringent version that additionally requires the governors of the dependency pairs to have the same incoming dependency relation.
They extend this by experimenting with using lemmas instead of wordforms to potentially increase recall by reducing data sparsity. They also explore a delexicalized approach using only morphological features. To facilitate manual validation of flagged inconsistencies, they developed a web-based annotation tool that visualizes the differing annotations for easy review.
Datasets
The study evaluates its methods on the UD v2 corpora for three languages:
- English
- French
- Finnish
These languages were chosen for their varying degrees of morphological richness. The authors combined the train, development, and test sets for each language into a single corpus for analysis. They also utilized parsebank data from the CoNLL'17 Shared Task for a delexicalized approach.
Key Insights and Ideas
- The Boyd et al. method is a strong starting point: The adapted method effectively flags potential annotation errors within a given corpus, demonstrating its utility for improving data quality.
- Lemmas can boost recall, but with caveats: Using lemmas instead of wordforms shows promise for less morphologically rich languages like English and French, increasing the number of detected inconsistencies. However, for morphologically complex languages like Finnish, this approach leads to a significant drop in precision due to the loss of inflectional information.
- Morphology is key for Finnish: For Finnish, the delexicalized approach, focusing on morphological features, surprisingly outperforms the lemma-based method in both precision and recall, highlighting the critical role of morphology in disambiguating syntactic functions.
- A dedicated tool is essential: The developed annotation tool significantly streamlines the manual validation process, making the detection of inconsistencies more practical.
- Parsebank data offers broader context: Leveraging large, automatically parsed datasets (parsebanks) can identify more inconsistencies than relying solely on manually annotated corpora, though precision may decrease.
Key Results
- The "non-fringe" heuristic using lemmas achieved precisions of 62% for English, 65% for French, and 19% for Finnish.
- Using wordforms instead of lemmas significantly improved precision for Finnish (from 19% to 72%) but offered only moderate gains for English and French, while reducing recall.
- The delexicalized approach for Finnish yielded a precision of 25% for 353 pairs, outperforming the lemma-based approach's 19% precision for 117 pairs.
- The annotation tool, when tested on 100 randomly sampled pairs per language, identified a significant number of erroneous trees, with precisions ranging from 16% to 57% depending on the language and heuristic.
- Using parsebank data, the method identified a substantial number of erroneous trees, but a significant portion (60-70%) of these were not flagged by the Boyd et al. method when operating solely within the manually annotated corpus.
Limitations and Future Work
- Morphological richness is a challenge: The lemma-based approach struggles with highly morphologically rich languages like Finnish, where inflections carry crucial syntactic information.
- Strict cut-offs in parsebank data: The limited increase in recall when using parsebank data might be due to a too strict cut-off for the minimum number of parsebank instances required to flag an inconsistency.
- Noise in automated parses: The quality of automated parses in web-crawled data can introduce noise, affecting the reliability of the parsebank-based method.
- Future Work: The authors suggest expanding the effort to the larger UD community by implementing user management for aggregated annotations across multiple annotators. They also highlight the need to address ensuring similar constructions across different corpora and languages are given the same analysis, moving beyond within-corpus consistency.
Summary updated