TL;DR
This paper introduces a novel method for evaluating inter-annotator agreement (IAA) in treebank annotation by leveraging derivation trees. Instead of just flagging disagreements, their system precisely pinpoints inconsistencies and groups them by structural type, offering a more insightful and actionable evaluation for improving annotation quality.
What Problem Does This Paper Address?
Creating high-quality treebanks, which represent the syntactic structure of sentences, is crucial for linguistic research and natural language processing. However, the complexity of annotation makes it prone to errors and inconsistencies between human annotators. Traditional methods for evaluating inter-annotator agreement (IAA) often highlight that disagreements exist but don't effectively explain why or how they occur. This makes it difficult for annotators to understand where to improve their skills or for guideline developers to refine instructions. The paper aims to bridge this gap by providing a more informative IAA evaluation that goes beyond simple agreement scores to reveal the nature and structure of annotation disagreements.
Methods
The core of the paper's contribution is the adaptation of a system (KBM system) originally developed for automatic discovery of treebank annotation inconsistencies to the specific task of IAA evaluation. The KBM system's novelty lies in its use of Tree Adjoining Grammar (TAG) principles to decompose full syntactic trees into smaller units called "elementary trees" (e-trees). These e-trees are then represented in a "derivation tree," which captures how they combine to form the full sentence structure.
For IAA evaluation, the system focuses on comparing "nuclei" – specific sequences of words identified as potential sites of inconsistency. Instead of comparing entire trees, it compares the derivation tree fragments that cover these nuclei. This allows for a more granular comparison, abstracting away from irrelevant parts of the tree. The system then categorizes inconsistencies based on the structural differences in these derivation tree fragments, providing a "structural type" for each disagreement.
Datasets
The authors evaluate their system on two corpora:
- A corpus of English web text (a pre-release subset of the English Web Treebank, dually annotated for annotator training).
- A corpus of Modern British English (a supplemental section of the Penn Parsed Corpus of Modern British English).
Key Insights and Ideas
- Pinpointing vs. Flagging: The system's ability to precisely pinpoint inconsistencies, rather than just flagging them, is a significant improvement. This allows annotators to focus on specific problematic areas.
- Structural Grouping: Grouping inconsistencies by their structural type is a critical insight. This reveals patterns in annotator behavior and potential ambiguities in the annotation guidelines, offering a deeper understanding of why disagreements occur.
- Abstraction from Irrelevant Details: By using derivation tree fragments, the system effectively ignores superficial differences in the broader sentence structure, focusing only on the relevant parts of the annotation for a given nucleus. This prevents the same underlying annotation issue from being reported multiple times due to minor variations elsewhere in the tree.
- High Precision: The evaluation demonstrates a very high precision for the reported inconsistencies, with most "structural problems" being actual inconsistencies that were simply miscategorized due to the system's focus on structural differences.
Key Results
The evaluation on both corpora showed that the KBM system successfully identified inconsistencies.
- In the English web text corpus, 129 structural inconsistency types were found, involving 130 nuclei. The system achieved high precision, with only a small percentage of reported inconsistencies being false positives or miscategorized.
- The Modern British English corpus, being larger, yielded more substantial cases of grouped nuclei, with the top two inconsistency types having 88 and 37 nuclei respectively.
- The system effectively grouped nuclei with similar annotation inconsistencies, even when the words themselves differed (e.g., "The word renaissance" and "The term renaissance"). This highlights the system's strength in identifying structural annotation patterns.
Limitations and Future Work
The authors acknowledge a few limitations and suggest future directions:
- "External Check" for Certain Inconsistencies: A specific class of inconsistencies, which KBM cannot pinpoint precisely, requires adopting an "external check" mechanism previously proposed by Kulick et al. (2012).
- Further Abstraction of Inconsistency Types: The current abstraction of inconsistency types could be taken further, for example, by examining specific types related to PP attachment or particular function tags.
- Application to Parser Evaluation: A major area for future work is applying this system to parser evaluation, which shares conceptual similarities with IAA evaluation. The authors note the connection to work on dependency structures and believe their TAG-based phrase structure analysis approach can be valuable here.
- Refinement of the System: The current work focuses on annotator input for IAA. Future work may involve applying the refined system to parser evaluation.
Summary updated