TL;DR
This paper introduces a new approach to annotating English syntax using the comprehensive Cambridge Grammar of the English Language (CGEL) framework. By developing detailed guidelines and conducting an interannotator study, the authors demonstrate that CGEL can be effectively applied to real-world corpora, offering a unified and linguistically rich representation of syntactic structure.
What Problem Does This Paper Address?
Syntactic parsing and annotation are crucial for natural language processing (NLP) tasks, but existing annotation schemes often focus on either constituent structure (like the Penn Treebank) or dependency structure (like Universal Dependencies), leading to different representations of the same linguistic phenomena. The Cambridge Grammar of the English Language (CGEL) offers a highly detailed and unified description of English syntax, encompassing both constituent categories and their grammatical functions. However, CGEL was not originally designed for corpus annotation, and it was unclear whether its extensive descriptions could be practically applied to annotate large, naturally occurring datasets consistently. This paper investigates the feasibility of using CGEL as a framework for syntactic annotation, aiming to bridge the gap between linguistic theory and practical NLP applications.
Methods
The core of this work involves developing a set of annotation guidelines based on the CGEL framework. This process required resolving ambiguities and underspecifications within CGEL itself when applied to real-world sentences. The authors then created a small corpus, "CGELBank," by annotating sentences from various sources. To evaluate the consistency and practicality of their approach, they conducted an interannotator agreement study. This study involved two linguists independently annotating a set of sentences, followed by adjudication of disagreements. They employed Tree Edit Distance (TED) as their primary evaluation metric, which is more sensitive to hierarchical structure and unary nodes than traditional metrics like Parseval. They also developed a "validator" script to automatically check for structural errors in the annotated trees, which helped improve annotator consistency.
Datasets
The primary dataset used for annotation and evaluation is CGELBank, a newly created corpus of English sentences annotated using the CGEL framework. CGELBank was built using sentences from several sources:
- English Web Treebank (EWT): A portion of this existing treebank was used to create CGELBank annotations.
- Twitter: Sentences sampled from Twitter were used, representing naturally occurring, informal language.
- Pilot sets: Smaller sets of sentences were used for initial development and for the interannotator study.
The paper also references existing treebanks like the Penn Treebank (PTB) and Universal Dependencies (UD) for comparison and context.
Key Insights and Ideas
- CGEL as a Unified Framework: The paper strongly advocates for CGEL's potential as a unified syntactic annotation scheme, effectively integrating constituent categories and grammatical functions in a single representation. This contrasts with many existing schemes that prioritize one over the other.
- Practicality of CGEL: Despite CGEL's theoretical depth, the authors demonstrate that it is indeed feasible to apply its principles consistently to annotate real-world text, even for complex phenomena like gapping.
- Identifying Underspecifications: The process of creating annotation guidelines revealed minor points of underspecification within CGEL that needed to be addressed for consistent annotation, highlighting the iterative nature of bridging linguistic theory and corpus annotation.
- The Value of a Validator: The development of an automated validation script proved crucial in improving annotator consistency and identifying subtle structural errors, demonstrating the benefit of integrating computational tools into the annotation process.
- Tree Edit Distance for Hierarchical Evaluation: The use of Tree Edit Distance (TED) is highlighted as a more appropriate metric for evaluating constituency tree annotations, especially when dealing with unary nodes and potential differences in gap representation, compared to simpler metrics.
Key Results
The interannotator agreement study yielded strong results, with the flex metric (which considers category, function, and lexeme with partial credit) showing an F1 score of 93.9%. Even the stricter metric, which requires exact matches, achieved an F1 score of 91.6%. Agreement on gaps was also good at 87.2%. While the exact tree match (full-tree) score was lower at 18.0% (before adjudication), this is expected given the complexity of tree structures. The validator script significantly improved agreement scores, demonstrating its effectiveness. Qualitative findings pointed to challenges in annotating structured names, currency expressions, and the boundary between complements, modifiers, and supplements, leading to refined guidelines.
Limitations and Future Work
- Limited Corpus Size: The current CGELBank is relatively small (over 200 trees), and the authors acknowledge the need for expansion.
- Informal Nature of Some CGEL Accounts: The paper notes that CGEL's account of some phenomena, like gaps, is described as "informal" and required further specification for annotation.
- Future Work: The authors plan to expand CGELBank, augment it with automatic conversions from existing treebanks, and develop CGEL-style parsers. They also intend to revisit specific linguistic phenomena that proved challenging during annotation, such as partitive fronting.
Summary updated