TL;DR
This paper introduces CCGweb, an open-source graphical tool for annotating sentences with Combinatory Categorial Grammar (CCG) derivations. It also presents the first quadrilingual (English, German, Italian, Dutch) CCG treebank created through direct annotation, along with detailed guidelines, addressing the limitations of previous conversion-based approaches and the lack of accessible tools.
What Problem Does This Paper Address?
Combinatory Categorial Grammar (CCG) is a powerful formalism for natural language processing, particularly in semantic parsing. While CCG treebanks exist for various languages, they have largely been created by automatically converting existing phrase structure or dependency treebanks. This conversion process is inherently error-prone and doesn't always adhere strictly to CCG principles. Furthermore, existing CCG annotation tools are often not open-source, lack detailed syntactic annotation guidelines, and offer limited control over the annotation process, making it difficult to scale and reuse these resources. This paper aims to bridge this gap by providing an accessible, user-friendly tool and a high-quality, directly annotated multilingual treebank.
Methods
The core of the paper is the CCGweb annotation tool. It's a web-based application implemented in Python, PHP, and JavaScript, designed for ease of deployment. The tool employs a "dynamic annotation" approach, also known as human-aided machine annotation. This means:
- Initial Parsing: A statistical CCG parser (EasyCCG) generates an initial derivation for each sentence.
- Annotator Constraints: Annotators interact with this initial parse using two main types of constraints:
- Lexical Category Constraints: Annotators can directly modify the lexical categories assigned to words. This is crucial because CCG is a lexicalized formalism, and many parsing decisions stem from word categories.
- Span Constraints: Annotators can select a span of tokens and constrain it to form a constituent in the final parse. This is essential for handling phenomena like attachment ambiguities that cannot be fully resolved by lexical categories alone.
- Re-parsing: After annotators apply constraints, the sentence is re-parsed, and the process is repeated until the annotator is satisfied.
- WYSIWYG Interface: Derivations are rendered graphically, mirroring common representations in CCG literature, allowing annotators to interact directly with the visual structure.
- Adjudication Support: A special "judge" user role allows for comparing multiple annotators' versions of a sentence, highlighting disagreements (struck-through categories, dashed spans) to facilitate the creation of a final, authoritative version.
Datasets
The paper introduces a new Quadrilingual Pilot CCG Treebank. This treebank consists of:
- 110 short sentences from the Tatoeba corpus, each in four languages: English, German, Italian, and Dutch.
- 4x100 adjudicated sentences: These are the core of the new resource, created through a rigorous annotation and adjudication process.
- > 10K single-annotator fully corrected sentences: These are derived from the Parallel Meaning Bank (PMB) and have been manually corrected by a single annotator.
- > 82K single-annotator partially corrected sentences: Also derived from the PMB, these sentences have had at least one change made by a single annotator to the automatically generated derivation.
The new treebank is distinguished by its direct annotation and adherence to explicit syntactic CCG guidelines, unlike previous treebanks that were often converted from other formalisms.
Key Insights and Ideas
- Direct Annotation is Crucial for Quality: The paper strongly advocates for direct annotation over conversion methods, highlighting that direct annotation leads to a more faithful representation of CCG principles and reduces errors.
- Multilingual CCG is Feasible: The successful creation of a quadrilingual treebank demonstrates the viability of extending CCG annotation to multiple languages with consistent guidelines.
- User-Friendly Tools Drive Progress: The development of an open-source, graphical tool like CCGweb is essential for making CCG annotation accessible, scalable, and reusable for research and teaching.
- Constraints are Powerful: The combination of lexical category and span constraints provides annotators with fine-grained control, enabling them to resolve ambiguities and ensure derivations conform to CCG theory.
- Adjudication is Key for Consensus: A structured adjudication process is vital for resolving disagreements between annotators and producing a high-quality, reliable dataset.
Key Results
The paper presents statistics on the annotation process, including the number of constraints applied per annotator and the distribution of disagreement classes (from none to intricate). While specific accuracy metrics for the parser are not the primary focus, the paper emphasizes the creation of a high-quality, adjudicated dataset. The release includes:
- The CCGweb annotation tool.
- A detailed annotation manual.
- The adjudicated quadrilingual pilot CCG treebank (4x100 sentences).
- Additional data from the Parallel Meaning Bank (over 10K fully corrected and over 82K partially corrected sentences).
Limitations and Future Work
Limitations:
- The pilot treebank is relatively small (4x100 adjudicated sentences), though it is supplemented by larger, less rigorously adjudicated datasets.
- The initial annotation guidelines were adapted from CCGrebank, and some specific instructions had to be developed during the process.
Future Work:
- More Extensive Direct Annotation: The authors envision expanding the direct annotation of multilingual data with CCG derivations.
- Evaluation of Parsers: The new resource will be used to evaluate unsupervised and distantly supervised CCG parsers.
- Teaching Tool: The CCGweb tool could be used as an interactive aid for teaching CCG.
- Integration of Other CCG Parsers: The tool's architecture allows for plugging in other CCG parsers with modifications.
Summary updated