TL;DR
This paper tackles the challenge of training dependency parsers when multiple annotated treebanks exist but use different linguistic conventions. The authors propose two simple yet effective methods: "shared features" and "guided parsing," which allow for leveraging data from multiple, incompatible treebanks to improve parser performance, especially when training data is scarce.
What Problem Does This Paper Address?
Building high-quality statistical parsers relies heavily on large amounts of annotated training data (treebanks). However, the field of linguistics has diverse theoretical models, leading to multiple treebanks for the same language that are annotated with different conventions. This incompatibility makes it difficult to simply combine these treebanks for training, as a parser trained on one might perform poorly on data annotated with another. This paper addresses the problem of how to effectively utilize multiple, incompatible treebanks to train a single, more robust dependency parser, thereby overcoming the high cost of annotation and the scarcity of data in many existing treebanks.
Methods
The paper introduces two primary methods for adapting parsers to incompatible treebanks:
- Shared Feature Representation: This method treats the problem as a "multiview learning" scenario. It involves creating a feature representation where parts are shared across different treebanks, while other parts are specific to each. The idea is that the learning algorithm can automatically learn which linguistic properties are common and which are unique to each annotation style. This is implemented by augmenting the feature vectors of each treebank with a shared feature representation, effectively creating a single, unified feature space.
- Guided Parsing: This approach uses a pipeline where a parser trained on one treebank (the "guide" parser) provides features to a second parser being trained on another treebank. This is inspired by parser combination techniques. Instead of just using simple features like word adjacency, the authors propose using "path-based" features that encode the relationship between words in the parse tree, allowing the second parser to leverage structural information from the first.
These two methods are orthogonal and can also be combined for potentially even better performance.
Datasets
The authors evaluate their methods on eight treebanks across four languages:
- German: Tiger and TüBa-D/Z
- Swedish: Talbanken05 and Syntag
- Italian: ISST and TUT
- English: Two different conversions of the Penn Treebank (referred to as WSJ Part 1 and WSJ Part 2), derived using different settings of a constituent-to-dependency conversion tool.
The paper notes that for Italian and Swedish, they retokenized the treebanks to ensure compatibility. For German and English, they used existing dependency versions or converted constituent treebanks. They also ensured consistent part-of-speech tag sets by applying an automatic tagger.
Key Insights and Ideas
- Incompatibility is a real problem: Simply concatenating incompatible treebanks leads to a parser that performs poorly on all of them.
- Shared features are great for small/hard treebanks: The "shared features" method is particularly effective when one of the training treebanks is very small or when the annotation style is inherently harder to parse.
- Guided parsing shines with mid-sized sets: The "guided parsing" method shows its strength when training set sizes are moderate, especially when the guide parser's annotation style is easier to parse.
- Combining methods is often best: The combined approach, using both shared features and guided parsing, consistently yields the best results across most scenarios and training set sizes.
- Harder treebanks benefit more: The adaptation methods provide the largest improvements on treebanks that are more difficult to parse due to their annotation style or complexity.
- Annotation style matters: The effectiveness of guided parsing can be influenced by how "hard" the guide parser's annotation style is.
Key Results
- Significant improvements: All adaptation methods (shared features, guided parsing, and combined) showed statistically significant improvements over a baseline parser trained on a single treebank for all eight treebanks tested.
- German: The combined method improved Tiger treebank parsing accuracy by 0.7 points (a 6% error reduction).
- English: The combined method achieved a notable 1.2-point improvement for the WSJ Part 2 parser, which uses a more semantically oriented annotation style. This was partly attributed to a significant improvement in handling conjunctions.
- Training set size impact: Relative error reductions were much stronger for small and medium-sized training sets compared to using full training sets. Shared features excelled with very small sets, while guided parsing was better for mid-sized sets.
- Combined method dominance: The combined method consistently outperformed the other two methods individually in most cases and achieved significant improvements for all eight treebanks when using full training sets.
Limitations and Future Work
- Generalizability: While the methods are presented for unlabeled dependency parsing, the authors believe they can be generalized to other parsing approaches (e.g., transition-based, labeled parsing, constituent parsing).
- Complexity: The paper focuses on "simple" adaptation methods. Future work could explore more complex approaches like quasi-synchronous grammars or automatic treebank transformation.
- Controlled setting: The English experiments were conducted in a more controlled setting, with no domain differences, which might have contributed to clearer results. Future work could explore how domain differences interact with these adaptation methods.
Summary updated