TL;DR
This paper introduces a hybrid approach to dependency parsing by combining a machine learning (ML) parser (LingPars) with a rule-based parser (EngGram). This hybridization significantly improves domain independence and robustness, outperforming the purely ML baseline and achieving state-of-the-art results on challenging domain adaptation tasks.
What Problem Does This Paper Address?
Machine learning (ML) models for natural language processing, including dependency parsers, often struggle with domain adaptation. When trained on one type of text (e.g., news articles), they perform poorly on text from different domains (e.g., spoken language or biomedical texts). This is because the statistical patterns learned from the training data don't generalize well to the new domain. Traditional rule-based parsers, while potentially more robust, are expensive to develop and maintain, and their performance can be brittle. This paper aims to bridge this gap by exploring how to leverage the strengths of both approaches – the broad coverage and adaptability of ML with the linguistic precision and robustness of rule-based systems – to create a more domain-independent dependency parser.
Methods
The core of this work is a hybrid system that combines a machine learning dependency parser, LingPars, with a rule-based Constraint Grammar (CG) parser, EngGram.
- LingPars: This is a language-independent ML parser that was originally developed for the CoNLL-X 2006 shared task. It prioritizes syntactic function over form and incorporates features like BARRIER conditions and tag chains. For the CoNLL-X 2007 task, it was modified to incorporate directedness markers, which, while improving accuracy, increased memory requirements.
- EngGram: This is a rule-based CG parser. For this project, a new English version was developed by seeding it with Danish rules from the well-established DanGram parser. This rule-based system provides linguistic insights and analyses.
- Hybridization: The key innovation is integrating EngGram's output into LingPars. This involves several compatibility steps:
- Input Adaptation: PTB tokenization and word classes (PoS) were fed into EngGram, bypassing its own modules.
- Output Adaptation: EngGram's CG function categories and attachment conventions were adapted to match the PTB (Penn Treebank) conventions. This included handling specific tokenization issues (like genitive 's) and mapping CG function categories to PTB ones.
- Integration: EngGram's analysis was used to calibrate LingPars's dependency and function label rankings. This meant assigning extra weights to readings supported by the rule-based analysis, effectively using EngGram as an "arbiter" to guide the ML parser.
Datasets
The paper focuses on the domain adaptation task (Task 2) of the CoNLL-X 2007 Shared Task.
- Training Data: Wall Street Journal (WSJ) data from the Penn Treebank (PTB).
- Test Data:
- pchemtb corpus: Biomedical data.
- CHILDES corpus: Spoken language data.
Key Insights and Ideas
- Synergy of ML and Rules: Combining rule-based linguistic knowledge with statistical ML models can overcome the limitations of each individual approach, leading to improved robustness and domain independence.
- Rule-Based Parsers as "Arbiters": Rule-based systems can provide valuable, high-precision information that helps disambiguate and correct errors in ML parsers, especially for complex linguistic phenomena like long-distance dependencies.
- Compatibility is Key: Significant effort is required to make different parsing systems (rule-based and ML) compatible, especially regarding tokenization, part-of-speech tags, and dependency labels.
- Treebank Quality Matters: The accuracy of the human-annotated treebanks themselves can be a bottleneck. In cases where the rule-based system's PoS tagging is more accurate than the treebank's, it's beneficial to let the rule-based system dictate the tags.
- Domain Adaptation Benefits: The hybrid system showed significantly better performance on out-of-domain data compared to the baseline ML parser, demonstrating its enhanced robustness.
Key Results
The hybrid LingPars system achieved impressive results:
- Best-Scoring System: It was the best-scoring system in the "open" section of both domain adaptation tasks in CoNLL-X 2007.
- Outperforming Baseline: It outperformed its probabilistic core system (LingPars without EngGram) on all scores.
- Specific Improvements:
- CHILDES: Achieved a 6.57 LAS percentage point improvement.
- pchemtb: Achieved a 3.42 LAS percentage point improvement.
- Domain Robustness: The hybrid system was more robust to domain shifts than the purely ML approach. For the pchemtb data, its performance on out-of-domain data was only marginally lower than on the training domain, and in fact, higher for attachment accuracy.
- Comparison to Closed Systems: It surpassed the top "closed class" system in the CHILDES domain.
Limitations and Future Work
- Compatibility Challenges: The paper highlights the significant effort required for input/output adaptation between the CG and PTB/CoNLL formats. This includes handling tokenization differences (e.g., genitive 's), mapping function categories, and dealing with different levels of granularity in labels.
- MWE Handling: No reliable way was found to predict PTB's multi-word expression (MWE) attachment behavior from CG dependency output.
- Root Attachment Ambiguity: Even with CG guidance, the probabilistic system struggled with global syntactic problems like root attachment ambiguity, requiring fixed CG-supported attachments in the first pass.
- Future Work: The authors suggest that harmonizing the CG and ML systems might be better achieved if the training data were an original dependency treebank or at least linguistically revised at the dependency level. Making constituent-to-dependency conversion principles public earlier could also aid annotation transfer.
Summary updated