Pro3Gres Parser in the CoNLL Domain Adaptation Shared Task

Gerold Schneider, Kaarel Kaljurand, Fabio Rinaldi, Tobias KuhnEMNLP 2007

TL;DR

This paper introduces Pro3Gres, a deep-syntactic dependency parser that combines a hand-written grammar with probabilistic disambiguation. While achieving average results in the CoNLL domain adaptation task, the authors highlight significant challenges in mapping their parser's output to the shared task's specific dependency representation, particularly for conjunctions and punctuation.

What Problem Does This Paper Address?

Parsing natural language into a structured dependency representation is a fundamental task in Natural Language Processing. However, achieving high accuracy across diverse domains and adapting existing parsers to new domains (like biomedical text or chemistry) remains a significant challenge. This paper addresses the problem of domain adaptation for dependency parsing, specifically focusing on how a parser built with a linguistic competence grammar and probabilistic disambiguation performs when applied to a new domain and how its output can be mapped to a standardized, shared-task-specific representation. The authors also explore the trade-offs between hand-crafted grammars and data-driven approaches.

Methods

Pro3Gres is a deep-syntactic dependency parser that employs a hybrid approach. It combines a hand-written competence grammar, which is described as being "deep-syntactic like a formal grammar-based parser" and close to LFG f-structure, with probabilistic performance disambiguation trained on the Penn Treebank. The parser uses tagger and chunker pre-processors and a post-processor graph converter to handle long-distance dependencies. A key aspect of its design policy is to integrate shallow approaches and aggressive pruning to maintain efficiency without compromising linguistic adequacy. The grammar itself is relatively small (around 1000 rules) and was developed from scratch in about a person-month. For domain adaptation, the authors explored two strategies: learning relational nouns from unlabeled domain texts and augmenting dependency probabilities based on discovered collocations.

Datasets

  • Penn Treebank: Used for training the probabilistic disambiguation component of Pro3Gres.
  • British National Corpus (BNC): Used for testing the parser's speed and general applicability.
  • Biomedical literature: Used for previous applications and domain adaptation experiments.
  • CoNLL Domain Adaptation Shared Task datasets: Used for evaluation in this paper, specifically for the biomedical domain and the CHILDES domain.

Key Insights and Ideas

  • The challenge of representation mapping: The paper strongly emphasizes that mapping between different dependency representations is a major hurdle, significantly impacting performance. This is particularly true when the target representation (CoNLL's) differs fundamentally from the parser's native representation (LFG-like f-structure), especially concerning how auxiliaries, prepositions, and conjunctions are treated.
  • Domain-independent grammar, domain-dependent disambiguation: The authors suggest that their hand-written competence grammar is relatively domain-independent, as evidenced by its decent performance on the CHILDES domain without specific adaptation. However, the probabilistic disambiguation component is inherently domain-dependent.
  • Hybrid approach benefits: The combination of a hand-written grammar and probabilistic disambiguation allows for exploring interactions between lexicon and grammar, aiming for both linguistic adequacy and performance.
  • Efficiency of hand-crafted grammars: The grammar's small size and rapid development time (one person-month) suggest that hand-crafted grammars can be a viable and efficient approach for building parsers, especially when combined with statistical methods.

Key Results

  • Average performance on CoNLL: The parser achieved average results in the CoNLL domain adaptation track. Specific scores reported include:
  • Labeled attachment score: 63.01%
    • Unlabeled attachment score: 66.53%
    • Label accuracy score: 76.62%
  • Performance dip due to mapping: These results are noted as being about 10% below what the authors typically achieve with their own dependency representation.
  • Specific relation performance: Relations like SBJ, NMOD, and ROOT performed better, while ADV, AMOD, PRN, and P showed poorer performance, indicating specific challenges in mapping these relation types. Performance on COORD was particularly disappointing.
  • CHILDES domain performance: The parser achieved slightly above-average results on the CHILDES domain without any adaptation (unlabeled attachment score: 60.27%).

Limitations and Future Work

  • Mapping issues: The most significant limitation identified is the poor mapping to the CoNLL dependency representation, especially for conjunctions and punctuation. This led to a substantial portion of dependencies remaining unmapped.
  • Treatment of specific linguistic phenomena: The paper notes difficulties in handling parentheticals (PRN) and the potential misinterpretation of ADV and AMOD roles.
  • Future work: The authors plan to adapt the parser to further domains and incorporate more unsupervised learning methods. They also suggest that using a dependency representation that maximally abstracts away from form to function could mitigate mapping problems.

Summary updated

BibTeX
@inproceedings{schneider2007pro,
  author = {Gerold Schneider and Kaarel Kaljurand and Fabio Rinaldi and Tobias Kuhn},
  title = {{Pro3Gres Parser in the CoNLL Domain Adaptation Shared Task}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2007},
}
Figure