On Different Approaches to Syntactic Analysis Into Bi-Lexical Dependencies An Empirical Comparison of Direct, PCFG-Based, and HPSG-Based Parsers

Angelina Ivanova, Stephan Oepen, Rebecca Dridan, Dan Flickinger, Lilja Øvrelid, Emanuele Lapponi

TL;DR

This paper empirically compares three different syntactic parsing approaches (data-driven, PCFG-based, and grammar-driven HPSG) for extracting bi-lexical dependencies. It finds that grammar-driven parsers offer superior accuracy and domain robustness, though this doesn't always translate to downstream task improvements.

What Problem Does This Paper Address?

Syntactic analysis, particularly extracting bi-lexical dependencies (direct head-argument relationships between words), is crucial for many Natural Language Processing (NLP) tasks. However, different parsing approaches exist, each with its own strengths and weaknesses. This paper aims to provide an empirical comparison of three distinct paradigms: direct data-driven dependency parsers, statistical phrase structure parsers (converted to dependencies), and hybrid grammar-driven HPSG parsers. The goal is to understand the trade-offs in accuracy, efficiency, and resilience to domain variation across these approaches, and to see how these intrinsic parsing improvements impact downstream applications.

Methods

The authors compare three main parsing approaches:

  • Direct Data-Driven Dependency Parser (B&N): A transition-based parser that directly outputs dependency structures.
  • PCFG-based Phrase Structure Parser (Berkeley): A statistical parser that generates phrase structure trees, which are then converted into dependencies.
  • Hybrid Grammar-Driven HPSG Parser (ERG): A parser based on Head-Driven Phrase Structure Grammar (HPSG), which produces deep syntactic analyses that are then converted into dependencies (DT scheme).

They also explore variations of the ERG parser, including one with pruning for efficiency (ERG_e). The evaluation focuses on both intrinsic metrics (accuracy, coverage, speed) and extrinsic evaluations on two downstream tasks: negation resolution and semantic dependency parsing.

Datasets

  • DeepBank: A treebank derived from the Wall Street Journal (WSJ) corpus, annotated with HPSG analyses. Used for training and tuning.
  • Redwoods Treebank: A collection of corpora from various domains and genres, also in HPSG format. Used for cross-domain testing.
  • Conan Doyle stories: Used for the negation resolution task.
  • WSJ Corpus: Used for in-domain testing.
  • SemEval 2014/2015 datasets: Used for semantic dependency parsing.

Key Insights and Ideas

  • Grammar-driven parsers excel in accuracy and domain robustness: The HPSG-based parser (ERG) consistently outperforms data-driven and PCFG-based parsers, especially when dealing with out-of-domain data. This highlights the value of explicit linguistic knowledge.
  • Accuracy gains don't always transfer to downstream tasks: While ERG shows higher intrinsic parsing accuracy, this advantage doesn't always translate into significant improvements in tasks like negation resolution. This suggests that some tasks might be less sensitive to fine-grained syntactic variations.
  • Trade-offs exist: There are clear trade-offs between accuracy, efficiency, and coverage. For instance, the pruned ERG parser (ERG_e) is faster but slightly less accurate and has lower coverage than the full ERG parser.
  • Tokenization matters: Differences in tokenization conventions (e.g., how punctuation is handled) can impact parser performance, though the ERG's approach, while increasing vocabulary size, seems to be handled well by its own parser.
  • Syntactic features are important for semantic parsing: The quality of syntactic dependencies significantly influences semantic dependency parsing performance, with "correlated" syntactic features (e.g., DT dependencies from ERG for DM targets) yielding better results.

Key Results

  • In-domain WSJ parsing: ERG parsers achieved the highest accuracy (around 93% TA, 86.7% LAS), significantly outperforming B&N and Berkeley. ERG_e offered a speed-up of about six times compared to ERG_a, with a slight accuracy drop.
  • Cross-domain parsing: ERG showed remarkable resilience to domain variation, with LAS drops of less than two points, while data-driven parsers experienced drops of up to 12 points.
  • Negation Resolution: Despite higher intrinsic accuracy, ERG did not consistently lead to significantly better performance on the negation resolution task compared to B&N.
  • Semantic Dependency Parsing: ERG parsers provided higher semantic dependency LF than B&N, particularly for the DM target representation, suggesting that accurate syntactic features are beneficial for this task.

Limitations and Future Work

  • Partial Coverage: The ERG parser, while accurate, fails to produce analyses for a small percentage of inputs, requiring fallback mechanisms.
  • Task Sensitivity: The negation resolution task appears less sensitive to parser accuracy differences than expected, suggesting that other tasks might be better suited for evaluating parser performance.
  • Future Work: The authors suggest exploring other PCFG and linear-time dependency parsers, investigating a wider variety of downstream tasks to understand their sensitivity to parser accuracy, and further exploring the benefits of linguistically richer representations.

Summary updated

Abstract

We compare three different approaches to parsing into syntactic, bi- lexical dependencies for English: a ‘direct’ data-driven dependency parser, a statistical phrase structure parser, and a hybrid, ‘deep’ grammar-driven parser. The analyses from the latter two are post- converted to bi-lexical dependencies. Through this ‘reduction’ of all three approaches to syntactic dependency parsers, we determine empirically what performance can be obtained for a common set of de- pendency types for English, across a broad variety of domains. In doing so, we observe what trade-offs apply along three dimensions, accuracy, efficiency, and resilience to domain variation. Our results suggest that the hand-built grammar in one of our parsers helps in both accuracy and cross-domain parsing performance, but these accuracy gains do not necessarily translate to improvements in the downstream task of negation resolution.
BibTeX
@article{ivanova2016different,
  author = {Angelina Ivanova and Stephan Oepen and Rebecca Dridan and Dan Flickinger and Lilja Øvrelid and Emanuele Lapponi},
  title = {{On Different Approaches to Syntactic Analysis Into Bi-Lexical Dependencies An Empirical Comparison of Direct, PCFG-Based, and HPSG-Based Parsers}},
  journal = {J. Lang. Model.},
  year = {2016},
  url = {https://openreview.net/forum?id=GIytD42eoK},
}
Figure