On Different Approaches to Syntactic Analysis Into Bi-Lexical Dependencies. An Empirical Comparison of Direct, PCFG-Based, and HPSG-Based Parsers

Angelina Ivanova, Stephan Oepen, Rebecca Dridan, Dan Flickinger, Lilja Øvrelid

Abstract

This study presents a rigorous cross-paradigm comparison of three distinct parsing technologies—direct dependency, PCFG-based, and HPSG-based—by reducing their outputs to a common dependency format. We evaluate these approaches on their accuracy, efficiency, and resilience to domain shift, demonstrating that the incorporation of hand-built linguistic grammars significantly enhances performance and stability.

Introduction

Syntactic parsing, the determination of grammatical structure, is a cornerstone of natural language processing. Most contemporary research focuses on bi-lexical dependencies, yet the field remains siloed, with comparisons typically restricted to models within the same paradigm. This paper addresses a fundamental question: Does "deep" linguistic knowledge provide a tangible advantage over purely data-driven methods? Specifically, we compare "shallow" statistical parsers against "deep" grammar-driven parsers, investigating their performance on standard news text and their ability to generalize to diverse genres such as spoken dialogues and technical essays.

Methodology

To ensure a fair comparison across disparate parsing architectures, we employ a "reduction" strategy, converting all outputs into a unified representation: Derivation Tree-Derived Dependencies (DT). Three primary engines are evaluated:

  1. Direct Dependency (B&N): A state-of-the-art transition-based parser.
  2. PCFG-Based (Berkeley): A statistical phrase-structure parser.
  3. HPSG-Based (PET/ERG): A hybrid parser utilizing the English Resource Grammar (ERG), combining hand-coded rules with a statistical disambiguation model.

A key challenge in cross-parser evaluation is inconsistent tokenization (e.g., "don't" vs. "do n't"). To mitigate this, we introduce a character-based evaluation metric ($LAS_c$), which measures accuracy based on character offsets rather than token indices.

Datasets

The parsers were trained and initially tested on DeepBank, a version of the Wall Street Journal (WSJ) corpus re-annotated with HPSG analyses. To test cross-domain resilience, we utilized the Redwoods Treebank, which includes diverse datasets ranging from software advocacy to spoken travel planning.

Table 1: Summary of the DeepBank and Redwoods datasets used for training and cross-domain evaluation.

Results and Discussion

Our results indicate a clear advantage for the HPSG-based approach. In-domain, the HPSG parser ($ERG_a$) achieved a Labeled Attachment Score (LAS) of 92.87%, outperforming the best data-driven parser (B&N) by over 6 percentage points.

The most striking finding, however, is the resilience of the grammar-driven approach to domain shift. While the statistical parsers experienced accuracy drops of 10–12 points when moving to out-of-domain data, the HPSG parser's performance remained remarkably stable, with drops of less than 2 points across all test sets.

Figure 1: Labeled Attachment Score (LAS) across different Part-of-Speech (PoS) categories for the three parsing approaches.

The hand-built rules of the HPSG parser act as "safety rails," preventing the system from proposing grammatically impossible structures in unfamiliar domains. Furthermore, the explicit lexical information in the ERG allows the parser to handle complex adjunct and argument distinctions more effectively than purely statistical models.

In terms of efficiency, while the full HPSG parser is slower, a pruned variant ($ERG_e$) achieves speeds comparable to the direct dependency parser (1.8s vs 1.4s per input) with minimal loss in accuracy.

Conclusion

This comparison demonstrates that "deep" linguistic grammars provide substantial gains in both accuracy and cross-domain robustness. The primary limitation remains "coverage" — the HPSG parser failed on approximately 15% of sentences that did not conform to its rule set. Future research should focus on hybrid architectures that utilize high-precision deep parsing as a primary engine, falling back to robust statistical models only when necessary.

Summary updated

BibTeX
@article{ivanova2016different,
  author = {Angelina Ivanova and Stephan Oepen and Rebecca Dridan and Dan Flickinger and Lilja Øvrelid},
  title = {{On Different Approaches to Syntactic Analysis Into Bi-Lexical Dependencies. An Empirical Comparison of Direct, PCFG-Based, and HPSG-Based Parsers}},
  journal = {J. Lang. Model.},
  year = {2016},
  url = {https://openreview.net/forum?id=GIytD42eoK},
}
Figure