Old School vs. New School: Comparing Transition-Based Parsers with and without Neural Network Enhancement

Miryam de Lhoneux, Sara Stymne, Joakim NivreTLT 2017 · OpenReview

Introduction

In recent years, neural networks have significantly boosted the accuracy of syntactic parsing. However, the community has lacked systematic, "apples-to-apples" comparisons between classical transition-based parsers—which rely on sparse, hand-crafted features—and modern neural parsers that learn dense vector representations. This study addresses whether neural models' superiority stems solely from massive training sets, if they possess a steeper learning curve, and which specific linguistic structures they handle more effectively than their classical predecessors.

Methodology

The researchers compared two representative transition-based systems:

  • Classical ("Old School"): MaltParser, optimized via MaltOptimizer, utilizing an arc-standard swap algorithm with a lazy oracle for non-projective trees.
  • Neural ("New School"): UDPipe (specifically the Parsito model), utilizing dense embeddings and beam search.

The evaluation used eight treebanks from the Universal Dependencies (UD) project (v1.3), strategically selected for typological variety: Czech, Chinese, Finnish, English, Ancient Greek (PROIEL), Kazakh, Tamil, and Hebrew. To analyze learning dynamics, both parsers were trained on truncated datasets ranging from 1,000 to 50,000 tokens.

Learning Dynamics and Data Efficiency

A common assumption in NLP is that neural networks require massive datasets to outperform classical machine learning. This study challenges that "Data Efficiency Myth." As shown in the learning curve experiments, the neural parser (UDPipe) often overtakes the classical parser (MaltParser) with just a few thousand training tokens.

Learning curves for eight languages showing LAS performance as training size increases from 1K to 14K tokens.

Furthermore, the hypothesis that neural models maintain a continuously steeper learning curve was disproven. While UDPipe improves rapidly during the first few thousand tokens, its learning curve eventually flattens, progressing at a pace nearly identical to MaltParser, albeit from a higher performance baseline.

Error Analysis: The Long-Distance Advantage

Fine-grained error analysis revealed that the primary advantage of neural parsers lies in resolving long-distance dependencies. Classical parsers, relying on greedy local inference, suffer significantly from error propagation as sentence length and dependency distance increase.

F1 score performance relative to dependency relation length, comparing UDPipe and MaltParser.

UDPipe demonstrated noticeably higher F1 scores for distant relations such as conjunctions (conj) and adverbial clauses (advcl). Conversely, MaltParser maintains a niche in extremely low-resource scenarios and specific short-range relations, such as numeric modifiers (nummod), where it occasionally edges out the neural model.

Key Results

  • Overall Accuracy: UDPipe outperformed MaltParser on six of the eight languages. In Czech, UDPipe achieved an 82.6% Labeled Attachment Score (LAS) against MaltParser's 80.3%. In English, the scores were 81.3% and 79.9%, respectively.
  • Low-Resource Performance: MaltParser proved more robust on the smallest treebanks, outperforming UDPipe in Kazakh (44.4% vs. 41.2% LAS) and Tamil (58.8% vs. 58.1% LAS).
  • Dependency Length: MaltParser’s accuracy degrades much faster than UDPipe’s as the distance between a dependent and its head increases.

Conclusion and Future Work

While neural enhancement provides a clear performance boost, particularly for long-distance dependencies, the role of beam search remains a confounding variable. UDPipe utilizes beam search to explore multiple paths, whereas MaltParser relies on greedy inference. Future research should isolate the impact of beam search to determine if the neural model's success is due to its dense representations or its search algorithm. Additionally, further fine-grained analysis is required to fully demystify the linguistic strengths of neural parsing architectures.

Summary updated

BibTeX
@inproceedings{lhoneux2017old,
  author = {Miryam de Lhoneux and Sara Stymne and Joakim Nivre},
  title = {{Old School vs. New School: Comparing Transition-Based Parsers with and without Neural Network Enhancement}},
  booktitle = {TLT},
  year = {2017},
  url = {https://openreview.net/forum?id=qv21rvdxcOV},
}
Figure