Introduction
In recent years, neural networks have significantly boosted the accuracy of syntactic parsing. However, the community has lacked systematic, "apples-to-apples" comparisons between classical transition-based parsers—which rely on sparse, hand-crafted features—and modern neural parsers that learn dense vector representations. This study addresses whether neural models' superiority stems solely from massive training sets, if they possess a steeper learning curve, and which specific linguistic structures they handle more effectively than their classical predecessors.
Methodology
The researchers compared two representative transition-based systems:
- Classical ("Old School"): MaltParser, optimized via MaltOptimizer, utilizing an arc-standard swap algorithm with a lazy oracle for non-projective trees.
- Neural ("New School"): UDPipe (specifically the Parsito model), utilizing dense embeddings and beam search.
The evaluation used eight treebanks from the Universal Dependencies (UD) project (v1.3), strategically selected for typological variety: Czech, Chinese, Finnish, English, Ancient Greek (PROIEL), Kazakh, Tamil, and Hebrew. To analyze learning dynamics, both parsers were trained on truncated datasets ranging from 1,000 to 50,000 tokens.
Learning Dynamics and Data Efficiency
A common assumption in NLP is that neural networks require massive datasets to outperform classical machine learning. This study challenges that "Data Efficiency Myth." As shown in the learning curve experiments, the neural parser (UDPipe) often overtakes the classical parser (MaltParser) with just a few thousand training tokens.
Furthermore, the hypothesis that neural models maintain a continuously steeper learning curve was disproven. While UDPipe improves rapidly during the first few thousand tokens, its learning curve eventually flattens, progressing at a pace nearly identical to MaltParser, albeit from a higher performance baseline.
Error Analysis: The Long-Distance Advantage
Fine-grained error analysis revealed that the primary advantage of neural parsers lies in resolving long-distance dependencies. Classical parsers, relying on greedy local inference, suffer significantly from error propagation as sentence length and dependency distance increase.
UDPipe demonstrated noticeably higher F1 scores for distant relations such as conjunctions (conj) and adverbial clauses (advcl). Conversely, MaltParser maintains a niche in extremely low-resource scenarios and specific short-range relations, such as numeric modifiers (nummod), where it occasionally edges out the neural model.
Key Results
- Overall Accuracy: UDPipe outperformed MaltParser on six of the eight languages. In Czech, UDPipe achieved an 82.6% Labeled Attachment Score (LAS) against MaltParser's 80.3%. In English, the scores were 81.3% and 79.9%, respectively.
- Low-Resource Performance: MaltParser proved more robust on the smallest treebanks, outperforming UDPipe in Kazakh (44.4% vs. 41.2% LAS) and Tamil (58.8% vs. 58.1% LAS).
- Dependency Length: MaltParser’s accuracy degrades much faster than UDPipe’s as the distance between a dependent and its head increases.
Conclusion and Future Work
While neural enhancement provides a clear performance boost, particularly for long-distance dependencies, the role of beam search remains a confounding variable. UDPipe utilizes beam search to explore multiple paths, whereas MaltParser relies on greedy inference. Future research should isolate the impact of beam search to determine if the neural model's success is due to its dense representations or its search algorithm. Additionally, further fine-grained analysis is required to fully demystify the linguistic strengths of neural parsing architectures.
Summary updated