Introduction
Dependency parsing is the task of predicting the grammatical structure of a sentence by linking words (nodes) with labeled directed arcs (edges). While performance varies across languages, a provocative 2020 study by Søgaard suggested that "leakage"—the overlap of identical tree structures between training and test sets—might be the primary driver of success. This implies that parsers might simply be memorizing common patterns rather than learning underlying linguistic logic. This paper re-evaluates this claim, investigating whether leakage is a genuine explanatory factor or a statistical artifact.
Refining the Leakage Metric
The authors argue that the original whole-tree, unlabeled graph approach does not capture how modern graph-based parsers actually function. Instead, they introduce labeled leakage, which accounts for specific word roles and parts of speech, and sub-tree leakage.
Because graph-based parsers make local decisions rather than predicting entire trees at once, decomposing a tree into its constituent sub-trees (consisting of a node, its parent, and its children) provides a more realistic measure of what a model might "recognize" from its training data.
Experimental Methodology
The study employs three main strategies:
- Metric Comparison: Comparing whole-tree vs. sub-tree leakage across various languages in the Universal Dependencies (UD) framework.
- Counterfactual Experiments: Training models on datasets where specific constructions (e.g., adjectival modifiers) were intentionally removed to test generalization capabilities.
- Zero-Shot Analysis: Evaluating performance on unseen languages, such as Faroese, using models trained on related Germanic languages.
Key Findings
The Monolingual Myth
In standard monolingual settings, leakage is a poor predictor of performance. For modern parsers like UDify, tree-level leakage actually showed a negative Spearman’s correlation ($\rho \approx -0.360$). This suggests that higher structural overlap does not necessarily lead to better results; in fact, "leaky" sentences are often just shorter and simpler, making them easier to parse regardless of overlap.
Generalization vs. Memorization
Counterfactual experiments revealed that parsers are capable of generalizing. Even when a specific construction was entirely absent from the training data, models could still correctly parse it about 50% of the time by leveraging knowledge from other local structures. Furthermore, training on a diverse set of unique structures often yielded better results than training on data that overlapped heavily with the test set.
The Zero-Shot Exception
Leakage plays a significantly different role in zero-shot cross-lingual scenarios. When a model encounters a new language, the presence of isomorphic sub-trees from related languages in the training set becomes a strong predictor of success, with Spearman’s $\rho$ reaching up to 0.609.
Conclusion
While leakage does not explain performance in typical monolingual settings, it remains a critical factor in zero-shot cross-lingual parsing. The results suggest that future research should prioritize structural diversity in training data to build more robust models that learn the "logic" of language rather than relying on structural overlap.
Summary updated