Anders Søgaard. Proceedings of the 16th International Workshop on Treebanks and Linguistic Theories. 2017.
Abstract
In this provocative position paper, Anders Søgaard argues that current methods for building treebanks—the gold-standard datasets used to train syntactic parsers—are "too nice" to researchers. He contends that our reliance on narrow domains and forced consensus leads to an overestimation of how well models actually generalize. To bridge the gap between benchmark performance and real-world utility, Søgaard proposes four radical shifts: embracing adversarial sampling, utilizing multiple data splits, preserving annotator disagreements, and moving beyond the strict requirement for "well-formed" trees.
The Problem: The Generalization Gap
Syntactic parsing—the process of identifying the grammatical structure of a sentence—is a cornerstone of natural language processing. However, there is a persistent and frustrating gap between a parser's performance on a "held-out" test set and its performance "in the wild."
Søgaard argues that current treebank design principles are the primary culprit. We often attribute performance drops to linguistic complexity (e.g., "Arabic is just harder to parse than English"), but the evidence suggests the real issue is domain shift and sampling bias. If our training and test sets are too similar, we aren't measuring robustness; we're measuring how well a model can memorize a specific niche of language.
The Myth of the Representative Sample
A fundamental critique in the paper is the breakdown of the "i.i.d." (independent and identically distributed) assumption in NLP. Because language is unbounded and constantly evolving, it is impossible to truly sample "randomly" from it. The English Penn Treebank (WSJ), for instance, is highly specific to 1980s financial journalism. Using it as a proxy for "the English language" leads to models that are brittle when faced with anything else.
Four Principles for Next-Generation Treebanks
Søgaard outlines a new blueprint for how we should conceptualize and build these datasets:
- Adversarial Sampling: Instead of trying to create a "representative" sample, we should intentionally include samples that are difficult or significantly different from the training data to stress-test model limits.
- Multiple Splits: Rather than a single standard train/test split, treebanks should provide multiple ways to "cut the cake"—such as splits based on time, genre, or sentence length—to test different dimensions of generalization.
- Learning from Disagreements: Traditionally, when two human annotators disagree, a third person "adjudicates" to pick a winner. Søgaard suggests keeping all annotations. Disagreement often signals genuine linguistic ambiguity that models should learn to navigate rather than ignore.
- Beyond Trees ("Crazy Trees"): Modern deep learning parsers (like seq2seq models) no longer require data to be a perfect tree or a directed acyclic graph (DAG). We should allow for more complex, cyclic, or non-traditional structures if they better describe linguistic reality.
Domain Shift vs. Language Shift
To support these claims, the paper highlights quantitative findings from existing literature. One of the most startling insights is that changing the domain can hurt performance more than changing the language itself.
- The Twitter Drop: A parser with 90.6% accuracy on newswire dropped to 73.6% on Twitter data—a 17% absolute drop.
- Cross-Lingual Transfer: An "impoverished" English model transferred to Portuguese saw a 14% absolute drop.
The fact that moving from News to Twitter was more damaging than moving from English to Portuguese underscores that our models are often overfit to specific writing styles rather than learning universal syntax.
Conclusion and Future Work
While these suggestions depart from decades of established methodology, they offer a path toward more honest evaluation. Moving away from a single "standard split" may make direct comparisons harder, but Søgaard suggests providing a list of standard alternative splits to mitigate this. By embracing the messiness of language—its ambiguities, its domain variances, and its non-linear structures—the NLP community can move toward models that are truly robust in the wild.
Summary updated