Abstract
Dependency parsing for low-resource languages often relies on zero-shot transfer from massive multilingual models. However, training on syntactically distant languages can introduce noise that degrades performance. We present TOWER (Treebank Selection via Hierarchical Source Selection), a framework that uses a syntax-based hierarchy to selectively curate training data for specific target languages. TOWER achieves significant performance gains—over 10 LAS points on average across 11 low-resource languages—compared to all-encompassing multilingual parsers.
Introduction
Identifying the grammatical structure of sentences is a foundational task in NLP. However, training high-quality dependency parsers requires large, manually annotated "treebanks." While the Universal Dependencies (UD) project has significantly expanded the availability of these resources, the vast majority of the world’s languages remain low-resource or zero-resource.
The standard approach to this scarcity has been "zero-shot transfer," where a single massive model (e.g., UDify) is trained on all available treebanks. The assumption is that the model will learn a "universal grammar." We challenge this "everything-at-once" trend, arguing that training on syntactically dissimilar languages can be counterproductive. Instead, we propose finding a "Goldilocks" zone: a subset of source languages that are syntactically similar enough to the target to be useful without introducing excessive noise.
The TOWER Framework
TOWER introduces a systematic way to navigate the trade-off between data quantity and syntactic relevance through three core components:
1. Syntactic Clustering
Rather than relying on geographic or genealogical proximity, TOWER groups languages based on URIEL vectors—manually coded linguistic features (e.g., word order). Hierarchical clustering on these features produces a "Tower" (a dendrogram) of syntactic similarity.
2. Treebank Selection (TBS) Heuristic
For a given target language, TOWER starts at its leaf in the hierarchy and "climbs" upwards. At each node, it adds treebanks from neighboring branches. The "climbing" stops when the relative increase in data volume no longer justifies the decrease in syntactic similarity, as determined by a mathematical heuristic.
3. Proxy Model Selection (MS)
In zero-shot scenarios, the lack of development data makes it difficult to know when to stop training. TOWER addresses this by aggregating data from the closest hierarchical neighbors to create a "proxy" development set, ensuring the model optimizes for relevant linguistic patterns.
Experimental Results
We evaluated TOWER using UD v2.5 across 138 test treebanks. The underlying architecture is a Shallow Biaffine Parser built on XLM-R.
TOWER demonstrates its most significant impact in the "LOW" resource category (languages with no training data). On average, across 11 such languages, TOWER achieved a 33.8 Labeled Attachment Score (LAS), substantially outperforming UDify (22.2) and UDapter (29.3).
Individual gains were even more pronounced. For Amharic (am), TOWER improved performance by over 35 LAS points compared to UDify. For Sanskrit (sa), it saw a 26-point boost. Ablation studies (removing the TBS component) showed a significant drop in performance, confirming that the success of TOWER is driven by its selective data strategy rather than just the power of the underlying transformer model.
Discussion and Conclusion
The success of TOWER highlights that for low-resource NLP, quality and relevance of data often trump sheer quantity. Interestingly, the hierarchy sometimes prioritizes "syntactic cousins" over genealogical relatives. For instance, training Tagalog on Celtic languages (which share rare Verb-Subject-Object order) yields better results than training on geographic neighbors.
While TOWER currently depends on the URIEL database and a relatively simple selection heuristic, its core philosophy—hierarchical source selection—offers a promising path for other low-resource tasks like Named Entity Recognition or Part-of-Speech tagging. By climbing the "Tower of Treebanks" strategically, we can build more accurate parsers for the world's diverse languages.
Summary updated