An Empirical Etudy of Non-Lexical Extensions to Delexicalized Transfer

Anders Søgaard, Julie WulffCOLING 2012 · OpenReview

Authors: Anders Søgaard, Julie Wulff
Venue: Proceedings of COLING 2012: Posters

Introduction

Building syntactic parsers—computational tools that analyze the grammatical structure of sentences—typically requires extensive manually annotated "treebanks." While such resources are plentiful for languages like English or German, they are non-existent for the vast majority of the world's languages. This paper investigates delexicalized transfer, a technique where a parser is trained on part-of-speech (POS) sequences from a source language and applied to a target language. The authors propose several extensions to this approach, focusing on how to effectively combine information from multiple source languages without letting one large dataset overwhelm the others.

Methods

The researchers employ an easy-first transition-based dependency parser. Unlike traditional left-to-right parsers, this model identifies the most certain grammatical connections first, building the structure incrementally.

To improve transfer performance, the study introduces four key innovations:

  1. Model Averaging: Instead of concatenating data from all source languages, the authors train separate "under-fitted" models (one pass over the data) for each and average their weight vectors. This acts as a form of regularization.
  2. Language-level Weighting: Source models are weighted based on their linguistic proximity to the target language. This is determined via genealogy (using a family tree) or typology (using structural features from the WALS database).
  3. Sentence-level Weighting: Individual source sentences are weighted by their "perplexity" relative to the target language, prioritizing sentences that structurally resemble the target.
  4. Unsupervised Model Selection: A random walk algorithm on a graph of models is used to select a diverse subset for averaging.
Linguistic genealogy tree used for language-level weighting.

Datasets and Evaluation

The approach was evaluated on Indo-European corpora from the CoNLL-X (2006) and CoNLL 2007 benchmarks. To ensure cross-linguistic compatibility, all language-specific POS tags were mapped to Google’s Universal Tag Set, ensuring that a "Noun" in Bulgarian is treated the same as a "Noun" in Spanish by the algorithm.

Key Results and Insights

The study demonstrates that deliberate "under-fitting" serves as an effective regularizer, preventing the model from over-specializing in source-specific idiosyncrasies that might not apply to the target. Furthermore, model averaging proved superior to data concatenation, as it ensures every source language contributes equally to the final grammar.

Performance comparison of different transfer methods across target languages.

As shown in the results table, weighted and averaged models consistently outperformed "flat" transfer. Specifically, using linguistic genealogy (w-gtree) and sentence-level importance weighting (weighted) yielded statistically significant improvements. The proposed system achieved the best reported results for a fully delexicalized model at the time, outperforming more complex contemporary methods while remaining computationally efficient.

Conclusion

This work highlights the value of incorporating linguistic knowledge—such as genealogy and typology—into machine learning models for low-resource NLP. While the current model is fully delexicalized, the authors acknowledge that incorporating lexical features, such as bilingual word clusters or lexical projections, is a logical next step to further improve performance.

Summary updated

BibTeX
@inproceedings{sgaard2012empirical,
  author = {Anders Søgaard and Julie Wulff},
  title = {{An Empirical Etudy of Non-Lexical Extensions to Delexicalized Transfer}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2012},
  url = {https://openreview.net/forum?id=f3LemCw3xXR},
}
Figure