Abstract: Ryan Cotterell, Georg Heigold. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.
2017.
Introduction
Morphological tagging is a foundational task in natural language processing (NLP) that involves identifying grammatical features—such as gender, number, case, and tense—for every word in a sentence. While state-of-the-art taggers perform exceptionally well for high-resource languages like English or Spanish, they typically require thousands of manually annotated sentences for training. This paper addresses the critical data gap for low-resource languages by proposing a character-level neural framework that leverages cross-lingual transfer learning.
The Morphological Tagging Problem
For the vast majority of the world's languages, large-scale annotated datasets do not exist. Traditional cross-lingual solutions often rely on expensive external resources like bitext (parallel translations) or bilingual lexicons to project information from a high-resource language to a low-resource one.
The authors tackle this challenge by "borrowing" knowledge from genetically related high-resource languages using only raw text and tags, avoiding the need for parallel corpora.
Character-Level Neural Architectures
The core of the proposed system is a character-level recurrent neural network (RNN), specifically using Long Short-Term Memory (LSTM) units. The architecture follows a two-tier approach:
- Word Embedding: A bidirectional LSTM (biLSTM) processes individual characters to create a vector representation (embedding) for each word.
- Sentence Processing: A second biLSTM processes the sequence of word embeddings to capture contextual information across the sentence.
To facilitate transfer learning, the authors introduce several Multi-Task Learning (MTL) variants:
- Language-Universal Softmax: Languages share character embeddings and a single output layer. A unique "language ID" symbol is prepended and postpended to every word to help the model distinguish between languages.
- Language-Specific Softmax: Languages share internal representations but utilize dedicated output layers for final predictions.
- Joint Tagging and Language ID: The model is trained to simultaneously predict the morphological tag and the language identity.
Experimental Setup and Results
The study evaluated 18 languages across four major families (Romance, Germanic, Slavic, and Uralic) using the Universal Dependencies (UD) v2.0 treebanks. To simulate low-resource conditions, training data for the target language was restricted to just 100 or 1,000 sentences.
Key findings include:
- Massive Performance Gains: In the most data-constrained settings (100 sentences), cross-lingual models improved accuracy by up to 30% over monolingual baselines.
- The Power of Sub-word Features: By operating on characters, the model learns that specific sequences (like suffixes) signal grammatical categories across related languages (e.g., Spanish -amos and Italian -iamo both indicating first-person plural).
- Genetic Relatedness: Transfer is most effective between closely related languages. Spanish serves as an excellent source for Catalan but is less effective for Romanian.
- Superiority over Baselines: The neural models consistently outperformed MARMOT, a top-tier non-neural tagger, and previous projection methods requiring parallel text.
Conclusion
This research demonstrates that character-level neural models can effectively bridge the gap between related languages, making high-quality morphological tagging viable for low-resource languages. By treating characters as a "universal language" of morphology, the proposed multi-task learning framework achieves significant accuracy gains without the need for expensive bilingual resources. Future work may extend this logic to other tasks like lemmatization or explore zero-resource scenarios.
Summary updated