Abstract
Wenjuan Han, Ge Wang, Yong Jiang, Kewei Tu. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
2019.
Introduction
Grammar induction—the unsupervised learning of syntactic structures from raw text—remains a fundamental challenge in natural language processing. While multilingual approaches can leverage shared structural patterns across languages, traditional methods often rely on rigid linguistic phylogeny (family trees) to determine parameter sharing. However, historical ancestry does not always correlate with contemporary syntactic behavior. This research introduces a data-driven approach that represents language identities as continuous vectors, allowing the model to "softly tie" languages based on actual structural similarity rather than historical labels.
Methodology
The proposed framework extends the Neural Dependency Model with Valence (NDMV) through three primary innovations:
- Continuous Language Embeddings: Each language is represented as a high-dimensional vector (embedding), enabling the model to quantify degrees of similarity between languages.
- Neural Parameter Prediction: A shared neural network takes these embeddings as input to predict grammar rule probabilities. This architecture facilitates generalization; similar embeddings naturally result in similar grammar parameters.
- Auxiliary Language Identification: To ensure embeddings capture meaningful syntactic information, an auxiliary task requires the model to identify the language of a sentence based on its Part-of-Speech (POS) tags. This forces the embeddings to encode the unique "syntactic fingerprints" of each language.
The training objective maximizes both the likelihood of sentence generation by the grammar model and the accuracy of the language identification task.
Experimental Setup
The model was evaluated using the Universal Dependency (UD) Treebank 1.4, covering 15 languages from 8 distinct families (e.g., Germanic, Romance, Slavonic, Finnic). Sentences were converted into universal POS tags to maintain a shared vocabulary. Performance was measured using Directed Dependency Accuracy (DDA), the percentage of words correctly assigned to their "head" word.
Results and Discussion
The integrated model (G+I) achieved a state-of-the-art average DDA of 55.6%, significantly outperforming the monolingual NDMV baseline (48.1%) and standard multilingual DMV approaches (48.4%).
Typological Insights
A key finding is that the learned embeddings prioritize typological similarity over genetic phylogeny. As shown in the t-SNE visualization, languages like English and Norwegian (Germanic) are positioned closer to certain Romance languages than to German or Dutch. This reflects their shared word-order characteristics (e.g., strict SVO order) rather than their historical roots.
The auxiliary task proved essential as a regularizer, preventing embedding collapse and helping the model distinguish subtle syntactic nuances. Furthermore, the model demonstrated superior performance in cross-family knowledge transfer, a scenario where traditional phylogeny-based models often struggle.
Conclusion
This work demonstrates that continuous language embeddings combined with auxiliary identification tasks provide a robust mechanism for multilingual grammar induction. By moving beyond rigid family trees to a data-driven typological representation, the model achieves significant gains in accuracy and generalization. Future work will explore lexicalized parsing using cross-lingual word embeddings and techniques to address data imbalance in low-resource languages.
Summary updated