Abstract
This paper introduces a novel framework for bilingual grammar induction that learns syntactic structures for two languages simultaneously without relying on parallel corpora or translation dictionaries. By employing three distinct regularization techniques—parameter-level, edge-score-level, and tree-level—the model facilitates knowledge sharing between languages. Experimental results across diverse language families demonstrate significant improvements in parsing accuracy, highlighting the effectiveness of joint bilingual training over traditional sequential transfer methods.
Introduction
Syntactic parsing is a fundamental task in natural language processing, yet it typically requires extensive human-labeled data. Grammar induction—the unsupervised learning of these structures from raw text—remains a significant challenge. While previous attempts at bilingual grammar induction often relied on "crutches" like parallel text or linguistic family trees, this research explores a more realistic and difficult scenario: improving grammar induction for two languages using only their respective raw texts, with no explicit mapping between them.
Methods: Knowledge Sharing via Regularization
The proposed framework is built upon Convex-MST, a state-of-the-art unsupervised discriminative parser. The core innovation lies in three regularization techniques that encourage two models (one for each language) to "whisper" structural hints to one another:
- Weight Regularization (W-Reg): This method applies an $L_2$ distance penalty to the model parameters (weights) of the two languages, assuming that underlying grammatical features share universal properties.
- Edge Score Regularization (E-Reg): Rather than forcing identical weights, this approach encourages similar scores for specific dependency edges (e.g., the link between a verb and its object), allowing for flexibility in word order.
- Parse Tree Regularization (T-Reg): Drawing from knowledge distillation, this method encourages the parse tree produced by one model to be structurally similar to the predictions of the other.
The models are optimized using coordinate descent, alternating between updating the predicted parse trees and refining the model parameters.
Key Insights and Results
The researchers evaluated their framework on the Universal Treebanks (v2.0) and Universal Dependencies (UD) Treebank 1.4, covering over 20 languages. Several key findings emerged:
- Bilingual Synergy: Jointly training two languages (Bilingual Induction) consistently outperformed training on one and transferring to another (Transfer Induction). For instance, in the English-Japanese (EN-JA) pair, the Japanese model achieved a +2.4 point boost in accuracy over the baseline.
- Abstraction is Robust: While W-Reg was effective for closely related languages, E-Reg and T-Reg proved far more robust for diverse language pairs. This suggests that edge scores and tree structures capture higher-level linguistic concepts that transcend raw feature weights.
- The Power of Regularization: As shown in the performance analysis, the choice of the regularization parameter ($\lambda$) is crucial. Both E-Reg and T-Reg consistently outperformed the simpler W-Reg across various settings.
Limitations and Future Work
The framework's performance was less consistent when languages were highly divergent, such as the English-Indonesian pair, where regularization occasionally proved counterproductive. Additionally, the current model relies on gold Part-of-Speech (POS) tags. Future work aims to eliminate this dependency by inducing POS tags alongside grammar and extending these regularization techniques to other unsupervised tasks like word sense disambiguation or semantic role labeling.
Summary updated