Abstract: Accurate multilingual transfer parsing typically relies on careful feature engineering. In this paper, we propose a hierarchical tensor-based approach for this task. This approach induces a compact feature representation by combining atomic features. However, unlike traditional tensor models, it enables us to incorporate prior knowledge about desired feature interactions, eliminating invalid feature combinations. To this end, we use a hierarchical structure that uses intermediate embeddings to capture desired feature combinations. Algebraically, this hierarchical tensor is equivalent to the sum of traditional tensors with shared components, and thus can be effectively trained with standard online algorithms. In both unsupervised and semi-supervised transfer scenarios, our hierarchical tensor consistently improves UAS and LAS over state-of-the-art multilingual transfer parsers and the traditional tensor model across 10 different languages.
Introduction
Dependency parsing—the process of determining the grammatical structure of a sentence—is a fundamental task in natural language processing. However, training accurate parsers typically requires massive amounts of annotated data, which is unavailable for the vast majority of the world's languages. Multilingual transfer parsing addresses this "resource-lean" problem by training a parser on one or more "resource-rich" source languages and applying it to a target language.
The primary challenge in transfer parsing is feature engineering. Researchers traditionally hand-craft complex feature templates that combine atomic features (like POS tags and arc direction) with typological knowledge (e.g., "does this language usually put subjects before verbs?"). While tensor-based models can automate feature combination by factorizing high-dimensional feature spaces, they are often "blind" to linguistic reality. A standard tensor will try to learn weights for impossible combinations—such as a word being both a Noun and a Verb simultaneously—leading to parameter explosion and sub-optimal performance.
Hierarchical Low-Rank Tensor Model
To overcome these limitations, we propose a Hierarchical Low-Rank Tensor scoring function. Instead of a single flat product of all features, the model builds representations in a tree-like hierarchy that explicitly incorporates linguistic prior knowledge.
As illustrated in the figure above, the model captures interactions in a structured manner:
- Bottom Level: The model first captures basic interactions between the head word ($H$), the modifier word ($M$), and the arc direction/length ($D$).
- Intermediate Level: It then incorporates typological features ($T_u, T_l$) that depend on these basic interactions. For example, it might combine the "Verb-Subject" ordering preference with the specific POS tags of the words involved.
- Top Level: Finally, it adds higher-level information like arc labels ($L$) and the POS tags of neighboring words ($H_c, M_c$).
Algebraically, this hierarchical structure is equivalent to a sum of traditional tensors with shared components. This formulation allows the model to be trained efficiently using standard online algorithms (like Passive-Aggressive updates) while ensuring that "invalid" feature combinations are never even considered.
Experimental Evaluation
The model was evaluated on the Multilingual Universal Dependency Treebank v2.0, covering 10 diverse languages: English (EN), French (FR), German (DE), Indonesian (ID), Italian (IT), Japanese (JA), Korean (KO), Brazilian-Portuguese (PT), Spanish (ES), and Swedish (SV).
Key Results
- Unsupervised Transfer: In a zero-shot scenario with no target-language annotations, the hierarchical tensor outperformed the state-of-the-art feature-based baseline (NT-Select) on 7 out of 10 languages, achieving an average 1.1% improvement in Unlabeled Attachment Score (UAS).
- Semi-supervised Transfer: When supplemented with just 50 annotated sentences from the target language, the improvement jumped to 1.7% UAS over the baseline and 4.5% over direct transfer.
- Comparison to Traditional Tensors: The hierarchical approach consistently outperformed "flat" multiway tensors. This confirms that explicitly excluding invalid feature combinations (like Noun $\wedge$ Verb) prevents the model from wasting its learning capacity on noise.
Conclusion and Future Work
The core innovation of this research is that we don't need to choose between manual feature engineering and "black box" tensors. By using a hierarchy, we can guide the tensor to only explore "legitimate" feature subspaces. This approach implements selective sharing, where the model knows which features are universal and which should be gated by a language's specific typology.
While the current model is arc-factored (first-order) and relies on the World Atlas of Language Structures (WALS) for typological features, it demonstrates a robust framework for structured prediction. Future work will explore higher-order structures and the application of hierarchical tensors to other tasks where prior knowledge about feature interactions is available.
Summary updated