Abstract
Multi-source cross-lingual transfer learning is essential for extending NLP capabilities to low-resource languages. However, current approaches often struggle with mutual interference between source languages and the loss of discriminative features during global alignment. We present DA-Net (Disentangled and Adaptive Network), which employs Feedback-guided Collaborative Disentanglement (FCD) to isolate language-specific features and Class-aware Parallel Adaptation (CPA) for fine-grained alignment between source and target languages. Our evaluation across 38 languages in tasks such as Named Entity Recognition (NER), Review Rating Classification (RRC), and Textual Entailment Prediction (TEP) shows that DA-Net consistently achieves state-of-the-art performance, especially in challenging cross-family transfer scenarios.
Introduction
While large-scale multilingual models like mBERT have revolutionized NLP, many languages still suffer from "data hungriness"—a lack of labeled datasets for specific tasks. Cross-lingual transfer learning attempts to solve this by training a model on a labeled "source" language (like English) and applying it to an unlabeled "target" language (like Swahili).
Real-world scenarios often provide multiple source languages, but existing multi-source methods face two major hurdles:
- Mutual Interference: Because all source languages typically share a single encoder, the resulting data representations become a "jumble" of different language features, confusing specialized classifiers.
- The Language Gap: Restrictive global alignment often strips away useful, language-specific nuances that aid classification, leading to poor performance on the target language.
Methodology
DA-Net addresses these issues through a modular architecture consisting of a shared encoder followed by language-specific branches. Each branch contains a disentangler, an adaptor, and a classifier.
The framework introduces two primary innovations:
- Feedback-guided Collaborative Disentanglement (FCD): This module uses a "max-min" strategy. It trains a disentangler to maximize the performance of its corresponding language-specific classifier while minimizing the performance of classifiers from other branches. This process "purifies" the representation by removing features that belong to other source languages.
- Class-aware Parallel Adaptation (CPA): Rather than global alignment, CPA aligns distributions at the class level (e.g., aligning "Positive Sentiment" features across languages). It utilizes a momentum-based model to generate pseudo-labels for target data and minimizes the distance between class-specific distributions using Maximum Mean Discrepancy (MMD).
Experimental Results
DA-Net was evaluated on 38 languages across NER (Wikiann), RRC (Amazon Reviews), and TEP (XNLI) tasks. The model demonstrated significant improvements, particularly for low-resource languages like Cantonese (zh-yue) and Esperanto (eo).
Key findings include:
- Performance Gains: DA-Net improved F1 scores by an average of 3% to 4.6% over previous state-of-the-art (SOTA) models in NER.
- Cross-Family Transfer: In NER, the model outperformed SOTA by over 17% for Persian and Urdu when transferring from English, Spanish, and Russian, demonstrating its robustness across linguistic families.
- Ablation Analysis: Removing FCD or CPA components resulted in significant performance declines, validating the necessity of both disentanglement and class-aware adaptation.
Conclusion
DA-Net provides a robust framework for multi-source cross-lingual transfer by treating classifiers as detectors to purify representations and avoiding the pitfalls of strict global alignment. While the linear scaling of branches with the number of source languages presents a computational challenge, the significant performance gains mark a substantial advancement in multilingual NLP. Future work will focus on improving the efficiency of scaling to a larger number of source languages.
Summary updated