DA-Net: A Disentangled and Adaptive Network for Multi-Source Cross-Lingual Transfer Learning

Ling Ge, Chunming Hu, Guanghui Ma, Jihong Liu, Hong Zhang

Abstract

Multi-source cross-lingual transfer learning is essential for extending NLP capabilities to low-resource languages. However, current approaches often struggle with mutual interference between source languages and the loss of discriminative features during global alignment. We present DA-Net (Disentangled and Adaptive Network), which employs Feedback-guided Collaborative Disentanglement (FCD) to isolate language-specific features and Class-aware Parallel Adaptation (CPA) for fine-grained alignment between source and target languages. Our evaluation across 38 languages in tasks such as Named Entity Recognition (NER), Review Rating Classification (RRC), and Textual Entailment Prediction (TEP) shows that DA-Net consistently achieves state-of-the-art performance, especially in challenging cross-family transfer scenarios.

Introduction

While large-scale multilingual models like mBERT have revolutionized NLP, many languages still suffer from "data hungriness"—a lack of labeled datasets for specific tasks. Cross-lingual transfer learning attempts to solve this by training a model on a labeled "source" language (like English) and applying it to an unlabeled "target" language (like Swahili).

Real-world scenarios often provide multiple source languages, but existing multi-source methods face two major hurdles:

  1. Mutual Interference: Because all source languages typically share a single encoder, the resulting data representations become a "jumble" of different language features, confusing specialized classifiers.
  2. The Language Gap: Restrictive global alignment often strips away useful, language-specific nuances that aid classification, leading to poor performance on the target language.

Methodology

DA-Net addresses these issues through a modular architecture consisting of a shared encoder followed by language-specific branches. Each branch contains a disentangler, an adaptor, and a classifier.

Overview of the DA-Net architecture, illustrating the flow from a shared encoder through collaborative disentanglement and parallel adaptation to final weighted prediction.

The framework introduces two primary innovations:

  • Feedback-guided Collaborative Disentanglement (FCD): This module uses a "max-min" strategy. It trains a disentangler to maximize the performance of its corresponding language-specific classifier while minimizing the performance of classifiers from other branches. This process "purifies" the representation by removing features that belong to other source languages.
  • Class-aware Parallel Adaptation (CPA): Rather than global alignment, CPA aligns distributions at the class level (e.g., aligning "Positive Sentiment" features across languages). It utilizes a momentum-based model to generate pseudo-labels for target data and minimizes the distance between class-specific distributions using Maximum Mean Discrepancy (MMD).

Experimental Results

DA-Net was evaluated on 38 languages across NER (Wikiann), RRC (Amazon Reviews), and TEP (XNLI) tasks. The model demonstrated significant improvements, particularly for low-resource languages like Cantonese (zh-yue) and Esperanto (eo).

Key findings include:

  • Performance Gains: DA-Net improved F1 scores by an average of 3% to 4.6% over previous state-of-the-art (SOTA) models in NER.
  • Cross-Family Transfer: In NER, the model outperformed SOTA by over 17% for Persian and Urdu when transferring from English, Spanish, and Russian, demonstrating its robustness across linguistic families.
  • Ablation Analysis: Removing FCD or CPA components resulted in significant performance declines, validating the necessity of both disentanglement and class-aware adaptation.
Ablation study comparing the full DA-Net against versions without Feedback-guided Collaborative Disentanglement (FCD) and Class-aware Parallel Adaptation (CPA) across multiple languages.

Conclusion

DA-Net provides a robust framework for multi-source cross-lingual transfer by treating classifiers as detectors to purify representations and avoiding the pitfalls of strict global alignment. While the linear scaling of branches with the number of source languages presents a computational challenge, the significant performance gains mark a substantial advancement in multilingual NLP. Future work will focus on improving the efficiency of scaling to a larger number of source languages.

Summary updated

Abstract

Multi-Source cross-lingual transfer learning deals with the transfer of task knowledge from multiple labelled source languages to an unlabeled target language under the language shift. Existing methods typically focus on weighting the predictions produced by language-specific classifiers of different sources that follow a shared encoder. However, all source languages share the same encoder, which is updated by all these languages. The extracted representations inevitably contain different source languages' information, which may disturb the learning of the language-specific classifiers. Additionally, due to the language gap, language-specific classifiers trained with source labels are unable to make accurate predictions for the target language. Both facts impair the model's performance. To address these challenges, we propose a Disentangled and Adaptive Network (DA-Net). Firstly, we devise a feedback-guided collaborative disentanglement method that seeks to purify input representations of classifiers, thereby mitigating mutual interference from multiple sources. Secondly, we propose a class-aware parallel adaptation method that aligns class-level distributions for each source-target language pair, thereby alleviating the language pairs' language gap. Experimental results on three different tasks involving 38 languages validate the effectiveness of our approach.
BibTeX
@inproceedings{ge2024da,
  author = {Ling Ge and Chunming Hu and Guanghui Ma and Jihong Liu and Hong Zhang},
  title = {{DA-Net: A Disentangled and Adaptive Network for Multi-Source Cross-Lingual Transfer Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i16.29761},
}
Figure