Parameter Differentiation Based Multilingual Neural Machine Translation

Qian Wang, Jiajun Zhang

Multilingual Neural Machine Translation (MNMT) aims to build a single model capable of translating between multiple language pairs. While this approach facilitates knowledge transfer—particularly benefiting low-resource languages—it often suffers from a "capacity bottleneck." When parameters are shared across diverse languages, high-resource languages frequently experience performance degradation due to negative transfer, where gradients from different tasks pull shared parameters in conflicting directions.

This paper introduces Parameter Differentiation (PD), a dynamic training method inspired by biological cellular differentiation. Instead of using a static, manually designed architecture, the model starts with fully shared parameters and evolves language-specific components based on observed gradient conflicts during training.

Methodology: From Shared to Specialized

The PD framework follows a three-stage process: initialization, detection, and differentiation.

  1. Initialization: The model begins as a standard Transformer with all parameters shared across all translation tasks.
  2. Detection: Periodically, the system evaluates the "interference degree" of shared parameters by calculating inter-task gradient similarity. If gradients for different tasks point in significantly different directions, they are identified as conflicting.
  3. Differentiation: Parameters flagged for high conflict are duplicated. These new copies are then reallocated to specific tasks or language groups. For instance, if gradients for European languages align but conflict with those for Asian languages, the model creates specialized versions for each group.
Conceptual overview of parameter differentiation, showing the transition from a fully shared model (a) to a partially specialized one (c).

The granularity of this differentiation can vary, ranging from entire layers to individual modules (e.g., self-attention) or specific operations (e.g., linear projections). To ensure stability after a split, the Adam optimizer states for the new specialized parameters are re-initialized using a warm-up update.

Experimental Results

The researchers evaluated PD on three major benchmarks: OPUS-100 (diverse languages), WMT (unbalanced data), and IWSLT’17 (many-to-many).

Performance Gains

The PD method consistently outperformed both standard multilingual baselines and individual bilingual models. On the OPUS-100 dataset, operation-level differentiation yielded average BLEU score improvements of +1.40 in one-to-many scenarios and +1.55 in many-to-one scenarios. Notably, on the WMT dataset, the method significantly boosted high-resource language performance (e.g., +2.21 BLEU for French-English) without compromising low-resource pairs like Estonian.

Linguistic Mirroring and Layer Specialization

A key finding is that the model’s self-organized sharing configuration naturally mirrors linguistic reality. As shown in the dendrogram below, the model tends to share more parameters between languages within the same family (e.g., Romance languages like French and Italian) than across different families.

Linguistic mirroring: The model's self-organized sharing configuration reflects language families over training steps.

Furthermore, the model exhibits layer-specific specialization. Differentiation occurs most frequently in the shallow layers of the encoder to handle diverse source language structures and in the deep layers of the decoder to manage the specifics of target language generation.

Conclusion and Future Work

Parameter Differentiation offers a scalable solution to the capacity bottleneck in MNMT by automatically balancing universal knowledge sharing with language-specific specialization. While the current implementation requires a manual upper bound on model size and introduces some computational overhead during gradient checks, it represents a significant step toward self-evolving neural architectures. Future research will explore PD's potential in zero-shot translation and incremental learning for adding new languages to existing models.

Summary updated

BibTeX
@inproceedings{wang2022parameter,
  author = {Qian Wang and Jiajun Zhang},
  title = {{Parameter Differentiation Based Multilingual Neural Machine Translation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i10.21396},
}
Figure