Serial or Parallel? Plug-able Adapter for multilingual machine translation

Yaoming Zhu, Jiangtao Feng, Chengqi Zhao, Mingxuan Wang, Lei Li

Abstract

Multilingual neural machine translation (MNMT) provides significant efficiency gains but often suffers from performance degradation compared to bilingual models due to parameter interference. This paper introduces CIAT (Counter-Interference Adapter for Translation), a framework utilizing pluggable embedding and parallel layer adapters to "de-noise" language-specific representations. By addressing semantic conflation in shared embeddings and negative interference in shared layers, CIAT achieves consistent BLEU score improvements across diverse language pairs with minimal parameter overhead. Our analysis highlights the superiority of parallel adapter placement and the high sensitivity of the decoder to these adaptations.

Introduction

While training a single model for multiple languages is desirable for deployment, it often leads to a performance gap, especially for resource-rich languages. This degradation is primarily attributed to two factors: Multilingual Embedding Deficiency, where shared embeddings conflate meanings across languages, and Multilingual Interference Effects, where shared layers struggle with conflicting language features.

The CIAT Framework

CIAT addresses these issues by adding lightweight, language-specific adapters to a frozen, pre-trained multilingual Transformer base.

The CIAT architecture featuring embedding adapters and parallel layer adapters within the encoder and decoder blocks.

Embedding Adapter

To resolve semantic conflation without the cost of fine-tuning massive embedding matrices, CIAT uses a small bottleneck feed-forward network as an embedding adapter. This component approximates a fine-tuned matrix, allowing for language-specific semantic adjustments with very few parameters.

Parallel De-noise Layer Adapter

A key innovation in CIAT is the use of parallel layer adapters. Unlike traditional serial adapters that follow Transformer sub-layers, these are placed in parallel, taking the same input and adding their output to the sub-layer's result. This design acts as a pre-layer de-noising mechanism, stripping away multilingual interference.

Experimental Setup

The authors evaluated CIAT on three standard benchmarks:

  • IWSLT: 8 languages $\leftrightarrow$ English (TED talks).
  • OPUS-100: 20 selected language pairs (17 high-resource, 3 low-resource) from a 99-language corpus.
  • WMT: 5 standard language pairs.

Analysis and Key Insights

Parallel vs. Serial Configuration

The choice of parallel over serial placement is a defining feature of CIAT.

Comparison between (a) Serial and (b) Parallel adapter configurations. The parallel design acts as a low-rank patch to de-noise biased multilingual inputs.

The parallel connection is more intuitive for multilingual MT as it directly counteracts the bias in shared representations. Empirical results confirm its superiority over serial configurations.

Decoder Sensitivity and Directional Asymmetry

Ablation studies revealed that adapters in the decoder—particularly the upper layers—have a significantly larger impact on translation quality than those in the encoder. Furthermore, CIAT yields larger improvements when translating from English to other languages ($en \rightarrow any$). This is likely because English is over-represented in the training data, making the baseline generation quality for English harder to improve upon.

Resolving Semantic Conflation

By measuring the Average Cosine Similarity (ACS) of cross-lingual synonyms, the authors demonstrated that the embedding adapter successfully pulls words with similar meanings closer together in the latent space, effectively mitigating the embedding deficiency.

Results

CIAT consistently outperformed strong baselines, including standard multilingual models and serial adapter models. It improved BLEU scores in 64 out of 66 evaluated language directions, with 42 directions seeing gains greater than 0.5 BLEU. The model demonstrates a favorable parameter-performance trade-off, where increasing layer adapter size yields reliable gains, while embedding adapters remain efficient even at smaller scales.

Conclusion

CIAT offers a robust and efficient solution for improving multilingual machine translation. By employing parallel de-noising adapters and targeted embedding adaptation, it mitigates the interference inherent in shared models. Future work may explore dynamic routing or applying this parallel architecture to other multi-task NLP domains.

Summary updated

BibTeX
@inproceedings{zhu2021serial,
  author = {Yaoming Zhu and Jiangtao Feng and Chengqi Zhao and Mingxuan Wang and Lei Li},
  title = {{Serial or Parallel? Plug-able Adapter for multilingual machine translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.findings-emnlp.240},
}
Figure