Abstract
Multilingual neural machine translation (MNMT) provides significant efficiency gains but often suffers from performance degradation compared to bilingual models due to parameter interference. This paper introduces CIAT (Counter-Interference Adapter for Translation), a framework utilizing pluggable embedding and parallel layer adapters to "de-noise" language-specific representations. By addressing semantic conflation in shared embeddings and negative interference in shared layers, CIAT achieves consistent BLEU score improvements across diverse language pairs with minimal parameter overhead. Our analysis highlights the superiority of parallel adapter placement and the high sensitivity of the decoder to these adaptations.
Introduction
While training a single model for multiple languages is desirable for deployment, it often leads to a performance gap, especially for resource-rich languages. This degradation is primarily attributed to two factors: Multilingual Embedding Deficiency, where shared embeddings conflate meanings across languages, and Multilingual Interference Effects, where shared layers struggle with conflicting language features.
The CIAT Framework
CIAT addresses these issues by adding lightweight, language-specific adapters to a frozen, pre-trained multilingual Transformer base.
Embedding Adapter
To resolve semantic conflation without the cost of fine-tuning massive embedding matrices, CIAT uses a small bottleneck feed-forward network as an embedding adapter. This component approximates a fine-tuned matrix, allowing for language-specific semantic adjustments with very few parameters.
Parallel De-noise Layer Adapter
A key innovation in CIAT is the use of parallel layer adapters. Unlike traditional serial adapters that follow Transformer sub-layers, these are placed in parallel, taking the same input and adding their output to the sub-layer's result. This design acts as a pre-layer de-noising mechanism, stripping away multilingual interference.
Experimental Setup
The authors evaluated CIAT on three standard benchmarks:
- IWSLT: 8 languages $\leftrightarrow$ English (TED talks).
- OPUS-100: 20 selected language pairs (17 high-resource, 3 low-resource) from a 99-language corpus.
- WMT: 5 standard language pairs.
Analysis and Key Insights
Parallel vs. Serial Configuration
The choice of parallel over serial placement is a defining feature of CIAT.
The parallel connection is more intuitive for multilingual MT as it directly counteracts the bias in shared representations. Empirical results confirm its superiority over serial configurations.
Decoder Sensitivity and Directional Asymmetry
Ablation studies revealed that adapters in the decoder—particularly the upper layers—have a significantly larger impact on translation quality than those in the encoder. Furthermore, CIAT yields larger improvements when translating from English to other languages ($en \rightarrow any$). This is likely because English is over-represented in the training data, making the baseline generation quality for English harder to improve upon.
Resolving Semantic Conflation
By measuring the Average Cosine Similarity (ACS) of cross-lingual synonyms, the authors demonstrated that the embedding adapter successfully pulls words with similar meanings closer together in the latent space, effectively mitigating the embedding deficiency.
Results
CIAT consistently outperformed strong baselines, including standard multilingual models and serial adapter models. It improved BLEU scores in 64 out of 66 evaluated language directions, with 42 directions seeing gains greater than 0.5 BLEU. The model demonstrates a favorable parameter-performance trade-off, where increasing layer adapter size yields reliable gains, while embedding adapters remain efficient even at smaller scales.
Conclusion
CIAT offers a robust and efficient solution for improving multilingual machine translation. By employing parallel de-noising adapters and targeted embedding adaptation, it mitigates the interference inherent in shared models. Future work may explore dynamic routing or applying this parallel architecture to other multi-task NLP domains.
Summary updated