Abstract: Large-scale cross-lingual language models (LM), such as mBERT, Unicoder and XLM, have achieved great success in cross-lingual representation learning. However, when applied to zero-shot cross-lingual transfer tasks, most existing methods use only single-language input for LM finetuning, without leveraging the intrinsic cross-lingual alignment between different languages that proves essential for multilingual tasks. In this paper, we propose FILTER, an enhanced fusion method that takes cross-lingual data as input for XLM finetuning. Specifically, FILTER first encodes text input in the source language and its translation in the target language independently in the shallow layers, then performs cross-language fusion to extract multilingual knowledge in the intermediate layers, and finally performs further language-specific encoding. During inference, the model makes predictions based on the text input in the target language and its translation in the source language. For simple tasks such as classification, translated text in the target language shares the same label as the source language. However, this shared label becomes less accurate or even unavailable for more complex tasks such as question answering, NER and POS tagging. To tackle this issue, we further propose an additional KL-divergence self-teaching loss for model training, based on auto-generated soft pseudo-labels for translated text in the target language. Extensive experiments demonstrate that FILTER achieves new state of the art on two challenging multilingual multi-task benchmarks, XTREME and XGLUE.
Introduction
In Natural Language Processing (NLP), large-scale models like mBERT or XLM-R perform exceptionally well in high-resource languages like English but often struggle with low-resource languages. The standard approach to bridge this gap is zero-shot cross-lingual transfer, where a model trained on English is expected to transfer its knowledge to other languages. A common technique, "translate-train," involves translating English training data to augment the dataset. However, most existing methods process these languages in isolation during fine-tuning, ignoring the intrinsic alignment—the "connective tissue"—between the source text and its translation. Furthermore, for complex tasks like Question Answering (QA) or Named Entity Recognition (NER), labels often do not survive translation perfectly, leading to noisy training data.
Methods
The authors propose FILTER (Fusion in the Intermediate Layers of TransformER), which introduces two major innovations to address these challenges.
The FILTER Architecture
Instead of simple concatenation or independent processing, FILTER utilizes a three-stage encoding approach:
- Local Layers ($m$): The source text and its translation are encoded independently in the shallow layers to capture language-specific nuances.
- Fusion Layers ($k$): The model performs cross-language fusion in the middle layers, allowing representations to align and share conceptual knowledge.
- Language-specific Layers: The top layers focus back on the specific language needed for the final prediction to avoid introducing excessive translation noise.
Self-Teaching Loss
To solve the "noisy label" problem, FILTER employs a teacher-student setup. A "teacher" model is first trained on the data to generate soft pseudo-labels (probability distributions) for the translated text. A "student" model is then trained using a KL-divergence loss to match these distributions. This is particularly effective for NER and QA, where exact label positions might shift during translation, but the underlying semantic signal remains captured in the soft distribution.
Results and Analysis
FILTER was evaluated on the XTREME and XGLUE benchmarks, covering tasks ranging from sentence classification to structured prediction and QA. FILTER achieved an average score of 77.0 on XTREME, outperforming the XLM-R baseline by 8.8 points. On XGLUE, it reached a score of 80.1, a 4.0-point improvement over the Unicoder baseline.
The "Goldilocks Zone" of Fusion
A key insight from the study is that the optimal amount of fusion varies by task. By adjusting the number of local ($m$) and fusion ($k$) layers, the researchers found that complex tasks like QA benefit from more fusion layers, whereas simpler classification tasks perform better with more independent "local" processing.
As shown in the performance plots, structured prediction tasks like POS tagging particularly benefited from the self-teaching loss, which improved performance by 0.7 to 1.0 points by effectively handling label noise.
Conclusion
FILTER demonstrates that leveraging cross-lingual alignment through intermediate fusion and soft-label self-teaching significantly enhances zero-shot transfer. While the current architecture requires manual tuning of the layer configuration ($m$ and $k$) for different tasks, future work may involve automated architecture searches to further streamline the deployment of high-performance multilingual models.
Summary updated