FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding

Yuwei Fang, Shuohang Wang, Zhe Gan, Siqi Sun, Jingjing Liu

Abstract

Large-scale cross-lingual language models (LM), such as mBERT, Unicoder and XLM, have achieved great success in cross-lingual representation learning. However, when applied to zero-shot cross-lingual transfer tasks, most existing methods use only single-language input for LM finetuning, without leveraging the intrinsic cross-lingual alignment between different languages that proves essential for multilingual tasks. In this paper, we propose FILTER, an enhanced fusion method that takes cross-lingual data as input for XLM finetuning. Specifically, FILTER first encodes text input in the source language and its translation in the target language independently in the shallow layers, then performs cross-language fusion to extract multilingual knowledge in the intermediate layers, and finally performs further language-specific encoding. During inference, the model makes predictions based on the text input in the target language and its translation in the source language. To tackle the issue of noisy labels in complex tasks like question answering and POS tagging, we further propose an additional KL-divergence self-teaching loss for model training, based on auto-generated soft pseudo-labels. Extensive experiments demonstrate that FILTER achieves new state of the art on two challenging multilingual multi-task benchmarks, XTREME and XGLUE.

Introduction

While large-scale multilingual models like mBERT and XLM-R have significantly advanced the field, they often struggle with a "domain gap" when transferring knowledge from high-resource languages (typically English) to low-resource ones. The common baseline approach, translate-train, involves translating English training data into a target language and treating the result as independent training samples.

However, this approach suffers from two primary limitations. First, it fails to leverage the intrinsic semantic alignment between a sentence and its translation by processing them separately. Second, it introduces significant label noise for complex tasks such as Named Entity Recognition (NER) or Question Answering (QA), where word order changes or span shifts in translation can render the original English labels inaccurate for the target language text.

The FILTER Framework

To address these challenges, we introduce FILTER (Fusion in the Intermediate Layers of TransformER), a novel architecture designed to explicitly model cross-lingual alignments during the fine-tuning process.

1. Architecture Design

Unlike standard Transformer stacks that process single-language inputs, FILTER adopts a "sandwich" architecture to facilitate deep cross-lingual fusion:

  • Local Layers ($m$ layers): The source text and its translation are encoded independently. This allows the model to capture language-specific features and nuances before any interaction occurs.
  • Cross-lingual Fusion Layers ($k$ layers): The representations from both languages are concatenated and passed through shared layers. This stage is critical as it allows the model to learn semantic alignments between the two languages.
  • Domain-specific Layers ($24-m-k$ layers): The model returns to processing languages individually to prepare for final task-specific predictions.
Comparison of standard translate-train, simple concatenation, and the proposed FILTER architecture.

2. Self-Teaching Loss

To mitigate the impact of noisy labels in translated data, we employ a teacher-student framework. A teacher FILTER model is first trained on a mix of "clean" source labels and potentially "noisy" transferred labels. Subsequently, a student model is trained not just on hard labels, but by matching the soft pseudo-labels (probability distributions) generated by the teacher. This self-teaching mechanism acts as a regularizer, encouraging the model to focus on core semantic features rather than translation artifacts.

Experimental Results and Analysis

FILTER was evaluated on the XTREME and XGLUE benchmarks, covering a wide array of tasks including classification, structured prediction, and retrieval across dozens of languages.

State-of-the-Art Performance

FILTER achieved significant improvements over existing baselines:

  • XTREME: Reached an average score of 77.0, an 8.8-point absolute improvement over the XLM-R baseline.
  • XGLUE: Achieved an average score of 80.1, outperforming the Unicoder baseline by 4.0 points.

The "Goldilocks" Fusion Insight

A key finding of our research is that the depth of fusion matters significantly and varies by task complexity. We observed that fusing languages too early (at the input) or too late (at the output) is sub-optimal.

Performance (F1 or Accuracy) across different tasks as a function of fusion layers (k) and local layers (m).

As shown in the figures above, simple classification tasks (like PAWS-X) benefit from fewer fusion layers. In contrast, complex tasks like Part-of-Speech (POS) tagging and Question Answering (MLQA) require a more balanced configuration of local and fusion layers to handle the intricate linguistic details before alignment.

Conclusion

FILTER demonstrates that explicit intermediate-layer fusion combined with a self-teaching loss can drastically reduce the cross-lingual transfer gap. By moving beyond simple data augmentation to structural cross-lingual alignment, FILTER sets a new standard for multilingual language understanding. Future work will explore automated methods for discovering optimal layer configurations ($m$ and $k$) for diverse tasks and reducing the computational overhead of the teacher-student training cycle.

Summary updated

Abstract

Large-scale cross-lingual language models (LM), such as mBERT, Unicoder and XLM, have achieved great success in cross-lingual representation learning. However, when applied to zero-shot cross-lingual transfer tasks, most existing methods use only single-language input for LM finetuning, without leveraging the intrinsic cross-lingual alignment between different languages that proves essential for multilingual tasks. In this paper, we propose FILTER, an enhanced fusion method that takes cross-lingual data as input for XLM finetuning. Specifically, FILTER first encodes text input in the source language and its translation in the target language independently in the shallow layers, then performs cross-language fusion to extract multilingual knowledge in the intermediate layers, and finally performs further language-specific encoding. During inference, the model makes predictions based on the text input in the target language and its translation in the source language. For simple tasks such as classification, translated text in the target language shares the same label as the source language. However, this shared label becomes less accurate or even unavailable for more complex tasks such as question answering, NER and POS tagging. To tackle this issue, we further propose an additional KL-divergence self-teaching loss for model training, based on auto-generated soft pseudo-labels for translated text in the target language. Extensive experiments demonstrate that FILTER achieves new state of the art on two challenging multilingual multi-task benchmarks, XTREME and XGLUE.
BibTeX
@inproceedings{fang2021filter,
  author = {Yuwei Fang and Shuohang Wang and Zhe Gan and Siqi Sun and Jingjing Liu},
  title = {{FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=BEUxu1YbKP},
}
Figure