Multi-Level Contrastive Learning for Cross-Lingual Alignment

Beiduo Chen, Wu Guo, Bin Gu, Quan Liu, Yongchao Wang

TL;DR

This paper introduces a novel "multi-level contrastive learning" (ML-CTL) framework to improve cross-lingual alignment in pre-trained language models. By jointly considering sentence-level and word-level semantic similarities, ML-CTL significantly boosts performance on zero-shot cross-lingual tasks. They also propose a "cross-zero NCE" (CZ-NCE) loss to mitigate training issues with small batch sizes.

What Problem Does This Paper Address?

Cross-lingual pre-trained models like BERT (mBERT) have revolutionized NLP by enabling models trained on one language (usually English) to perform well on others. However, current methods often rely on implicit cross-lingual alignment, primarily at the sentence level. While some approaches attempt word-level alignment, they can lose the rich contextual information from transformer models. This paper aims to bridge this gap by developing a method that explicitly integrates both sentence and word-level alignment within a single contrastive learning framework, leading to more robust cross-lingual understanding. Additionally, they address a practical challenge in contrastive learning: the need for large batch sizes, which can be computationally prohibitive.

Methods

The core innovation is the Multi-Level Contrastive Learning (ML-CTL) framework. It operates in two main stages:

  1. Word Positioned Sample (WPS) Construction: For each parallel sentence pair, they augment sentences with the start and end positions of words. This is done using a bilingual dictionary to ensure accurate word alignment.
  2. Concatenated Contextual Embedding (CCE) Extraction: A modified BERT encoder takes these WPSs and generates a "concatenated contextual embedding" (CCE). This CCE combines sentence-level representations (derived from averaging the first and last transformer layers) with word-level representations, capturing both sentence meaning and individual word context.
  3. Contrastive Learning: The ML-CTL framework then uses these CCEs in a contrastive learning setup. Positive pairs are CCEs from aligned sentences/words in different languages, while negative pairs are CCEs from unaligned sentences/words.

To address the issue of small batch sizes in contrastive learning, they introduce Cross-Zero NCE (CZ-NCE) loss. This loss function modifies the standard infoNCE loss by adjusting its lower bound, aiming to prevent the loss from collapsing to zero and becoming overly sensitive to floating-point errors, which is a common problem with small batches.

Datasets

  • Training: The United Nations (UN) Parallel Corpus v1.0 is used for pre-training. They also construct a large dataset of 250 million parallel Word Positioned Samples (WPSs) using Google Translate.
  • Evaluation: The Xtreme benchmark is used for evaluating zero-shot cross-lingual performance. Specific tasks include XNLI and PAWS-X (sentence classification), UD-POS and PANX (NER) (structured prediction), and BUCC and Tatoeba (retrieval).

Key Insights and Ideas

  • Word-level alignment is crucial: Explicitly incorporating word-level alignment alongside sentence-level alignment significantly enhances cross-lingual understanding, going beyond what implicit methods or sentence-only contrastive learning can achieve.
  • Contextual word embeddings matter: Using BERT's dynamic, contextual word embeddings (via CCE) is more effective for alignment than static word embeddings.
  • CZ-NCE is a practical fix: The proposed CZ-NCE loss offers a viable solution for contrastive learning with limited computational resources (small batch sizes) by stabilizing training and mitigating floating-point errors.
  • Visualizing alignment: t-SNE visualizations clearly demonstrate that ML-CTL-CZ leads to better disentanglement and separability of token embeddings across languages, indicating improved cross-lingual alignment.

Key Results

  • The proposed ML-CTL-CZ model significantly outperforms strong baselines like mBERT, XLM, and MMTE on multiple zero-shot cross-lingual tasks in the Xtreme benchmark.
  • For example, on the BUCC retrieval task, ML-CTL-CZ achieves a remarkable 78.4% F1 score, a substantial improvement over mBERT's 56.7%.
  • Ablation studies confirm that both the multi-level approach (CCE) and the CZ-NCE loss contribute to performance gains. The CCE component provides a larger boost than the CZ-NCE loss alone, but CZ-NCE is shown to be effective in small batch settings.

Limitations and Future Work

  • Computational Cost: While CZ-NCE helps with small batch sizes, contrastive learning, in general, can still be computationally intensive.
  • Dictionary Reliance: The WPS construction relies on bilingual dictionaries, which might introduce noise or limitations if the dictionary is incomplete or inaccurate.
  • Future Work: The paper suggests that this framework can be extended to other cross-lingual tasks and potentially improved with more sophisticated CCE extraction or contrastive loss functions. The authors demonstrate the effectiveness of their approach, paving the way for further research in multi-level alignment for cross-lingual NLP.

Summary updated

Abstract

Cross-language pre-trained models such as multilingual BERT (mBERT) have achieved significant performance in various cross-lingual downstream NLP tasks. This paper proposes a multi-level contrastive learning (ML-CTL) framework to further improve the cross-lingual ability of pre-trained models. The proposed method uses translated parallel data to encourage the model to generate similar semantic embeddings for different languages. However, unlike the sentence-level alignment used in most previous studies, in this paper, we explicitly integrate the word-level information of each pair of parallel sentences into contrastive learning. Moreover, cross-zero noise contrastive estimation (CZ-NCE) loss is proposed to alleviate the impact of the floating-point error in the training process with a small batch size. The proposed method significantly improves the cross-lingual transfer ability of our basic model (mBERT) and outperforms on multiple zero-shot cross-lingual downstream tasks compared to the same-size models in the Xtreme benchmark.
BibTeX
@inproceedings{chen2022multi,
  author = {Beiduo Chen and Wu Guo and Bin Gu and Quan Liu and Yongchao Wang},
  title = {{Multi-Level Contrastive Learning for Cross-Lingual Alignment}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9747720},
}
Figure