Multi-Granularity Contrasting for Cross-Lingual Pre-Training

Shicheng Li, Pengcheng Yang, Fuli Luo, Jun XieACL 2021 · DOI

TL;DR

This paper introduces Multi-Granularity Contrasting (MGC), a novel framework for cross-lingual pre-training that combines the strengths of traditional embedding alignment and modern masked language modeling. MGC enhances language-universal representations by explicitly aligning semantic equivalents across languages at multiple granularities (word, span, and sentence), leading to significant improvements in downstream tasks like machine translation and cross-lingual understanding.

What Problem Does This Paper Address?

Cross-lingual pre-training aims to equip models with representations that work across multiple languages. Two dominant approaches exist: traditional embedding alignment, which forces semantically equivalent words/phrases to have similar vector representations, and modern language modeling (like XLM), which leverages bidirectional contexts but relies solely on shared subword vocabularies for cross-lingual information. The latter lacks the fine-grained supervision of embedding alignment. This paper addresses this gap by proposing a method that integrates both paradigms, aiming for richer, more explicit cross-lingual understanding without sacrificing the benefits of contextualized representations.

Methods

The core of this work is the Multi-Granularity Contrasting (MGC) framework. It builds upon the standard masked language modeling (MLM) objective, which predicts masked tokens based on their context. The novelty lies in adding contrastive learning objectives that explicitly encourage semantic equivalents across languages to have similar representations, while pushing apart non-equivalents.

MGC operates at three granularities:

  • Word-Level Contrasting: This aims to align individual words. Two strategies are proposed:
  • Hard Contrasting: Uses external word aligners to create pseudo-parallel word pairs.
  • Soft Contrasting: Leverages multi-head attention to implicitly learn word alignments, avoiding reliance on external tools and handling non-one-to-one alignments.
  • Span-Level Contrasting: Extends the idea to contiguous sequences of words (spans). It uses statistical machine translation tools to identify pseudo-parallel spans and applies a similar contrastive loss.
  • Sentence-Level Contrasting: Focuses on aligning entire sentences. To mitigate the issue of overly distinct positive and negative samples, they employ back-translation to introduce noise into the positive sentence pairs, creating more competitive training signals.

Datasets

The paper utilizes large-scale datasets for pre-training and standard benchmarks for evaluation.

  • Pre-training:
    • 15 languages from the XNLI dataset (English, French, Spanish, German, Greek, Bulgarian, Russian, Turkish, Arabic, Vietnamese, Thai, Chinese, Hindi, Swahili, and Urdu).
  • Monolingual data from Common-Crawl Corpus.
  • Bilingual data from the OPUS website.
  • Evaluation:
    • Machine Translation: IWSLT14 (De-En), WMT14 (En-De, En-Fr).
  • Cross-Lingual Language Understanding: XNLI (15 languages).

Key Insights and Ideas

  • Bridging the Gap: The paper successfully bridges the gap between traditional embedding alignment and modern language modeling by combining their strengths. This is crucial for achieving truly language-universal representations.
  • Multi-Granularity is Key: Applying contrastive learning at word, span, and sentence levels provides complementary signals. The paper shows that each granularity contributes, with sentence-level contrasting having a significant impact.
  • Back-Translation for Robustness: Using back-translation for sentence-level contrasting is an effective way to alleviate the vanishing gradient problem caused by overly distinct positive/negative samples, leading to more robust training.
  • Soft Contrasting's Advantage: Soft contrasting for word alignment is shown to be more flexible than hard contrasting, as it doesn't rely on external aligners and can handle more complex alignment scenarios.

Key Results

  • XNLI Performance: MGC significantly improves average accuracy on the XNLI benchmark. Under the cross-lingual transfer setting, it boosts the average accuracy of their implemented XLM-R from 74.4% to 76.0%.
  • Machine Translation Performance: MGC achieves state-of-the-art results on machine translation tasks. For instance, it shows a 1.6% BLEU score improvement over the base XLM on the IWSLT14 DE-EN task and outperforms mBERT and MASS on multiple benchmarks.
  • Ablation Studies: Ablation studies confirm the contribution of each contrastive loss component. Removing sentence-level contrasting leads to a 1.3 BLEU drop on WMT14 EN-DE, highlighting its importance. Augmenting sentence-level contrasting with back-translation yields a 0.7 BLEU improvement.

Limitations and Future Work

  • Hard vs. Soft Contrasting: The paper notes that there isn't a definitive conclusion on the superiority of hard versus soft contrasting for word-level alignment, suggesting further exploration might be beneficial.
  • External Aligners: While soft contrasting mitigates the need for external aligners, hard contrasting still relies on them, which can introduce errors.
  • Future Directions: The authors suggest that their framework can be applied to various downstream tasks, implying potential for further research in applying MGC to other cross-lingual NLP problems. The success of multi-granularity contrasting also opens avenues for exploring even finer-grained or different types of semantic units.

Summary updated

BibTeX
@inproceedings{li2021multi,
  author = {Shicheng Li and Pengcheng Yang and Fuli Luo and Jun Xie},
  title = {{Multi-Granularity Contrasting for Cross-Lingual Pre-Training}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.findings-acl.149},
}
Figure