TL;DR
This paper introduces a novel "multi-level contrastive learning" (ML-CTL) framework to improve cross-lingual alignment in pre-trained language models. By jointly considering sentence-level and word-level semantic similarities, ML-CTL significantly boosts performance on zero-shot cross-lingual tasks. They also propose a "cross-zero NCE" (CZ-NCE) loss to mitigate training issues with small batch sizes.
What Problem Does This Paper Address?
Cross-lingual pre-trained models like BERT (mBERT) have revolutionized NLP by enabling models trained on one language (usually English) to perform well on others. However, current methods often rely on implicit cross-lingual alignment, primarily at the sentence level. While some approaches attempt word-level alignment, they can lose the rich contextual information from transformer models. This paper aims to bridge this gap by developing a method that explicitly integrates both sentence and word-level alignment within a single contrastive learning framework, leading to more robust cross-lingual understanding. Additionally, they address a practical challenge in contrastive learning: the need for large batch sizes, which can be computationally prohibitive.
Methods
The core innovation is the Multi-Level Contrastive Learning (ML-CTL) framework. It operates in two main stages:
- Word Positioned Sample (WPS) Construction: For each parallel sentence pair, they augment sentences with the start and end positions of words. This is done using a bilingual dictionary to ensure accurate word alignment.
- Concatenated Contextual Embedding (CCE) Extraction: A modified BERT encoder takes these WPSs and generates a "concatenated contextual embedding" (CCE). This CCE combines sentence-level representations (derived from averaging the first and last transformer layers) with word-level representations, capturing both sentence meaning and individual word context.
- Contrastive Learning: The ML-CTL framework then uses these CCEs in a contrastive learning setup. Positive pairs are CCEs from aligned sentences/words in different languages, while negative pairs are CCEs from unaligned sentences/words.
To address the issue of small batch sizes in contrastive learning, they introduce Cross-Zero NCE (CZ-NCE) loss. This loss function modifies the standard infoNCE loss by adjusting its lower bound, aiming to prevent the loss from collapsing to zero and becoming overly sensitive to floating-point errors, which is a common problem with small batches.
Datasets
- Training: The United Nations (UN) Parallel Corpus v1.0 is used for pre-training. They also construct a large dataset of 250 million parallel Word Positioned Samples (WPSs) using Google Translate.
- Evaluation: The Xtreme benchmark is used for evaluating zero-shot cross-lingual performance. Specific tasks include XNLI and PAWS-X (sentence classification), UD-POS and PANX (NER) (structured prediction), and BUCC and Tatoeba (retrieval).
Key Insights and Ideas
- Word-level alignment is crucial: Explicitly incorporating word-level alignment alongside sentence-level alignment significantly enhances cross-lingual understanding, going beyond what implicit methods or sentence-only contrastive learning can achieve.
- Contextual word embeddings matter: Using BERT's dynamic, contextual word embeddings (via CCE) is more effective for alignment than static word embeddings.
- CZ-NCE is a practical fix: The proposed CZ-NCE loss offers a viable solution for contrastive learning with limited computational resources (small batch sizes) by stabilizing training and mitigating floating-point errors.
- Visualizing alignment: t-SNE visualizations clearly demonstrate that ML-CTL-CZ leads to better disentanglement and separability of token embeddings across languages, indicating improved cross-lingual alignment.
Key Results
- The proposed ML-CTL-CZ model significantly outperforms strong baselines like mBERT, XLM, and MMTE on multiple zero-shot cross-lingual tasks in the Xtreme benchmark.
- For example, on the BUCC retrieval task, ML-CTL-CZ achieves a remarkable 78.4% F1 score, a substantial improvement over mBERT's 56.7%.
- Ablation studies confirm that both the multi-level approach (CCE) and the CZ-NCE loss contribute to performance gains. The CCE component provides a larger boost than the CZ-NCE loss alone, but CZ-NCE is shown to be effective in small batch settings.
Limitations and Future Work
- Computational Cost: While CZ-NCE helps with small batch sizes, contrastive learning, in general, can still be computationally intensive.
- Dictionary Reliance: The WPS construction relies on bilingual dictionaries, which might introduce noise or limitations if the dictionary is incomplete or inaccurate.
- Future Work: The paper suggests that this framework can be extended to other cross-lingual tasks and potentially improved with more sophisticated CCE extraction or contrastive loss functions. The authors demonstrate the effectiveness of their approach, paving the way for further research in multi-level alignment for cross-lingual NLP.
Summary updated