NeighXLM: Enhancing Cross-Lingual Transfer in Low-Resource Languages via Neighbor-Augmented Contrastive Pretraining

Sicheng Wang, Wenyi Wu, Zibo ZhangEMNLP 2025

TL;DR

This paper introduces NeighXLM, a novel pretraining framework that significantly improves cross-lingual transfer for low-resource languages by augmenting contrastive learning with semantically similar neighbors mined from unlabeled corpora. This approach overcomes the limitations of scarce parallel data, leading to better performance on tasks like cross-lingual retrieval and zero-shot classification.

What Problem Does This Paper Address?

Modern NLP models excel on high-resource languages, but their performance plummets for low-resource languages due to a lack of labeled data. While previous methods have tried to bridge this gap using techniques like representation alignment and contrastive learning, they often struggle with the extreme scarcity of parallel data needed for effective cross-lingual supervision. Existing approaches that mine "pseudo-parallel" data from monolingual corpora can be unreliable, especially for typologically distant languages, as nearest neighbors in embedding space might not be true translations. This paper tackles the challenge of obtaining high-quality cross-lingual supervision for low-resource languages without relying on translation systems or human annotations.

Methods

NeighXLM builds upon contrastive learning for sentence representations. The core innovation lies in its neighbor-augmented input construction. For each source-target sentence pair (anchor-positive) in a small parallel dataset, the method retrieves k semantically similar "neighbor" sentences from a large unlabeled corpus in the target language. These neighbors are identified using sentence embeddings from a pre-trained multilingual encoder.

The training process uses a dual-encoder setup with a main encoder (Q) and a momentum encoder (K) for stability, similar to MoCo. The contrastive loss is applied to pull anchors closer to their positives and their mined neighbors, while pushing them away from negative samples stored in a dynamic queue. A key aspect is the neighbor-augmented contrastive objective, which uses a standard InfoNCE loss (without an additive margin) for the anchor-neighbor pairs. To handle potential noise from less reliable neighbors, an inverse rank-based weighting scheme is employed, giving more importance to higher-ranked neighbors. The overall training objective combines this neighbor-augmented loss with a basic contrastive loss and a queue-based contrastive loss.

Datasets

  • Training Data:
    • A small set of 2,048 parallel sentence pairs from the Tatoeba training set (simulating a low-resource setting).
  • 2 million Swahili (sw) sentences as unlabeled corpora for neighbor mining.
  • Evaluation Datasets:
    • Cross-Lingual Sentence Retrieval: Tatoeba and FLORES benchmarks.
  • Zero-Shot Classification: MasakhaNEWS dataset (covering 16 languages, evaluated on Swahili).
  • Zero-Shot Question Answering: KenSwQuAD and SD-QA (Swahili QA benchmarks).

Key Insights and Ideas

  • Leveraging Unlabeled Data is Crucial: The paper demonstrates that even with a small amount of parallel data, augmenting it with semantically similar sentences from large unlabeled corpora can significantly boost cross-lingual transfer.
  • Semantic Neighbors as Supervision: The core idea of using semantically similar sentences as positive examples in contrastive learning is effective, especially when true parallel data is scarce.
  • Robustness to Neighbor Quality: The inverse rank-based weighting mechanism is vital for mitigating the impact of potentially noisy or misaligned neighbors, making the approach robust.
  • Beyond Translation Pairs: NeighXLM shows that explicit translation pairs are not the only way to achieve cross-lingual alignment; semantic similarity within a language can be a powerful proxy.
  • Preservation of Multilingual Space: The method not only improves alignment for the target low-resource language but also appears to enhance or preserve representation quality for other languages.

Key Results

  • Cross-Lingual Retrieval: NeighXLM consistently outperforms the base encoder (InfoXLM) and a vanilla contrastive model across nearly all layers on both Tatoeba and FLORES benchmarks, showing significant gains in retrieval accuracy. For instance, on FLORES, NeighXLM (k=2) achieves a bi-directional average accuracy of 95.89%, compared to 93.78% for Vanilla Contrastive and 29.74% for the Base Encoder.
  • Zero-Shot Classification: NeighXLM (k=2) achieves the best performance in several categories on the MasakhaNEWS dataset, including technology (F1 = 0.711), and the highest macro-average F1 score of 0.741.
  • Zero-Shot Question Answering: The NeighXLM variant with k=2 achieves the best overall performance on KenSwQuAD and SD-QA, with the highest F1 and EM scores.
  • Representation Visualization: UMAP projections show that NeighXLM promotes semantic clustering across languages, rather than language-identity clustering, indicating better cross-lingual alignment.

Limitations and Future Work

  • Dependence on Unlabeled Corpora: The method still requires access to sufficient unlabeled corpora in the target language. For extremely low-resource languages with very limited monolingual data, its effectiveness might be reduced.
  • Simulated Low-Resource Setting: The experiments were conducted on Swahili, which, while low-resource and typologically distant, might have had some prior exposure in the base encoder's pretraining data. Genuinely low-resource languages with limited evaluation benchmarks were not used.
  • Future Work: The paper suggests exploring alternative negative queue interactions, though initial experiments with augmenting the loss to push neighbors away from negatives did not yield improvements. The authors also note the need to evaluate on languages with even scarcer data and fewer evaluation resources.

Summary updated

BibTeX
@inproceedings{wang2025neighxlm,
  author = {Sicheng Wang and Wenyi Wu and Zibo Zhang},
  title = {{NeighXLM: Enhancing Cross-Lingual Transfer in Low-Resource Languages via Neighbor-Augmented Contrastive Pretraining}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure