Instance-based Transfer Learning for Multilingual Deep Retrieval

Andrew O. Arnold, William W. CohenarXiv 2019 · OpenReview

Abstract

We focus on the problem of search in the multilingual setting. Examining the problems of next-sentence prediction and inverse cloze, we show that at large scale, instance-based transfer learning is surprisingly effective in the multilingual setting, leading to positive transfer on all of the 35 target languages and two tasks tested. We analyze this improvement and argue that the most natural explanation, namely direct vocabulary overlap between languages, only partially explains the performance gains: in fact, we demonstrate target-language improvement can occur after adding data from an auxiliary language even with no vocabulary in common with the target. This surprising result is due to the effect of transitive vocabulary overlaps between pairs of auxiliary and target languages.

Introduction

Deep retrieval—the use of neural networks to identify semantically relevant documents—is a cornerstone of modern Information Retrieval (IR). However, scaling these systems to dozens of languages typically involves complex engineering, such as training dozens of individual models or employing sophisticated fine-tuning regimes. This research investigates a simpler alternative: instance-based transfer learning. By pooling data from 35 diverse languages into a single training set, we evaluate whether a unified model can outperform specialized per-language counterparts.

Methodology

The study utilizes a Siamese Network architecture to learn embeddings for query-context pairs. To maintain scalability, the model employs a "bag-of-n-grams" approach with a shared vocabulary of 350 million unigrams and bigrams. Two self-supervised tasks were evaluated using a massive Wikipedia dataset (720 million sentence pairs):

  1. Next-Sentence Prediction (NSP): Identifying the correct subsequent sentence from a pool.
  2. Inverse Cloze (IC): Predicting the surrounding context of a given sentence.

The core experiment compares the performance of models trained on single languages against a "combined" model trained on all 35 languages simultaneously.

Key Results: Universal Positive Transfer

The most significant finding is the absence of "negative transfer." In many multilingual settings, interference between languages can degrade performance. Here, every single one of the 35 languages—from high-resource English to low-resource Telugu—showed improvement when trained in the combined model.

Relative improvement at NSP (left) and IC (right) tasks across target languages.

As shown in the figure above, the harder Inverse Cloze (IC) task saw substantially higher relative gains (averaging 96%) compared to Next-Sentence Prediction (31%). Low-resource languages naturally saw the largest boosts, with some improvements exceeding 200%.

Analysis: The Power of Transitive Overlap

Why does pooling data work so well? While direct vocabulary overlap (shared words like "computer") is a factor, it does not fully explain the gains. The researchers discovered a "transitive overlap" effect: if Language A shares vocabulary with Language B, and Language B shares with Language C, Language A can benefit from Language C's data even if they share no words directly. Language B acts as a semantic pivot.

Vocabulary overlap heatmap highlighting language clusters, including the Lsjbot-driven cluster (green box).

This phenomenon is clearly visible in the vocabulary overlap heatmap. A notable anomaly was found in the high overlap between Swedish (sv) and Philippine languages like Cebuano (ceb) and Waray (war) (the green box). This was traced to "Lsjbot," an automated bot that generated millions of Wikipedia articles across these specific languages using similar templates, creating a massive artificial vocabulary bridge.

Conclusion and Future Work

Instance-based transfer learning proves to be a remarkably simple yet powerful tool for multilingual retrieval. By leveraging transitive vocabulary overlaps, models can effectively "borrow" knowledge from distant languages. Future research should explore whether these gains persist when using more modern subword tokenization (like BPE) and how these improvements translate to downstream tasks like production web search.

Summary updated

Abstract

We focus on the problem of search in the multilingual setting. Examining the problems of next-sentence prediction and inverse cloze, we show that at large scale, instance-based transfer learning is surprisingly effective in the multilingual setting, leading to positive transfer on all of the 35 target languages and two tasks tested. We analyze this improvement and argue that the most natural explanation, namely direct vocabulary overlap between languages, only partially explains the performance gains: in fact, we demonstrate target-language improvement can occur after adding data from an auxiliary language even with no vocabulary in common with the target. This surprising result is due to the effect of transitive vocabulary overlaps between pairs of auxiliary and target languages.
BibTeX
@misc{arnold2019instance,
  author = {Andrew O. Arnold and William W. Cohen},
  title = {{Instance-based Transfer Learning for Multilingual Deep Retrieval}},
  year = {2019},
  eprint = {1911.06111},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1911.06111},
}
Figure