Nearest Neighbour Few-Shot Learning for Cross-lingual Classification

M. Saiful Bari, Batool Haider, Saab Mansour

Introduction

Massively pre-trained multilingual models like XLM-R have revolutionized NLP, enabling "zero-shot" transfer where a model trained in English can perform tasks in dozens of other languages. However, a significant performance gap remains between these zero-shot models and those with full supervision.

When developers have a small amount of data (e.g., 5 to 15 examples) for a specific language, the standard approach is fine-tuning. Unfortunately, updating millions of parameters on such a tiny dataset almost inevitably leads to overfitting, where the model memorizes noise and loses its general reasoning capabilities. Additionally, fine-tuning is computationally expensive and slow. This paper proposes a non-parametric alternative that is both more robust and significantly faster.

The NNFS Approach

The researchers developed Nearest Neighbor Few-shot Inference (NNFS). Instead of updating the model's weights, they treat few-shot learning as an inference-time similarity search. By keeping the model "frozen," they prevent it from forgetting its massive pre-training knowledge.

The NNFS pipeline: from source training to similarity-based classification.

The NNFS pipeline consists of four key steps:

  1. Source Training: The model is first fine-tuned on a high-resource language (typically English) for the target task.
  2. Feature Normalization: To account for the "cross-lingual shift" between languages, the data is centered by subtracting the mean representation of the source language and applying L2 normalization.
  3. Prototypical Rectification: A "prototype" vector is calculated for each class using the few labeled examples (the support set). These prototypes are further refined using the unlabeled data being classified (the query set) through transductive inference.
  4. Similarity Search: New sentences are classified by finding which class prototype is closest to the sentence's vector representation using cosine distance.

Experimental Setup

The approach was evaluated across 16 distinct languages using two major benchmarks:

  • XNLI (Cross-lingual Natural Language Inference): A 3-class logic task (entailment, neutral, contradiction) covering 15 languages.
  • PAWS-X (Paraphrase Adversaries from Word Scrambling): A binary classification task for paraphrase detection covering 7 languages.

Key Insights and Results

Performance and Consistency

NNFS consistently outperformed zero-shot baselines and traditional fine-tuning across nearly all 16 languages. On average, the method provided a +0.6 gain on XNLI and a +1.0 gain on PAWS-X compared to the strongest few-shot fine-tuning baselines.

Performance comparison on the XNLI dataset across 14 languages using 10-shot support sets.

Efficiency and Generalization

  • Speed: Because it avoids backpropagation, NNFS is roughly 38x to 40x faster than traditional few-shot fine-tuning, adding only a marginal overhead (~1.3x to 1.7x) to standard zero-shot inference.
  • Cross-Task Generalization: A surprising finding was that a model trained on XNLI (logic) could be adapted to PAWS-X (paraphrasing) using NNFS to achieve a +7.3 point gain over traditional fine-tuning. This suggests the embeddings capture high-level, task-agnostic semantic features.
  • Normalization: Simply subtracting the "mean" of the English training data proved to be a "secret sauce," significantly reducing language-specific bias and aligning target language embeddings.

Limitations and Future Work

While powerful, the method is currently limited to classification tasks; adapting this nearest-neighbor logic to generative tasks like translation remains an open challenge. Furthermore, performance can vary depending on the specific samples chosen for the support set. Future work may explore the "crossover point" where traditional fine-tuning might eventually overtake NNFS as more labeled data becomes available beyond the 15-sample threshold.

Summary updated

Abstract

Even though large pre-trained multilingual models (e.g. mBERT, XLM-R) have led to significant performance gains on a wide range of cross-lingual NLP tasks, success on many downstream tasks still relies on the availability of sufficient annotated data. Traditional fine-tuning of pre-trained models using only a few target samples can cause over-fitting. This can be quite limiting as most languages in the world are under-resourced. In this work, we investigate cross-lingual adaptation using a simple nearest neighbor few-shot (<15 samples) inference technique for classification tasks. We experiment using a total of 16 distinct languages across two NLP tasks- XNLI and PAWS-X. Our approach consistently improves traditional fine-tuning using only a handful of labeled samples in target locales. We also demonstrate its generalization capability across tasks.
BibTeX
@inproceedings{bari2021nearest,
  author = {M. Saiful Bari and Batool Haider and Saab Mansour},
  title = {{Nearest Neighbour Few-Shot Learning for Cross-lingual Classification}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=HM1Plgqczc},
}
Figure