Revisiting k-NN for Fine-Tuning Pre-trained Language Models

Lei Li, Jing Chen, Botzhong Tian, Ningyu Zhang

Introduction

Modern Natural Language Processing (NLP) is dominated by Pre-trained Language Models (PLMs) such as BERT and RoBERTa. These models are characterized as "eager learners" because, during the fine-tuning process, they attempt to compress the entire training dataset into a fixed set of model parameters. While this approach has led to state-of-the-art results, it is not without flaws: PLMs are often prone to overfitting, sensitive to noisy data, and demand significant computational resources.

In contrast, $k$-Nearest Neighbor ($k$-NN) classifiers represent a "lazy learning" paradigm. They do not possess a traditional training phase; instead, they store training data and make predictions by retrieving the most similar examples at inference time. While $k$-NN is robust to noise and avoids parameter overfitting, it lacks the deep semantic understanding inherent in PLMs. This paper proposes a hybrid framework, UNION-ALL, which effectively fuses these two paradigms to create more robust and efficient classifiers.

The UNION-ALL Framework

The UNION-ALL method integrates $k$-NN into the PLM pipeline through two distinct stages: calibrated training and inference interpolation.

1. $k$-NN-Calibrated Training

Rather than treating all training examples with equal importance, the authors use $k$-NN as a "difficulty indicator." By analyzing PLM embeddings, the system determines if a $k$-NN classifier struggles to correctly categorize a specific training instance. If it does, that instance is labeled as "hard." Inspired by Focal Loss, the training loss is then reweighted to force the PLM to focus its learning capacity on these challenging examples, thereby refining its internal decision boundaries.

2. Inference Interpolation

During the testing phase, the model does not rely exclusively on the PLM’s parametric output. Instead, it calculates a probability distribution based on the $k$-nearest neighbors retrieved from the training set. This non-parametric distribution is then linearly interpolated with the PLM’s predicted distribution to produce the final output. To represent text for the $k$-NN search, the model utilizes the $l_2$-normalized embedding of the [CLS] token from the PLM's final layer.

Experimental Setup

The researchers evaluated UNION-ALL across eight diverse NLP benchmarks:

  • Sentiment Analysis: SST-5
  • Question Classification: TREC
  • Natural Language Inference (NLI): MNLI, QNLI, and CB
  • Question Answering (QA): BoolQ
  • Relation Extraction: SemEval and TACREV

These tasks were tested in three distinct environments: full-supervised (entire dataset), few-shot (16 examples per class), and zero-shot (no training data).

Key Insights and Results

The UNION-ALL method consistently outperformed both vanilla fine-tuning (FT) and prompt-tuning (PT) across nearly all evaluated settings.

  • Few-Shot Performance: The benefits were most pronounced in data-scarce environments. For instance, on the MNLI task, UNION-ALL achieved an accuracy of 51.7%, a significant improvement over the vanilla fine-tuning score of 44.4%.
  • Zero-Shot Gains: In zero-shot scenarios, simply applying the inference interpolation (UNION-INF) led to massive performance jumps. On the TREC task, accuracy soared from 22.6% to 82.4%, highlighting the power of non-parametric retrieval when no model training is possible.
  • Full Supervision: Even with ample data, the hybrid approach provided a steady boost, raising the average score across all eight tasks from 82.9 to 83.8.
  • Robustness: The authors found that the choice of similarity metric (Euclidean distance vs. Cosine similarity) had a negligible impact, suggesting the framework is robust to the specific definition of "closeness."

Limitations and Future Directions

Despite its performance gains, the authors identify several practical hurdles for deployment:

  • Memory Overhead: Maintaining a "datastore" of all training features for retrieval can be memory-intensive for large-scale datasets.
  • Hyperparameter Sensitivity: The model introduces new variables, such as the interpolation weight ($\lambda$) and the number of neighbors ($k$), which require careful tuning.
  • Search Efficiency: As the datastore grows, the time required for $k$-NN retrieval increases, potentially impacting real-time inference speeds.

Looking forward, the authors suggest that this approach could be particularly valuable for Large Language Models (LLMs). By retrieving factual information from external databases during the reasoning process, a $k$-NN-augmented LLM might significantly reduce the occurrence of "hallucinations."

Summary updated

BibTeX
@inproceedings{li2023revisiting,
  author = {Lei Li and Jing Chen and Botzhong Tian and Ningyu Zhang},
  title = {{Revisiting k-NN for Fine-Tuning Pre-trained Language Models}},
  booktitle = {CCL},
  year = {2023},
  url = {https://openreview.net/forum?id=moA8CoF5Xb},
}
Figure