Plug and Play Knowledge Distillation for kNN-LM with External Logits

Xuyang Jin, Tao Ge, Furu WeiAACL/IJCNLP 2022 · OpenReview

TL;DR

This paper introduces "Plug and Play Knowledge Distillation" (PP-KD), a novel method to improve kNN-Language Models (kNN-LMs) for causal language modeling. PP-KD leverages external "logits" (probability distributions over the vocabulary) from a teacher LM during inference, offering a significant performance boost, especially for smaller models, without requiring retraining.

What Problem Does This Paper Address?

Knowledge Distillation (KD) has been highly successful in various Natural Language Processing (NLP) tasks, like natural language understanding (NLU) and sequence-to-sequence (seq2seq) modeling. However, applying KD to causal language modeling (LM) has been challenging, with even standard logit-based KD methods showing only marginal improvements in perplexity. The authors aim to bridge this gap by enhancing state-of-the-art kNN-LMs, which are powerful for causal LM but still struggle with distillation. They propose a new KD approach that is "plug and play," meaning it can be easily integrated and removed during inference without modifying the student model's training process.

Methods

The core innovation is Plug and Play Knowledge Distillation (PP-KD). Unlike traditional KD where teacher knowledge is embedded during student training, PP-KD stores teacher knowledge externally as logits.

Here's how it works:

  • Datastore Augmentation: A standard kNN-LM builds a datastore containing retrieved training contexts (K) and their hard-label targets (V). PP-KD augments this by adding a third component: the teacher's logits (U) for those same training contexts.
  • Inference-Time Enhancement: During inference, after retrieving the k nearest neighbors (contexts), PP-KD uses both their hard-label targets (from V) and their associated teacher logits (from U).
  • Logit Aggregation: The probability distribution for the next token is computed by combining the probabilities derived from the hard-label targets (Phard-kNN) and the teacher's logits (Plogit-kNN). This is done via a weighted average: PkNN = μ * Phard-kNN + (1 - μ) * Plogit-kNN.
  • Plug and Play: The key is that this process happens during inference. If you want to disable PP-KD, you simply set μ = 1.0, reverting to the vanilla kNN-LM. You can also switch teachers by swapping out the U component of the datastore.
  • Teacher Types: The authors explore two types of teachers:
  • Homogeneous Teacher: A powerful causal LM (like GPT-2).
  • Heterogeneous Teacher: A masked LM (like BERT). They argue that masked LMs' bidirectional context provides informative logits without leaking test-time information. They also show that mixing logits from both types of teachers can further improve performance.

Datasets

The experiments were conducted on the WIKITEXT-103 dataset, a standard benchmark for language modeling. They also used BookCorpus for some experiments. These are existing datasets, and the paper focuses on improving the LM's performance on them rather than introducing new benchmarks.

Key Insights and Ideas

  • Logits are Key for Causal LM Distillation: The paper highlights that while standard KD struggles with causal LM perplexity, leveraging external logits from a teacher model can provide a significant boost.
  • "Plug and Play" is a Game Changer: The ability to apply distillation benefits at inference time without retraining is a major practical advantage. This makes it easy to experiment with different teachers or disable the distillation if needed.
  • Heterogeneous Teachers Offer Unique Value: Using logits from masked LMs (which have a different perspective than causal LMs) can be surprisingly effective, potentially due to their bidirectional context providing richer information.
  • Teacher Ensembling (via Logits) Works: Mixing logits from homogeneous and heterogeneous teachers, or even just using a stronger teacher, leads to better results, mirroring the benefits of teacher ensembling in other KD scenarios.
  • Small Models Benefit Most: PP-KD shows particularly strong improvements for smaller kNN-LM models, making them more competitive.

Key Results

  • PP-KD significantly improves perplexity on WIKITEXT-103 for kNN-LMs of various sizes, outperforming both models trained from scratch and those using conventional logit-based KD.
  • For instance, a large kNN-LM distilled with a powerful homogeneous teacher achieved a perplexity of 21.37, compared to 25.25 for the baseline kNN-LM.
  • Using a heterogeneous (masked LM) teacher also yielded improvements, with PP-KD achieving a perplexity of 21.02, slightly outperforming the homogeneous teacher's result.
  • Mixing logits from homogeneous and heterogeneous teachers further boosted performance, reaching 20.83.
  • The method introduces negligible latency overhead during inference, as the main cost is the kNN retrieval itself, which is already part of the kNN-LM.
  • The datastore size for logits is comparable to the size of the kNN-LM's own datastore, making it storage-efficient.

Limitations and Future Work

  • Improvement Room: The authors acknowledge that while PP-KD shows promising results, there's still room for improvement in terms of more efficient implementation, developing more informative logits, and conducting deeper analyses.
  • Future Directions: They suggest exploring more efficient ways to implement PP-KD, investigating advanced techniques for generating better teacher logits, and connecting PP-KD with emerging retrieval-augmented modeling approaches. They also express interest in building a broader connection between PP-KD and the larger field of retrieval-augmented modeling.

Summary updated

BibTeX
@inproceedings{jin2022plug,
  author = {Xuyang Jin and Tao Ge and Furu Wei},
  title = {{Plug and Play Knowledge Distillation for kNN-LM with External Logits}},
  booktitle = {AACL/IJCNLP},
  year = {2022},
  url = {https://openreview.net/forum?id=7JDDBNIsJUq},
}
Figure