Nonparametric Masked Language Modeling

Sewon Min, Weijia Shi, Mike Lewis, Xilun Chen, Wen-tau Yih, Hannaneh Hajishirzi, Luke Zettlemoyer

Abstract

This paper introduces NPM (NonParametric Masked Language Model), a novel architecture that replaces the traditional fixed-vocabulary softmax layer with a retrieval mechanism over a raw text corpus. By mapping masked queries to a dense vector space and retrieving phrases directly from a reference corpus, NPM (354M parameters) achieves superior performance on rare words and factual updates compared to massive parametric models like GPT-3 (175B parameters).

Introduction: The Softmax Bottleneck

Standard language models (LMs) predict tokens by calculating a probability distribution over a fixed, finite vocabulary. This "softmax bottleneck" creates three primary challenges:

  1. Rare Words: Models struggle to predict infrequent terms or specific word senses.
  2. Static Knowledge: Once trained, a model's knowledge is frozen; updating it with new facts requires expensive retraining.
  3. Expressivity: Fixed vocabularies limit the representation of diverse scripts and technical domains.

NPM addresses these issues by moving away from memorizing facts in weights and toward retrieving them from a modular reference corpus.

Architecture and Methodology

NPM utilizes a standard encoder (based on RoBERTa-large) but replaces the output layer with a nonparametric distribution over every phrase in a reference corpus. Instead of mapping hidden states to a vocabulary list, it maps the masked query into a dense vector space.

NPM retrieves phrases from a reference corpus to fill masked queries, bypassing the fixed vocabulary bottleneck.

Dual-Vector Retrieval

To handle phrases of varying lengths, NPM represents a query using two vectors: $q^{start}$ and $q^{end}$. The model retrieves the nearest phrase from a reference index by matching these vectors against an index of all tokens in that corpus. This allows the model to predict any string present in the reference data, effectively creating an infinite vocabulary.

The dual-vector retrieval mechanism maps queries to start and end vectors in a shared vector space with the reference corpus.

The model is trained using a phrase-level contrastive objective, maximizing the similarity between masked queries and the correct ground-truth phrases while using other sequences in the batch as negatives.

Evaluation and Key Insights

NPM was trained on 13 billion tokens from Wikipedia and CC-News and evaluated across classification, fact-probing, and translation tasks.

Performance and Efficiency

Despite its smaller size, NPM outperformed GPT-3 on several fact-probing benchmarks. On the LAMA-TREx benchmark, NPM's accuracy exceeded that of GPT-3 even when the latter was given retrieve-and-generate assistance. NPM is significantly more parameter-efficient, outperforming models 37x to 500x larger than itself across various zero-shot tasks.

Modular Knowledge and Rare Entities

Knowledge in NPM is modular. On the TempLAMA dataset, when facts changed (e.g., a player moving teams), updating the reference corpus allowed the model to adapt immediately, with accuracy on changed facts jumping from 2.9% to 17.5% without any weight updates. It also excels at the "long tail," predicting rare entities like Thessaloniki or non-Latin scripts (e.g., Korean) more effectively than parametric counterparts by bypassing the need for a fixed multilingual vocabulary.

Limitations and Future Work

While powerful, NPM faces challenges in inference latency due to the complexity of billion-token retrieval compared to simple matrix multiplication. Additionally, the memory footprint for storing embeddings is substantial (e.g., 1.4TB for a full Wikipedia index). Current research focuses on extending this nonparametric approach from masked modeling to full autoregressive generation to support broader applications like chatbots.

Summary updated

Abstract

Existing language models (LMs) predict tokens with a softmax over a finite vocabulary, which can make it difficult to predict rare tokens or phrases. We introduce NPM, the first nonparametric masked language model that replaces this softmax with a nonparametric distribution over every phrase in a reference corpus. NPM fills in the [MASK] solely from retrieving a token from a text corpus. We show that NPM can be efficiently trained with a contrastive objective and an in-batch approximation to full corpus retrieval. Zero-shot evaluation on 16 tasks including classification, fact probing and question answering demonstrates that NPM outperforms significantly larger parametric models, either with or without a retrieve-and-generate approach. It is particularly better at dealing with rare patterns (word senses or facts) and predicting rare or nearly unseen words (e.g., non-Latin script). We release the model and code at github.com/facebookresearch/NPM.
BibTeX
@inproceedings{min2023nonparametric,
  author = {Sewon Min and Weijia Shi and Mike Lewis and Xilun Chen and Wen-tau Yih and Hannaneh Hajishirzi and Luke Zettlemoyer},
  title = {{Nonparametric Masked Language Modeling}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.132},
}
Figure