Abstract
This paper introduces NPM (NonParametric Masked Language Model), a novel architecture that replaces the traditional fixed-vocabulary softmax layer with a retrieval mechanism over a raw text corpus. By mapping masked queries to a dense vector space and retrieving phrases directly from a reference corpus, NPM (354M parameters) achieves superior performance on rare words and factual updates compared to massive parametric models like GPT-3 (175B parameters).
Introduction: The Softmax Bottleneck
Standard language models (LMs) predict tokens by calculating a probability distribution over a fixed, finite vocabulary. This "softmax bottleneck" creates three primary challenges:
- Rare Words: Models struggle to predict infrequent terms or specific word senses.
- Static Knowledge: Once trained, a model's knowledge is frozen; updating it with new facts requires expensive retraining.
- Expressivity: Fixed vocabularies limit the representation of diverse scripts and technical domains.
NPM addresses these issues by moving away from memorizing facts in weights and toward retrieving them from a modular reference corpus.
Architecture and Methodology
NPM utilizes a standard encoder (based on RoBERTa-large) but replaces the output layer with a nonparametric distribution over every phrase in a reference corpus. Instead of mapping hidden states to a vocabulary list, it maps the masked query into a dense vector space.
Dual-Vector Retrieval
To handle phrases of varying lengths, NPM represents a query using two vectors: $q^{start}$ and $q^{end}$. The model retrieves the nearest phrase from a reference index by matching these vectors against an index of all tokens in that corpus. This allows the model to predict any string present in the reference data, effectively creating an infinite vocabulary.
The model is trained using a phrase-level contrastive objective, maximizing the similarity between masked queries and the correct ground-truth phrases while using other sequences in the batch as negatives.
Evaluation and Key Insights
NPM was trained on 13 billion tokens from Wikipedia and CC-News and evaluated across classification, fact-probing, and translation tasks.
Performance and Efficiency
Despite its smaller size, NPM outperformed GPT-3 on several fact-probing benchmarks. On the LAMA-TREx benchmark, NPM's accuracy exceeded that of GPT-3 even when the latter was given retrieve-and-generate assistance. NPM is significantly more parameter-efficient, outperforming models 37x to 500x larger than itself across various zero-shot tasks.
Modular Knowledge and Rare Entities
Knowledge in NPM is modular. On the TempLAMA dataset, when facts changed (e.g., a player moving teams), updating the reference corpus allowed the model to adapt immediately, with accuracy on changed facts jumping from 2.9% to 17.5% without any weight updates. It also excels at the "long tail," predicting rare entities like Thessaloniki or non-Latin scripts (e.g., Korean) more effectively than parametric counterparts by bypassing the need for a fixed multilingual vocabulary.
Limitations and Future Work
While powerful, NPM faces challenges in inference latency due to the complexity of billion-token retrieval compared to simple matrix multiplication. Additionally, the memory footprint for storing embeddings is substantial (e.g., 1.4TB for a full Wikipedia index). Current research focuses on extending this nonparametric approach from masked modeling to full autoregressive generation to support broader applications like chatbots.
Summary updated