GPT Perdetry Test: Generating new meanings for new words

Nikolay Malkin, Sameera Lanka, Pranav Goel, Sudha Rao, Nebojsa JojicNAACL 2021 · OpenReview

Abstract

Can an artificial intelligence understand a word it has never encountered? As language evolves through slang and technical innovation, "Out-of-Vocabulary" (OOV) terms present a significant challenge for traditional NLP models. This paper demonstrates that GPT-3 can generate plausible, dictionary-style definitions for "nonce words" (invented words like perdetry or drobbler) by leveraging subword tokenization and phonosemantics. Remarkably, humans often find these AI-generated meanings so fitting that they sometimes prefer them over definitions invented by human writers.

Introduction

Language is a moving target. New words—neologisms—emerge constantly through social media and technical innovation. While traditional models often break down when encountering OOV terms, modern Large Language Models (LLMs) like GPT-3 use subword tokenization to process unfamiliar strings. However, processing a string of characters does not necessarily imply an understanding of its potential meaning. This study investigates whether GPT-3 has captured the subtle relationships between the sound/orthography of a word and its meaning—a concept known as phonosemantics—to the point where it can effectively "hallucinate" useful definitions for words that do not exist.

Methodology: The Generation Pipeline

The researchers developed a multi-step pipeline to test GPT-3’s linguistic imagination. First, a character-level LSTM was trained on English vocabulary to sample new, pronounceable nonce words (e.g., butherment, shoutze). These words were then used to prompt GPT-3 (specifically the davinci variant) in a dictionary format to generate definitions.

The generation pipeline: an LSTM creates a nonce word, which GPT-3 then defines.

To evaluate these definitions, the authors conducted human studies on Amazon Mechanical Turk. Participants performed a matching task, where they paired words with definitions, and a preference task, where they compared GPT-3’s definitions for existing neologisms against those written by human authors in the Dictionary of Obscure Sorrows.

Phonosemantics and Lexical Strata

A key insight of the study is that GPT-3 does not merely look for formal Latin or Greek roots. It appears to capture "phonesthemes"—clusters of letters that carry meaning despite not being formal morphemes (such as how gl- often relates to light in glow or glimmer).

Furthermore, the model recognizes "lexical strata." Words that sound French or Latinate are typically assigned abstract or formal meanings, while Germanic-sounding clusters are assigned more concrete meanings. This allows the model to fill "plausible lexical gaps"; for example, defining drobbler as "a person who enjoys listening to music" creates a word that feels like it should naturally exist in the English lexicon.

Results and Analysis

In the human matching task, participants correctly paired GPT-3's definitions with their respective nonce words 70.7% of the time for nouns, significantly better than the 50% random chance baseline. GPT-3 also demonstrated high internal consistency, matching its own definitions back to the words it generated with 92.4% to 100% accuracy.

Human matching accuracy across different log-likelihood (LLD) ranges, with example word-definition pairs.

The study found a strong correlation between GPT-3’s internal log-likelihood (LLD) scores and human confidence. As shown in the analysis of LLD ranges, when the model assigns a high likelihood to a word-definition pairing, human matching accuracy increases. Notably, when compared against human-authored definitions for creative neologisms, users preferred the GPT-3 definition in 40% of cases.

Conclusion

The "GPT Perdetry Test" reveals that LLMs possess a quantifiable grasp of the "vibe" or phonetic essence of words. While the model still requires post-processing to filter repetitive or offensive outputs, its ability to generate fitting definitions for non-existent words has broad implications. This work could lead to improved tools for understanding emerging slang, optimizing brand names based on phonetic appeal, or procedurally generating content for linguistic games.

Summary updated

BibTeX
@inproceedings{malkin2021gpt,
  author = {Nikolay Malkin and Sameera Lanka and Pranav Goel and Sudha Rao and Nebojsa Jojic},
  title = {{GPT Perdetry Test: Generating new meanings for new words}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=UY-tmcQScs},
}

Loading related content...

Figure