Perplexity as a Metric for Linguistic Distance

20 concepts13 papers

Language models are traditionally trained to minimize perplexity—a metric that quantifies how "surprised" a model is when encountering a sequence of text. Mathematically, perplexity is the exponentiated cross-entropy of a text sequence under the model's predicted probability distribution. If a model predicts every upcoming token perfectly, its perplexity is 1; if the text deviates from what the model has learned, the perplexity climbs.

This standard evaluation metric can be inverted into a powerful analytical tool. By training a language model on a specific linguistic source—such as a particular historical era, a distinct dialect, or a specific author profile—and using it to evaluate an external text, the resulting perplexity serves as a continuous, quantitative proxy for linguistic distance. Instead of relying on manual feature engineering or rigid categorical classifications, researchers can use the statistical surprise of a model to map out relationships across time, space, and language variants.

Mapping Dialects and Text Varieties

When applied to different text varieties (diatopic variation), cross-corpus perplexity reveals how far a text deviates from a baseline standard. For instance, Timothy Baldwin et al. demonstrated in How Noisy Social Media Text, How Diffrnt Social Media Sources? that by calculating the perplexity of various social media platforms using models trained on edited prose, one can place different online sources on a predictable continuum of linguistic complexity rather than treating social media as a uniform block of noise.

This intuition scales directly to dialectology. To quantify how distinct a dialect is from a standard language without manual annotation, Alexander Johnson et al. in Automatic Dialect Density Estimation for African American English measured the surprisal of African American English utterances against a language model trained on standard English. High surprisal scores effectively captured morphosyntactic and phonetic shifts, providing an automated estimate of dialect density. Similarly, Yixia Li et al. in Cantonese natural language processing in the transformers era: a survey and current challenges utilized surprisal analysis to measure the cross-entropy loss of pre-trained models, objectively mapping the linguistic divergence between colloquial Cantonese and Standard Chinese.

By training character-level language models on different variants of a low-resource language like Nahuatl, Robert Pugh et al. showed in Investigating variation in written forms of Nahuatl using character-based language models that cross-variant perplexity acts as a reliable proxy for linguistic similarity. Crucially, the authors observed a negative correlation between this computational perplexity and human-reported mutual intelligibility, demonstrating that the model's statistical surprise mirrors actual human comprehension barriers.

Tracking Diachronic Language Evolution

Languages evolve gradually, making rigid chronological boundaries problematic for historical linguistics. Perplexity offers a continuous alternative for tracking diachronic change. If language changes systematically over time, a model trained on text from century $X$ should find text from century $X+1$ slightly surprising, and text from century $X+5$ highly surprising.

This chronological correlation allows for the automatic dating of historical documents. Yonatan Belinkov et al. in Shamela: A Large-Scale Historical Arabic Corpus trained a series of n-gram language models on distinct chronological periods of Arabic literature. By evaluating an undated text across all models, the document could be triaged to the specific time interval whose model yielded the lowest perplexity.

To formalize this, researchers introduced the concept of Perplexity-Based Language Distance (PLD). Jose Ramom Pichel Campos et al. applied character-based n-gram PLD to European Portuguese in Measuring language distance among historical varieties using perplexity. Application to European Portuguese., confirming that closer historical periods exhibit smaller perplexity distances, validating it as a metric for tracking long-term language evolution. Andrea Zugarini et al. extended this framework to medieval Italian varieties in Vulgaris: Analysis of a Corpus for Middle-Age Varieties of Italian Language, using neural language models to compute symmetric PLD metrics by averaging cross-corpus perplexities.

Rather than forcing texts into predefined historical bins, Sidsel Boldsen et al. in Identifying Temporal Trends Based on Perplexity and Clustering: Are We Looking at Language Change? used perplexity scores as a direct proxy for temporal distance, clustering documents based on continuous linguistic drift. As highlighted by Melis Çelikkol et al. in Exploring Diachronic and Diatopic Changes in Dialect Continua: Tasks, Datasets and Challenges, these perplexity-based measures allow researchers to objectively capture subtle, multi-dimensional diachronic and diatopic variations that traditional discrete binning misses.

Cross-Lingual Influence and Cognitive Proxies

The utility of perplexity extends beyond dialects and historical eras to cross-lingual contexts, capturing how one language influences another or how well different languages can be mutually understood.

In native language identification, Binod Gyawali et al. leveraged language models trained on specific native language backgrounds in Native Language Identification: a Simple n-gram Based Approach. When an author writes in a second language, their native syntax and word choices leave statistical fingerprints; by measuring the cross-lingual perplexity against different background models, the authors quantified the likelihood of specific native language influences based on which model was least "surprised."

Furthermore, computational models can serve as proxies for human cognitive patterns in cross-lingual comprehension. Iuliia Zaitova et al. demonstrated in Mapping Phonology to Semantics: A Computational Model of Cross-Lingual Spoken-Word Recognition that a model's cross-lingual performance in spoken-word recognition accurately predicts human mutual intelligibility and reflects the genetic relationships among Slavic languages. Similarly, Jessica Nieder et al. in A Computational Model for the Assessment of Mutual Intelligibility Among Closely Related Languages built computational models utilizing phonetic sound classes and semantic word embeddings to automatically assess mutual intelligibility among closely related languages, successfully mimicking human psycholinguistic patterns.

Granular Intertextuality

Perplexity can also be applied at a highly granular level to detect subtle textual relationships within a single corpus or across closely related works. Maria Konstantinidou et al. introduced the Positive Cross Value (PCV) metric in Exploring intertextuality across the Homeric poems through language models to explore intertextuality in the Homeric poems. By comparing the verse-level perplexity of a segment when modeled by its own epic versus an alternative epic, the authors identified specific passages that were less surprising to the other poem's model. This asymmetric surprise uncovers hidden linguistic resemblances and oral composition patterns at a resolution that aggregate corpus metrics cannot reach.

Key Foundational Work

Shamela: A Large-Scale Historical Arabic Corpus (2016) — contributed 2 concept elements to this direction.
How Noisy Social Media Text, How Diffrnt Social Media Sources? (2013) — contributed 1 concept elements to this direction.
Native Language Identification: a Simple n-gram Based Approach (2013) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_perplexity_based_language_distance_f12d4ad6,
  title = {perplexity-based language distance},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/perplexity-based-language-distance/rdr_f12d4ad6c40e4e189b7facac6af7a580},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_perplexity_based_language_distance_f12d4ad6_v1,
  title = {perplexity-based language distance},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/perplexity-based-language-distance/rdr_f12d4ad6c40e4e189b7facac6af7a580/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake