Evaluating Text Quality via Language Model Likelihood

106 concepts103 papers2 proposals

How do we determine if a piece of machine-generated text is fluent, natural, or accurate? Traditional evaluation metrics rely heavily on token overlap against human-written "gold standard" references (such as the BLEU score). However, language is inherently flexible; a perfectly valid translation or summary might use entirely different phrasing than the reference, causing overlap metrics to penalize it unfairly.

To bypass this limitation, researchers leverage a fundamental insight: a well-trained language model captures the statistical distribution of natural language. Instead of checking what words match a reference, we can measure how likely the generated sequence is to occur within the language's true distribution. By calculating the probability, log-likelihood, or perplexity of a text sequence under a language model, we obtain an intrinsic, reference-free proxy for linguistic quality and fluency.

From Local N-Grams to Global Autoregressive Probabilities

The mechanism of likelihood-based evaluation has evolved alongside language modeling architectures, but the underlying mathematical principle remains the same.

In classical statistical sequence modeling, text probability is decomposed into local transitions using $n$-gram models. These models assume a Markov property where the probability of a word depends only on the preceding $n-1$ words. For instance, in machine translation pipelines, high-order $n$-gram models are integrated to score target-language fluency by favoring word sequences that frequently appear in large monolingual corpora.

As shown by Xu et al. in Zipporah: a Fast and Scalable Data Cleaning System for Noisy Web-Crawled Parallel Corpora, calculating the negative log-likelihood of a sentence using an $n$-gram model—normalized by sentence length—provides a highly scalable way to filter out noisy, unnatural text. Similarly, Li et al. demonstrate in AM-FM: A Semantic Framework for Translation Quality Assessment that $n$-gram probabilities can successfully estimate target-language fluency without requiring any source text or reference translations.

With the advent of deep learning, autoregressive neural network architectures replaced rigid $n$-gram windows with continuous hidden states and attention mechanisms. Instead of looking back only a few words, these models compute the joint probability of an entire sequence by sequentially predicting the next token based on the full preceding context. In The Unreasonable Effectiveness of Transformer Language Models in Grammatical Error Correction, Alikaniotis et al. demonstrate that the next-word prediction objective of autoregressive transformers allows them to score candidate sentences effectively. They match supervised systems in correcting grammatical errors simply by identifying which correction yields the highest sequence likelihood.

Why Internal Likelihood Outperforms Explicit Prompting

As Large Language Models (LLMs) have scaled, a common approach to evaluation has been "LLM-as-a-judge," where a model is prompted to generate an explicit numerical score (e.g., "Rate this text from 1 to 5"). However, this approach introduces significant noise, as models can be highly sensitive to prompt formatting and often struggle to map abstract quality concepts to discrete labels.

A more robust alternative is to extract the model's internal likelihood of generating the text. In The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metrics, Deutsch et al. highlight that calculating the model's internal likelihood of an output sequence yields more stable and accurate correlations with human judgment than explicit scoring prompts.

This probabilistic approach works because it treats evaluation as a continuous density estimation problem rather than a discrete classification task. For example, when evaluating multiple-choice options or text completions, Luan et al. introduce a full-text perplexity scoring method in BOSE: A Systematic Evaluation Method Optimized for Base Models. Instead of forcing a base model to output an abstract label like "A" or "B"—which it may not have been instruction-tuned to do—they calculate the probability of the entire answer string concatenated to the prompt. This aligns the evaluation directly with the model's pre-training objective, providing a granular, continuous signal that accurately reflects the model's true internal knowledge.

The Narcissistic Bias of Likelihood Metrics

While likelihood-based evaluation offers a powerful, reference-free mechanism for quality estimation, it is not without systemic vulnerabilities. The most prominent of these is model-specific bias.

Because a language model's likelihood scores are derived entirely from its own learned parameters, it naturally assigns higher probabilities to text structures that mirror its own training distribution. In LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scores, Liu et al. uncover a severe "self-bias" in reference-free evaluation: LLMs consistently assign higher likelihoods and quality scores to text generated by their own architecture. This suggests that when we use a model's internal probability distribution as an absolute metric for quality, we may inadvertently be measuring how well the text conforms to that specific model's idiosyncratic style, rather than an objective standard of human fluency.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_n_gram_language_modeling_c40a21e3,
  title = {n-gram language modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/n-gram-language-modeling/rdr_c40a21e31b324820b4ad3ba0e99da529},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_n_gram_language_modeling_c40a21e3_v1,
  title = {n-gram language modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/n-gram-language-modeling/rdr_c40a21e31b324820b4ad3ba0e99da529/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake