Evaluating Computational Language Models with Scaling Properties of Natural Language

Shuntaro Takahashi, Kumiko Tanaka-Ishii

Abstract

Traditional evaluation metrics for language models often overlook the global statistical structures inherent in natural language. This paper proposes a model-agnostic framework based on universal scaling laws—such as Zipf’s, Heaps’, and Taylor’s laws—to assess the "naturalness" of generated text. Our analysis reveals that while most models can replicate basic vocabulary distributions, only gated neural architectures successfully capture the long-range correlations and bursty patterns characteristic of human writing.

Introduction: Beyond Local Accuracy

Evaluating the quality of language models (LMs) has historically relied on metrics like Perplexity, which measures prediction accuracy, or BLEU/ROUGE, which focus on local n-gram overlaps. However, these metrics have significant blind spots. Perplexity is often intractable for models like Generative Adversarial Networks (GANs), and n-gram overlaps fail to capture the long-term, global structure of a document.

This research argues for a "physics-based" approach to language evaluation. Natural language is not random; it follows universal scaling laws. A truly effective model should not only predict the next word but also produce a body of text that scales statistically like human writing.

Methodology: The Five Scaling Properties

The researchers investigated five specific scaling properties to serve as a robust evaluation framework:

  1. Zipf’s Law: The relationship between word rank and frequency.
  2. Heaps’ Law: The growth rate of vocabulary size relative to text length.
  3. Ebeling’s Method: Fluctuation analysis of character occurrences.
  4. Taylor’s Law: The power-law relationship between the mean and variance of word occurrences, quantifying "burstiness."
  5. Long-Range Correlation: Autocorrelation functions measuring pattern repetition over long distances.
Overview of the five scaling properties used to evaluate language models: (a) Zipf’s Law, (b) Heaps’ Law, (c) Ebeling’s Method, (d) Taylor’s Law, and (e) Long-Range Correlation.

By generating one million words from various models—including n-grams, Probabilistic Context-Free Grammars (PCFG), and neural LMs—the authors fit power-law functions ($y \propto z^\kappa$) to the results. Comparing these exponents ($\kappa$) to human-written corpora (e.g., Wikitext-2, Shakespeare) allowed for a quantitative "naturalness" score.

Key Insights: The Gating Threshold and Taylor’s Law

A critical finding of the study is the "gating threshold." Simple RNNs and n-gram models are essentially memoryless. Only models with gating mechanisms, such as LSTMs and GRUs, were able to break the $\zeta = 0.50$ barrier of Taylor’s law. This value represents the transition from random, independent behavior to the "long memory" clustering seen in human language.

Taylor’s Law (ζ ≈ 0.62) serves as a critical indicator of 'burstiness' and long-term structure in natural language.

Among the metrics tested, Taylor’s Law ($\zeta$) emerged as the most reliable indicator of quality. It showed a strong correlation with perplexity: as models improved, their Taylor exponent moved closer to the human baseline of $\approx 0.62$. Conversely, Zipf’s and Heaps’ laws were found to be "easy" for models to mimic, making them poor discriminators for high-end model performance.

Comparative Results

  • Human Baseline: Wikitext-2 yielded a Taylor exponent of 0.62.
  • AWD-LSTM: This highly optimized model performed best among neural architectures, reaching an exponent of 0.59.
  • n-gram Models: Consistently produced an exponent of 0.50, indicating a total lack of long-term structure.
  • GANs and PCFGs: Both struggled significantly. GANs failed to produce the "long tail" of infrequent words, while PCFGs lacked the inter-sentence structures necessary for long-range memory.

Conclusion and Future Work

While gated RNNs show promise in capturing the statistical signatures of natural language, a gap remains between current models and human-level long-range correlation. Future research should apply this framework to Transformer-based models to determine if self-attention mechanisms can more effectively bridge this gap. This scaling-law approach provides a powerful, model-agnostic tool for the ongoing development of more natural computational language models.

Summary updated

Abstract

In this article, we evaluate computational models of natural language with respect to the universal statistical behaviors of natural language. Statistical mechanical analyses have revealed that natural language text is characterized by scaling properties, which quantify the global structure in the vocabulary population and the long memory of a text. We study whether five scaling properties (given by Zipf’s law, Heaps’ law, Ebeling’s method, Taylor’s law, and long-range correlation analysis) can serve for evaluation of computational models. Specifically, we test n-gram language models, a probabilistic context-free grammar, language models based on Simon/Pitman-Yor processes, neural language models, and generative adversarial networks for text generation. Our analysis reveals that language models based on recurrent neural networks with a gating mechanism (i.e., long short-term memory; a gated recurrent unit; and quasi-recurrent neural networks) are the only computational models that can reproduce the long memory behavior of natural language. Furthermore, through comparison with recently proposed model-based evaluation methods, we find that the exponent of Taylor’s law is a good indicator of model quality.
BibTeX
@article{takahashi2019evaluating,
  author = {Shuntaro Takahashi and Kumiko Tanaka-Ishii},
  title = {{Evaluating Computational Language Models with Scaling Properties of Natural Language}},
  journal = {Comput. Linguistics},
  year = {2019},
  doi = {10.1162/coli\_a\_00355},
}
Figure