Evaluating Text Generation Quality Using Spectral Distances of Surprisal

Zhichen Liu, Yongyuan Li, Yang Xu, Yu Wang, Yingfang Yuan, Zuhao YangEMNLP 2025 · OpenReview

TL;DR

This paper introduces FACE-2, a novel metric for evaluating open-ended text generation that analyzes the "spectral" patterns of text surprisal. FACE-2 significantly outperforms existing metrics in identifying the impact of model scaling (up to 70B parameters) and shows stronger agreement with human preferences, offering a more nuanced evaluation of language model outputs.

What Problem Does This Paper Address?

Evaluating the quality of text generated by Large Language Models (LLMs) is a critical challenge. Traditional metrics often rely on static measures like likelihood or semantic similarity, which are becoming less effective as LLMs grow in size and sophistication. Specifically, these metrics struggle to capture the "model scaling effect" – the observation that larger models generally produce better outputs. Furthermore, many existing metrics were developed and tested on older, smaller models, making their conclusions potentially outdated for today's state-of-the-art LLMs. The authors aim to bridge this gap by proposing a more robust and psycholinguistically-inspired evaluation metric that can better assess modern LLMs.

Methods

The core of FACE-2 lies in analyzing the dynamic patterns of text surprisal, rather than just its static value. Surprisal, a measure of how unexpected a word is given the preceding context, is calculated for each token in a generated text. This sequence of surprisal values is then treated as a signal and transformed into the frequency domain using a Fourier transform. The resulting "spectrum" captures the dynamic fluctuations of surprisal.

FACE-2 introduces several improvements over its predecessor, FACE-1:

  • Optimized Surprisal Processing: The surprisal sequences are normalized using z-score normalization to improve numerical stability and make them evaluator-independent.
  • Improved Spectrum Representation: Instead of just using the real part of the Fourier transform (as in FACE-1), FACE-2 uses the L2-norm of the complex spectral components, capturing more information.
  • New Distance Functions: The authors propose and evaluate several new distance functions (Earth Mover's Distance (EMD), Kullback-Leibler divergence (KL), and Jensen-Shannon divergence (JS)) for comparing the spectra of generated texts against human-written texts, aiming for more mathematically sound comparisons.

Datasets

The experiments utilize a variety of datasets for text generation and evaluation, covering both English and Chinese. These include:

  • Written Modality: Wikipedia, News articles, and Story datasets.
  • Dialogue Modality: Specific datasets for extended investigation.
  • Instruction-Tuned Models: QA datasets like PubmedQA and LIMA.

The authors also mention using the MT-Bench dataset for evaluating agreement with human preferences, noting its larger scale and recency compared to datasets used in prior work.

Key Insights and Ideas

  • Dynamics Matter: The "dynamics" or temporal patterns of surprisal are more informative for evaluating text quality than static surprisal values alone. This aligns with psycholinguistic theories about how humans process language.
  • Model Scaling is Detectable: FACE-2 is demonstrably better at capturing the "scaling effect" where larger models produce better text, a phenomenon that many older metrics fail to reflect.
  • Robustness Across Languages and Models: FACE-2 shows consistent performance across different model families (LLaMA, Gemma, Qwen, Yi) and languages (English, Chinese).
  • Evaluator Independence: The z-score normalization step makes the spectral features of surprisal less dependent on the specific evaluator model used, which is a significant advantage.
  • Human Agreement: The new spectral distance metrics (especially EMD and JS) show improved alignment with human judgments of text quality.

Key Results

  • Model Scaling Effect: FACE-2 achieved significantly higher "valid cell ratios" (indicating better adherence to the "large > medium > small" model performance assumption) compared to FACE-1 and raw surprisal across various datasets and models. For instance, FACE-2 with EMD achieved a 0.600 valid cell ratio for English, while surprisal was only 0.363.
  • Human Preference Agreement: FACE-2 demonstrated stronger correlations with human preferences than most other metrics, including MAUVE and BERTScore, on the MT-Bench dataset. For example, FACE-2 using EMD showed a Pearson correlation of 0.66 with human preferences, while BERTScore was only 0.49.
  • Performance of New Metrics: The new distance functions (EMD, KL, JS) generally outperformed older ones like SO and CORR in reflecting model scaling and human agreement.

Limitations and Future Work

The authors acknowledge several limitations:

  • Model Scale: Experiments did not include the largest models (100B+ parameters) or commercial models like GPT-4, which would be valuable for further validation.
  • Model Type: The study primarily focused on base LLMs, not chat-models, which are more common in real-world applications. More experiments with chat-models are needed, though public dialogue data with human ground-truth is challenging to obtain.
  • Distance Function Selection: While EMD, JS, and CORR showed promise, determining the optimal distance function for FACE-2 requires further investigation.
  • Interpretability: The exact mechanism by which spectral features of surprisal reflect generator model sources is not fully understood and warrants further research to make FACE more interpretable.

Future work will focus on addressing these limitations, particularly by including larger and chat-based models, systematically investigating distance functions, and enhancing the interpretability of the spectral features.

Summary updated

Abstract

We propose a novel automatic evaluation metric for open-ended text generation, which is a substantial improvement of the recently developed method, Fourier analysis of cross-entropy (FACE), hence, FACE-2. FACE-2 is a psycholinguistically inspired metric that extracts the dynamic patterns (spectrum) of text surprisal. Examined with open-ended text generation tasks, FACE-2 significantly outperforms a broad set of baseline metrics in revealing the model scaling effect, which scales up to models of 70B parameters, while many other existing metrics fail to capture this effect. We have also confirmed the advantage of FACE-2 in producing stronger agreement with human preferences from a large human-annotated dataset. We advocate for including metrics that mine the dynamics of likelihood in evaluating open-ended text generation, which covers broader aspects of human language than only using static likelihood-based or semantic-based metrics. Code repository: https://github.com/CLCS-SUSTech/FACEScore.
BibTeX
@inproceedings{liu2025evaluating,
  author = {Zhichen Liu and Yongyuan Li and Yang Xu and Yu Wang and Yingfang Yuan and Zuhao Yang},
  title = {{Evaluating Text Generation Quality Using Spectral Distances of Surprisal}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=z019LfisDi},
}
Figure