Towards Better Open-Ended Text Generation: A Multicriteria Evaluation Framework

Esteban Garces Arias, Hannah Blocher, Julian Rodemann, Meimingwei Li, Christian Heumann, Matthias Aßenmacher

This paper from researchers at LMU Munich addresses a fundamental headache in Natural Language Processing: how do we actually know if a Large Language Model (LLM) is writing "well"? In open-ended generation, there is no single "right" answer, and traditional metrics often pull in opposite directions. The authors propose a new multicriteria framework to move beyond one-dimensional evaluation.

TL;DR

The authors introduce a multicriteria framework for evaluating open-ended text generation that balances the inherent trade-offs between coherence, diversity, and perplexity. By combining statistical ranking methods (like Bradley-Terry and UFG depth) with a novel aggregate metric called Q*Text, they provide a more robust way to compare decoding strategies and models.

What Problem Does This Paper Address?

Evaluating open-ended text generation (like story writing or chatting) is notoriously difficult. Unlike translation or summarization, there isn't a single reference text to compare against. Researchers usually rely on three main automatic metrics:

  1. Coherence: Does the text make sense and follow the prompt?
  2. Diversity: Is the text varied, or does it repeat the same phrases?
  3. Perplexity: How "surprised" is the model by its own output?

The problem is the trade-off. If you optimize purely for coherence, models often become repetitive and boring (low diversity). If you push for high diversity, the text often becomes nonsensical (low coherence). Most current benchmarks either pick one metric or use a simple weighted sum, which fails to capture these nuances. This paper seeks a principled way to aggregate these conflicting signals.

Methods

The authors propose three distinct methods tailored to different benchmarking needs, as illustrated in their evaluation framework:

Multicriteria evaluation framework for benchmarking models and decoding strategies

1. Relative Ranking (Scenario 1)

For practitioners who just need to know which method is better, the paper suggests two ordinal approaches:

  • Extended Bradley-Terry Model: A standard statistical tool for turning pairwise comparisons into a total ranking. It assigns a "worth parameter" to each method.
  • Union-Free Generic (ufg) Depth: A more sophisticated approach that allows for incomparability. It recognizes that sometimes Method A is better at coherence while Method B is better at diversity, and it might be statistically dishonest to force one above the other.

2. Cardinal Assessment (Scenario 2)

For researchers who need a single score to track progress, they introduce Q*Text. This metric aggregates normalized scores for coherence, diversity, and perplexity. Crucially, it uses Gaussian penalty functions to punish "extreme" values—meaning it heavily penalizes text that is either extremely repetitive or completely incoherent.

Datasets

The study is massive in scale, evaluating over 1.8 million text generations. They used three distinct domains to ensure the framework generalizes:

  • News: 2,000 articles from Wikinews.
  • Wikipedia: 1,314 articles from the WikiText-103 dataset.
  • Stories: 1,947 examples from the Project Gutenberg split of the BookCorpus.

The experimental setup involved six model architectures (including Llama 3.1, Mistral, and Qwen 2) and 59 different decoding hyperparameter configurations.

Summary of the experimental setup, including models, datasets, and decoding strategies

Key Insights and Ideas

  • The "Incomparability" Reality: Using UFG depth, the authors found that many top-performing decoding methods are actually incomparable. This suggests that "one-size-fits-all" rankings in NLP leaderboards might be oversimplifying the truth.
  • Balanced is Best: The most successful decoding strategies (like Contrastive Search with moderate penalties) are those that occupy the "middle ground." Extreme settings lead to either "degenerate" (repetitive) or "erratic" (incoherent) text.
  • Small Models Can Punch Up: Interestingly, a smaller model like GPT2-XL, when paired with a well-tuned decoding strategy, can nearly match human performance scores on the Q*Text metric, sometimes outperforming larger models with poor decoding settings.

Key Results

  • Human Alignment: QText achieved a Spearman correlation of 0.5545* with human ratings, demonstrating it captures human preferences better than individual raw metrics.
  • Case Study Success: In qualitative tests, Q*Text correctly assigned near-zero scores to "degenerate" outputs (repetitive loops) and "erratic" outputs (gibberish), while giving high scores (80+) to balanced, human-like continuations.
Case study comparing different decoding methods on a single prompt, showing how Q*Text penalizes degenerate and erratic content
  • Top Performers: Across all datasets, Mistral 3 with Contrastive Search consistently emerged as a top-tier method in the Bradley-Terry rankings.

Limitations and Future Work

The authors identify several areas for improvement:

  • Metric Dependence: The coherence metric relies on a specific model (OPT 2.7B), which might introduce its own biases.
  • Scope of Metrics: While they focused on three core dimensions, future versions could include factuality, style consistency, or safety.
  • Computational Complexity: The UFG depth method is computationally expensive ($O(2^m)$), making it difficult to scale to hundreds of models simultaneously.

Overall, this framework provides a much-needed "multilingual" lens for evaluation, acknowledging that the best text generation isn't just about one number, but about the right balance.

Summary updated

Abstract

Open-ended text generation has become a prominent task in natural language processing due to the rise of powerful (large) language models. However, evaluating the quality of these models and the employed decoding strategies remains challenging due to trade-offs among widely used metrics such as coherence, diversity, and perplexity. This paper addresses the specific problem of multicriteria evaluation for open-ended text generation, proposing novel methods for both relative and absolute rankings of decoding methods. Specifically, we employ benchmarking approaches based on partial orderings and present a new summary metric to balance existing automatic indicators, providing a more holistic evaluation of text generation quality. Our experiments demonstrate that the proposed approaches offer a robust way to compare decoding strategies and serve as valuable tools to guide model selection for open-ended text generation tasks. We suggest future directions for improving evaluation methodologies in text generation and make our code, datasets, and models publicly available.
BibTeX
@inproceedings{arias2025towards,
  author = {Esteban Garces Arias and Hannah Blocher and Julian Rodemann and Meimingwei Li and Christian Heumann and Matthias Aßenmacher},
  title = {{Towards Better Open-Ended Text Generation: A Multicriteria Evaluation Framework}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure