Automatic Evaluation of Information Ordering: Kendall's Tau

Mirella Lapata

Abstract

In natural language generation tasks such as multi-document summarization, the logical ordering of information is critical for coherence. This paper evaluates Kendall’s $\tau$, a non-parametric rank correlation metric, as an automated tool for assessing information sequences. Through two behavioral experiments involving human subjective ratings and online reading time measurements, the study validates $\tau$ as a reliable proxy for text well-formedness. Results show a significant correlation between $\tau$ and human judgments ($r = 0.45$), as well as a strong relationship with cognitive processing effort ($r = -0.48$), suggesting that the metric effectively captures the structural quality of generated documents.

Introduction

Generating high-quality text involves more than just selecting relevant facts; it requires arranging those facts into a logical, coherent sequence. Incoherent information ordering can significantly impede a reader's ability to process and understand a document. Traditionally, evaluating the "fluency" or "coherence" of system-generated text has relied on expensive and time-consuming human judgments. While metrics like BLEU are standard for machine translation, they focus on word-level overlap and fail to capture high-level structural quality. This research addresses the need for a robust, automated, and inexpensive metric specifically designed to evaluate information ordering.

Kendall's $\tau$ as an Evaluation Metric

The author proposes Kendall’s $\tau$ as a solution for measuring the distance between a system-generated sequence and a human-authored "gold standard." Kendall’s $\tau$ is based on the number of adjacent swaps (transpositions) required to transform one ranking into another.

Comparison of reference and system orderings illustrating how transpositions affect the metric.

Unlike other rank correlation coefficients like Spearman’s, $\tau$ is non-parametric and less biased in small samples. It offers an intuitive interpretation: it represents the difference between the probability of pairs being in the correct relative order versus the wrong order. A key advantage of this metric is its representation independence; it can be applied to any sequenceable units, such as sentences, discourse trees, or database entries.

Experimental Validation

To validate the metric, the study utilized a corpus of news articles preprocessed to ensure that low ratings were attributable to ordering rather than linguistic brokenness (e.g., replacing pronouns with referents). Two experiments were conducted:

  1. Judgment Elicitation: 179 participants rated the "understandability" of texts with varying $\tau$ values on a 7-point scale.
  2. Processing Effort: 32 participants performed an online reading task where sentence-by-sentence reading times were recorded to measure cognitive load.

Results and Discussion

The experimental results provide strong evidence for the validity of Kendall’s $\tau$. Human ratings showed a significant correlation ($r = 0.45$) with the metric. Notably, the average agreement between two human judges was $r = 0.56$, indicating that the automated metric performs nearly as well as human-to-human consistency.

Mean human ratings across binned $\tau$ values, showing higher ratings for better-ordered text.

Furthermore, the metric correlated negatively with reading times ($r = -0.48$), meaning that texts with higher $\tau$ scores (closer to the gold standard) were physically faster for humans to read. This confirms that $\tau$ is a valid indicator of the cognitive effort required to understand a text.

Conclusion and Future Work

Kendall’s $\tau$ provides a simple yet powerful automated alternative to human evaluation for information ordering tasks. While the current study uses a single gold standard, future work could explore averaging $\tau$ across multiple human references. Additionally, weighting transpositions based on their position—recognizing that swapping an introduction with a conclusion is more disruptive than swapping two internal facts—could further refine the metric's sensitivity to document structure.

Summary updated

Abstract

This article considers the automatic evaluation of information ordering, a task underlying many text-based applications such as concept-to-text generation and multidocument summarization. We propose an evaluation method based on Kendall's τ, a metric of rank correlation. The method is inexpensive, robust, and representation independent. We show that Kendall's τ correlates reliably with human ratings and reading times.
BibTeX
@article{lapata2006automatic,
  author = {Mirella Lapata},
  title = {{Automatic Evaluation of Information Ordering: Kendall's Tau}},
  journal = {Comput. Linguistics},
  year = {2006},
  doi = {10.1162/coli.2006.32.4.471},
}
Figure