Abstract
In natural language generation tasks such as multi-document summarization, the logical ordering of information is critical for coherence. This paper evaluates Kendall’s $\tau$, a non-parametric rank correlation metric, as an automated tool for assessing information sequences. Through two behavioral experiments involving human subjective ratings and online reading time measurements, the study validates $\tau$ as a reliable proxy for text well-formedness. Results show a significant correlation between $\tau$ and human judgments ($r = 0.45$), as well as a strong relationship with cognitive processing effort ($r = -0.48$), suggesting that the metric effectively captures the structural quality of generated documents.
Introduction
Generating high-quality text involves more than just selecting relevant facts; it requires arranging those facts into a logical, coherent sequence. Incoherent information ordering can significantly impede a reader's ability to process and understand a document. Traditionally, evaluating the "fluency" or "coherence" of system-generated text has relied on expensive and time-consuming human judgments. While metrics like BLEU are standard for machine translation, they focus on word-level overlap and fail to capture high-level structural quality. This research addresses the need for a robust, automated, and inexpensive metric specifically designed to evaluate information ordering.
Kendall's $\tau$ as an Evaluation Metric
The author proposes Kendall’s $\tau$ as a solution for measuring the distance between a system-generated sequence and a human-authored "gold standard." Kendall’s $\tau$ is based on the number of adjacent swaps (transpositions) required to transform one ranking into another.
Unlike other rank correlation coefficients like Spearman’s, $\tau$ is non-parametric and less biased in small samples. It offers an intuitive interpretation: it represents the difference between the probability of pairs being in the correct relative order versus the wrong order. A key advantage of this metric is its representation independence; it can be applied to any sequenceable units, such as sentences, discourse trees, or database entries.
Experimental Validation
To validate the metric, the study utilized a corpus of news articles preprocessed to ensure that low ratings were attributable to ordering rather than linguistic brokenness (e.g., replacing pronouns with referents). Two experiments were conducted:
- Judgment Elicitation: 179 participants rated the "understandability" of texts with varying $\tau$ values on a 7-point scale.
- Processing Effort: 32 participants performed an online reading task where sentence-by-sentence reading times were recorded to measure cognitive load.
Results and Discussion
The experimental results provide strong evidence for the validity of Kendall’s $\tau$. Human ratings showed a significant correlation ($r = 0.45$) with the metric. Notably, the average agreement between two human judges was $r = 0.56$, indicating that the automated metric performs nearly as well as human-to-human consistency.
Furthermore, the metric correlated negatively with reading times ($r = -0.48$), meaning that texts with higher $\tau$ scores (closer to the gold standard) were physically faster for humans to read. This confirms that $\tau$ is a valid indicator of the cognitive effort required to understand a text.
Conclusion and Future Work
Kendall’s $\tau$ provides a simple yet powerful automated alternative to human evaluation for information ordering tasks. While the current study uses a single gold standard, future work could explore averaging $\tau$ across multiple human references. Additionally, weighting transpositions based on their position—recognizing that swapping an introduction with a conclusion is more disruptive than swapping two internal facts—could further refine the metric's sensitivity to document structure.
Summary updated