Abstract
To date, few attempts have been made to develop and validate methods for automatic evaluation of linguistic quality in text summarization. We present the first systematic assessment of several diverse classes of metrics designed to capture various aspects of well-written text. We train and test linguistic quality models on consecutive years of NIST evaluation data in order to show the generality of results. For grammaticality, the best results come from a set of syntactic features. Focus, coherence, and referential clarity are best evaluated by a class of features measuring local coherence on the basis of cosine similarity between sentences, coreference information, and summarization-specific features. Our best results are 90% accuracy for pairwise comparisons of competing systems over a test set of several inputs and 70% for ranking summaries of a specific input.
Introduction
In the field of automatic text summarization, evaluation has historically focused on content coverage. Metrics like ROUGE, which measure n-gram overlap between machine-generated summaries and human references, have become the standard. However, ROUGE is largely indifferent to the linguistic quality of the output. A summary can contain all the correct facts while remaining a "word salad" of disconnected sentences.
Manual evaluation by humans remains the gold standard for assessing linguistic quality, but it is prohibitively slow and expensive for iterative development. This research addresses the need for reliable, automatic metrics that can rank machine-produced summaries based on their linguistic merits—specifically grammaticality, non-redundancy, referential clarity, focus, and structure.
Methodology and Feature Engineering
The study breaks down "linguistic quality" into five dimensions used in NIST evaluations. To predict these scores, the authors implemented several feature classes:
- Language Models (LM): Unigram, bigram, and trigram probabilities derived from the Gigaword corpus to assess naturalness.
- Sentence Fluency: A suite of 38 shallow syntactic features (e.g., sentence length, phrase types) designed to detect ungrammaticality.
- Continuity: A sophisticated class combining cosine similarity between adjacent sentences, coreference tracking, and "summarization-specific" features that check if sentences retain their original source-document context.
- Entity Coherence: Based on the "Entity Grid" model, which tracks subject/object transitions across sentences.
- Coh-Metrix: A standard tool for text readability and cohesion.
The researchers utilized a Ranking SVM to learn optimal feature weights. To ensure generalizability, models were trained on DUC 2006 data (50 inputs, 35 systems) and tested on DUC 2007 data (45 inputs, 32 systems).
Key Results
The models were evaluated on their ability to perform pairwise rankings—deciding which of two summaries is linguistically superior.
System-Level Performance
When ranking the overall performance of different summarization systems across the entire test set, the models achieved remarkable accuracy. The "Meta-ranker," which combines all feature sets, reached accuracies between 87.8% and 92.9% across the five dimensions.
Input-Level Performance
Ranking different summaries for a single specific topic (input-level) proved more challenging but still yielded strong results. The Meta-ranker achieved approximately 70% accuracy for most dimensions, significantly outperforming baseline language models.
Discussion and Insights
One of the most significant findings is the lack of correlation between content coverage (ROUGE) and linguistic quality. A system’s ability to select the right information does not guarantee a readable output.
Furthermore, the "Continuity" feature set emerged as a powerful predictor for coherence and referential clarity in extractive summaries. Interestingly, specialized syntactic features outperformed standard Language Models for catching grammatical errors. The study also noted that features designed for coherence (like the Entity Grid) were surprisingly effective at predicting grammaticality, suggesting that structural and grammatical errors often co-occur in machine-generated text.
Conclusion
This work demonstrates that automatic evaluation of linguistic quality is not only possible but can achieve high levels of accuracy in ranking summarization systems. While the current models are optimized for extractive summaries, they provide a robust foundation for future metrics as the field moves toward more complex abstractive summarization techniques.
Summary updated