Abstract
Summarizing texts involves significant cognitive efforts to compress information. While advances in automatic summarization systems have drawn attention from the NLP and linguistics communities to this topic, there is a lack of computational studies of linguistic patterns in human-written summaries. This work presents a large-scale corpus study of human-written single-sentence summaries. We analyzed the linguistic compression patterns from source documents to summaries at different granularities, and we found that summaries are generally written with morphological expansion, increased lexical diversity, and similar positional arrangements of specific words compared to the source across different genres. We also studied how linguistic compressions of different factors affect reader judgments of quality through a human study, with the results showing that the use of morphological and syntactic changes by summary writers matches reader preferences while lexical diversity and word specificity preferences are not aligned between summary writers and readers.
Introduction
With the rapid advancement of Large Language Models (LLMs), automatic text summarization has reached new heights of fluency. However, these models frequently struggle with "hallucinations"—generating factually incorrect information—and "verbose copying," where they simply repeat large chunks of the source text. To improve these systems, it is essential to understand the "gold standard" of human summarization. While qualitative linguistic theories suggest humans use "macrorules" like deletion and generalization, there has been a lack of large-scale quantitative data on the specific morphological, syntactic, and semantic strategies humans employ when compressing a document into a single, punchy sentence.
Methodology
This study, presented at EMNLP 2023, utilized four diverse datasets to capture patterns across genres: WikiHow (instructional), XSum (news), and SciTLDR (scientific abstracts written by both authors and peers). The researchers employed a three-tiered analysis:
- Feature Classification: Using binary logistic regression to determine if morphological (word length, morpheme count), syntactic (tree height, POS variety), or semantic (word specificity, lexical diversity) features could distinguish a summary from its source.
- Positional Analysis: Using Dynamic Time Warping (DTW) to align sentences of varying lengths and create "average contours" of information density and word specificity.
- Human Evaluation: A study where native speakers rated summaries on quality aspects like fluency and novelty, followed by Representational Similarity Analysis (RSA) to correlate linguistic features with these human judgments.
The Linguistic Patterns of Summarization
The "Expansion" Paradox
Contrary to the intuition that summaries might use simpler language to save space, the researchers discovered a phenomenon termed morphological expansion. Human summaries tend to utilize longer words with a higher number of morphemes (the smallest units of meaning). Essentially, human writers often replace several simple words with a single, information-dense "heavyweight" word.
As shown in the analysis of feature differences, morphological and syntactic shifts are highly consistent across datasets. In fact, for the WikiHow dataset, a model could distinguish a summary from a source sentence with 91.6% accuracy based on just three linguistic features.
Positional Mimicry
Beyond word choice, the study found that summaries often replicate the "shape" of their source sentences. By mapping word specificity across the relative position of a sentence, the researchers found that the peaks and valleys of specificity in a summary often mirror those of the source text. This suggests that even when compressing information, human writers attempt to maintain the original flow and structural arrangement of the information.
The Writer-Reader Mismatch
A critical finding of the study is the misalignment between what summary writers produce and what readers value. While writers lean heavily into lexical diversity (using a wide variety of unique words) and high word specificity, the human study revealed that readers are not particularly impressed by these traits. Instead, readers prioritize relevance (t=6.604) and factual coverage (t=3.178) above all else. While the morphological and syntactic strategies used by writers align with reader preferences for quality, the emphasis on lexical variety appears to be a writer-centric habit that does not significantly improve the reader's experience.
Conclusion
This research provides a quantitative map of the human summarization process, highlighting that compression is not merely about deletion, but about sophisticated linguistic transformation. By identifying the gap between writer strategies and reader preferences, this work offers a path forward for tuning automatic summarization models to better reflect human-centric quality standards. Future work may involve intervention studies using eye-tracking to see how these specific linguistic features affect real-time processing by readers.
Summary updated