Introduction
In the field of multi-document summarization (MDS), the primary focus has traditionally been on developing more sophisticated algorithms. However, performance often fluctuates wildly across different document sets, even when using the same model. This variability suggests that some inputs are inherently more difficult to summarize than others. In their 2008 paper, Ani Nenkova and Annie Louis investigate the "correlates of difficulty"—the measurable characteristics of an input set that predict how well an automatic summarizer will perform.
Proving the Input Effect
Before identifying specific features of difficulty, the authors sought to quantify how much the input itself contributes to performance variance. Using an Analysis of Variance (ANOVA) on DUC 2001 data, they compared the influence of the input set against the influence of the specific summarization algorithm (the "summarizer").
As shown in the ANOVA table above, the "input" factor yielded an F-statistic of 59.4, significantly higher than the 34.4 observed for the "summarizer" factor. This confirms that the inherent difficulty of the document set often plays a larger role in determining summary quality than the choice of algorithm.
The Role of Topical Cohesiveness
The researchers hypothesized that "topical cohesiveness"—how tightly the documents in a set relate to a single narrative—is a primary driver of difficulty. They categorized document sets into types such as "single event," "biographical," "multiple distinct events," and "opinion."
The results, visualized above, demonstrate a clear performance gap. Inputs categorized as "Cohesive" (biographies, single events, and subjects) consistently achieved higher coverage scores than "Non-Cohesive" inputs (multiple events or opinions). Furthermore, the data shows that systems generally perform better as the target summary length increases, as the pressure for extreme compression is reduced.
Information-Theoretic Indicators
Beyond broad categories, the authors identified specific linguistic and information-theoretic features that correlate with difficulty:
- Set Entropy: This measures the predictability and repetition within a document set. High entropy indicates a diverse vocabulary with little word repetition, making it difficult for systems to identify representative information. Entropy showed a strong negative correlation (-0.42) with summary quality.
- KL Divergence: This measures how much the word distribution of the input differs from a general background corpus. Interestingly, high KL Divergence—indicating a very niche or focused topic—actually makes summarization easier, as key terms stand out more prominently.
- Topic Signatures: By identifying "signature terms" (words uniquely descriptive of the set), the authors could measure how much these terms overlapped across documents. Higher overlap signaled a more cohesive, and thus easier, input.
Predictive Power and Future Directions
Using these features, the researchers developed a logistic regression classifier to predict whether an input set would be "easy" or "difficult." The model achieved an accuracy of 69.27% on DUC data, significantly outperforming a random baseline.
This work suggests a shift in how MDS systems are designed. Rather than applying a one-size-fits-all algorithm, future systems could include a "difficulty detector." When a non-cohesive or high-entropy set is identified, the system could route the task to a specialized strategy designed to handle fragmented information or conflicting perspectives. By understanding the nature of the input, we can build summarizers that are not just more powerful, but more robust.
Summary based on "Can You Summarize This? Identifying Correlates of Input Difficulty for Multi-Document Summarization" by Ani Nenkova and Annie Louis (2008).
Summary updated