TL;DR
This paper investigates whether human-written multi-document summaries are more extractive or generative than single-document summaries. They found that multi-document summaries are significantly less extractive, suggesting that current extractive summarization techniques might not be sufficient for handling multiple documents.
What Problem Does This Paper Address?
The explosion of online information necessitates efficient ways to digest large amounts of text. Multi-document summarization aims to condense information from several sources into a concise summary. While significant effort has gone into developing technologies for this task, there's a gap in understanding the fundamental nature of human-generated multi-document summaries. Specifically, do humans "cut and paste" extensively from source documents (extractive) or do they rephrase and synthesize information more freely (generative)? This understanding is crucial for designing effective summarization systems.
Methods
The authors propose a novel approach to characterize summaries by analyzing the "tiling" of summary sentences with n-grams from source documents. The core idea is that if a summary is extractive, we'd expect to find long contiguous sequences of words (n-grams) in the summary that are directly lifted from the source documents. They developed a method to find the "minimal tiling" of each summary sentence using n-grams, optimizing for longer n-grams to represent direct extraction. This involves:
- Breaking down documents into sentences: The source documents and summary sentences are tokenized.
- Finding n-gram overlap: For each summary sentence, they search for n-grams present in both the summary and the source documents.
- Minimal tiling: They cover each summary sentence with the fewest possible n-grams, prioritizing longer ones. This process is normalized by converting text to lowercase and removing punctuation.
- Statistical analysis: They then compare the characteristics of these "tiles" (e.g., average length, presence of exact sentence matches) between single-document and multi-document summaries.
Datasets
The study uses data from the 2001 Document Understanding Conference (DUC). This corpus contains documents describing 60 events, with each event represented by 3 to 20 separate documents. For their analysis, they specifically used the training portion of the DUC 2001 summarization corpus, which includes 295 documents distributed over 30 clusters. They utilized the 100-word multi-document summaries provided by NIST for evaluation.
Key Insights and Ideas
- Human summarizers are less extractive for multi-document tasks: This is the central finding. Humans don't seem to be simply copying large chunks of text when summarizing multiple documents.
- Shift from long n-grams to shorter ones: The analysis suggests that for multi-document summaries, humans tend to borrow text at the "bigram level" rather than extracting long sequences of words or full sentences, as they do for single-document summaries.
- Implications for automatic summarization: Current automatic summarization systems often rely on extractive techniques. The findings suggest these methods might be insufficient for multi-document summarization, as they don't capture the generative nature observed in human summaries.
- Evaluation metric caution: Automatic evaluation metrics that rely heavily on n-gram overlap might be less reliable for multi-document summaries if human references are more generative and use their own words.
Key Results
- Average tile size: The average length of an n-gram tile was significantly smaller for multi-document summaries (2.33 words) compared to single-document summaries (4.47 words).
- Exact sentence matches: 61% of single-document summary sentences exactly matched a sentence in the source document. For multi-document summaries, this figure was 0%.
- Complete tiling from a single sentence: 7.6% of single-document summary sentences could be fully tiled by phrases from a single source sentence. This was 0% for multi-document summaries.
- Tiling coverage: On average, 93% of a single-document summary sentence could be tiled from a single source sentence. This dropped to an average of 36% for multi-document summaries.
- Statistical significance: Statistical tests confirmed that the average tile length from single-document summaries is significantly larger than that from multi-document summaries.
- Maximum tile length: The maximum tile length found was 38 words for single-document summaries, while it was only 24 words for multi-document summaries.
Limitations and Future Work
- Corpus specificity: The authors acknowledge that their findings are based on the DUC corpus and may not generalize to all summarization corpora, which might exhibit different properties.
- System output analysis: They propose applying their n-gram tiling method to the output of automatic summarization systems to empirically quantify the extent to which current systems reformulate text.
- Exploring different summarization goals: Future work could investigate how different summarization goals (e.g., describing an event vs. providing a biography) affect the degree to which humans employ rewriting versus extraction.
- Generative approaches: Given that humans appear to be generating summary text that differs widely from source sentences, the authors suggest that approaches incorporating generation and reformulation techniques hold significant promise for multi-document summarization.
Summary updated