Data Augmentation for Abstractive Query-Focused Multi-Document Summarization

Ramakanth Pasunuru, Asli Celikyilmaz, Michel Galley, Chenyan Xiong, Yizhe Zhang, Mohit Bansal, Jianfeng Gao

Abstract

The progress in Query-focused Multi-Document Summarization (QMDS) has been limited by the lack of sufficient large-scale high-quality training datasets. We present two QMDS training datasets, which we construct using two data augmentation methods: (1) transferring the commonly used single-document CNN/Daily Mail summarization dataset to create the QMDSCNN dataset, and (2) mining search-query logs to create the QMDSIR dataset. These two datasets have complementary properties, i.e., QMDSCNN has real summaries but queries are simulated, while QMDSIR has real queries but simulated summaries. To cover both these real summary and query aspects, we build abstractive end-to-end neural network models on the combined datasets that yield new state-of-the-art transfer results on DUC datasets. We also introduce new hierarchical encoders that enable a more efficient encoding of the query together with multiple documents. Empirical results demonstrate that our data augmentation and encoding methods outperform baseline models on automatic metrics, as well as on human evaluations along multiple attributes.

Introduction

Summarizing a single document is a well-established task supported by massive datasets like CNN/Daily Mail. However, Query-focused Multi-Document Summarization (QMDS)—where a model synthesizes an answer to a specific query from a collection of documents—presents a significantly higher challenge. The primary bottleneck in advancing QMDS is data scarcity. High-quality datasets require human-written summaries that span multiple documents and specifically address a query. Existing datasets are often too small for deep learning or use generic "topics" as queries, which do not reflect natural information-seeking behavior.

Data Augmentation Strategies

To address the data scarcity problem, we propose two novel data augmentation methods that create large-scale training sets with complementary strengths.

  1. QMDSCNN: This dataset is derived from the single-document CNN/Daily Mail corpus. We "multi-document-ify" it by using article titles as queries and chunking the main article into smaller documents. We then use the BM25 algorithm to retrieve related paragraphs from other articles to provide additional context. This results in a dataset with high-quality, human-written summaries but simulated queries.
  2. QMDSIR: This dataset is constructed by mining Bing search logs for queries that triggered "answer passages." We use the original search query, the answer passage as the gold summary, and the top-ranked search results as the input documents. This provides a dataset with real, natural language queries but simulated summaries.
Illustration of the QMDS task and the QMDSIR dataset structure.

The combination of these datasets allows models to learn from both high-quality prose and realistic user queries, leading to more robust performance.

The HEROSumm Model

We introduce HEROSumm (HiErarchical QueRy focused Order-aware multi-document Summarization), a novel architecture designed to handle the complexities of QMDS more effectively than standard transformers.

  • Hierarchical Encoding: The model utilizes both local (intra-document) and global (inter-document) encoder layers. This allows the decoder to attend to specific facts within a document while maintaining a coherent narrative across the entire set.
  • Query Component: A dedicated transformer layer encodes the query separately, ensuring that document representations are conditioned on the specific information need from the outset.
  • Ordering Component: Instead of relying on fixed external rankings, HEROSumm learns to rank and order the importance of documents internally during training, reducing errors from noisy retrieval.
The HEROSumm architecture featuring hierarchical encoders and a transformer decoder.

Experimental Results

Our empirical evaluation shows that models trained on the augmented QMDSCNN and QMDSIR datasets significantly outperform those trained on existing large-scale datasets like WikiSum when transferred to "gold standard" benchmarks such as DUC 2006 and 2007.

On the QMDSIR dataset, the HEROSumm-Joint model achieved a ROUGE-L score of 41.15, marking a substantial improvement over the baseline of 39.40. Human evaluations on Amazon Mechanical Turk further confirmed these findings, with summaries generated using our augmented data being preferred across all metrics: Informativeness, Non-redundancy, Coherence, and Order. For instance, on DUC 2006, our model's informativeness was preferred by a margin of 107 to 30 over the WikiSum-trained baseline.

Conclusion and Future Work

This work demonstrates that data augmentation is a powerful tool for overcoming scarcity in QMDS. By combining datasets with complementary properties—real summaries with simulated queries and vice versa—we can train models that generalize better to real-world tasks.

Future research will explore more sophisticated ways to combine these datasets, such as weighted sampling or curriculum learning. Additionally, while our model excels in informativeness and coherence, improving "focus" remains an area for further refinement. Bridging the performance gap between extractive and abstractive models also remains a key challenge for the field.

Summary updated

Abstract

The progress in Query-focused Multi-Document Summarization (QMDS) has been limited by the lack of sufficient largescale high-quality training datasets. We present two QMDS training datasets, which we construct using two data augmentation methods: (1) transferring the commonly used single-document CNN/Daily Mail summarization dataset to create the QMDSCNN dataset, and (2) mining search-query logs to create the QMDSIR dataset. These two datasets have complementary properties, i.e., QMDSCNN has real summaries but queries are simulated, while QMDSIR has real queries but simulated summaries. To cover both these real summary and query aspects, we build abstractive end-to-end neural network models on the combined datasets that yield new state-of-the-art transfer results on DUC datasets. We also introduce new hierarchical encoders that enable a more efficient encoding of the query together with multiple documents. Empirical results demonstrate that our data augmentation and encoding methods outperform baseline models on automatic metrics, as well as on human evaluations along multiple attributes.
BibTeX
@inproceedings{pasunuru2021data,
  author = {Ramakanth Pasunuru and Asli Celikyilmaz and Michel Galley and Chenyan Xiong and Yizhe Zhang and Mohit Bansal and Jianfeng Gao},
  title = {{Data Augmentation for Abstractive Query-Focused Multi-Document Summarization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=aEbjuTMVNS},
}
Figure