Data Augmentation for Abstractive Query-Focused Multi-Document Summarization

Ramakanth Pasunuru, Asli Celikyilmaz, Michel Galley, Chenyan Xiong, Yizhe Zhang, Mohit Bansal, Jianfeng Gao

Abstract

The progress in Query-focused Multi-Document Summarization (QMDS) has been limited by the lack of sufficient large-scale high-quality training datasets. We present two QMDS training datasets, which we construct using two data augmentation methods: (1) transferring the commonly used single-document CNN/Daily Mail summarization dataset to create the QMDSCNN dataset, and (2) mining search-query logs to create the QMDSIR dataset. These two datasets have complementary properties, i.e., QMDSCNN has real summaries but queries are simulated, while QMDSIR has real queries but simulated summaries. To cover both these real summary and query aspects, we build abstractive end-to-end neural network models on the combined datasets that yield new state-of-the-art transfer results on DUC datasets. We also introduce new hierarchical encoders that enable a more efficient encoding of the query together with multiple documents. Empirical results demonstrate that our data augmentation and encoding methods outperform baseline models on automatic metrics, as well as on human evaluations along multiple attributes.

Introduction

While Single-Document Summarization (SDS) has flourished thanks to massive datasets like CNN/Daily Mail, Query-focused Multi-Document Summarization (QMDS) has lagged behind. QMDS is the task of generating a summary from multiple documents that specifically answers a user's query—a scenario much closer to how we actually use search engines or virtual assistants.

Illustration of the QMDS task, showing how a query and multiple input documents are synthesized into a single gold summary.

The challenge is two-fold: the lack of large-scale, high-quality datasets where multiple documents are paired with human-written summaries for specific queries, and the modeling complexity required to understand both local document context and global inter-document relationships relative to the query.

Methodology: The HEROSumm Model

We propose a new model architecture, HEROSumm (HiErarchical QueRy focused Order-aware multi-document Summarization), which introduces three key components to handle these complexities:

  • Query Encoder: Instead of simply prepending the query, a dedicated transformer layer conditions document features based on the query.
  • Hierarchical Encoding: HEROSumm allows the decoder to attend to both local layers (specific document details) and global layers (inter-document relationships), ensuring no salient detail is lost.
  • Self-Ordering Component: Rather than relying on an external ranker, HEROSumm learns to rank document importance end-to-end using a novel sinusoidal positional encoding based on importance scores from a self-attention mechanism.
Architectural comparison between a baseline encoder-decoder QMDS model (left) and the proposed HEROSumm encoder (right).

Data Augmentation Strategies

Our core contribution is the creation of two complementary datasets to bridge the data gap:

  1. QMDSCNN (~300K samples): Created by restructuring the CNN/Daily Mail dataset. Article titles serve as queries, and the main text is "chunked" into multiple smaller documents. This provides high-quality human summaries but with simulated queries.
  2. QMDSIR (~100K samples): Mined from Bing search logs. This uses real user queries and the "answer passage" shown in search results as the gold summary, providing real-world queries but automatically generated summaries.

By combining a dataset with "real summaries/fake queries" (QMDSCNN) and one with "real queries/fake summaries" (QMDSIR), the model learns to handle both aspects of the task effectively.

Key Results

Empirical results demonstrate that our data augmentation and encoding methods outperform baseline models. On the QMDSIR dataset, the HEROSumm-Joint model achieved a ROUGE-L score of 41.15, a significant improvement over the 39.40 baseline. Furthermore, models trained on these augmented datasets significantly outperformed those trained on WikiSum when tested on human-annotated DUC datasets. Human evaluations on Amazon Mechanical Turk confirmed these gains, with our model winning on informativeness, coherence, and document ordering by a wide margin.

Conclusion

This research addresses the data scarcity bottleneck in QMDS through innovative data augmentation and a hierarchical model architecture. By leveraging existing news data and search logs, we enable the training of robust abstractive models that achieve state-of-the-art performance on classic benchmarks.

Summary updated

Abstract

The progress in Query-focused Multi-Document Summarization (QMDS) has been limited by the lack of sufficient largescale high-quality training datasets. We present two QMDS training datasets, which we construct using two data augmentation methods: (1) transferring the commonly used single-document CNN/Daily Mail summarization dataset to create the QMDSCNN dataset, and (2) mining search-query logs to create the QMDSIR dataset. These two datasets have complementary properties, i.e., QMDSCNN has real summaries but queries are simulated, while QMDSIR has real queries but simulated summaries. To cover both these real summary and query aspects, we build abstractive end-to-end neural network models on the combined datasets that yield new state-of-the-art transfer results on DUC datasets. We also introduce new hierarchical encoders that enable a more efficient encoding of the query together with multiple documents. Empirical results demonstrate that our data augmentation and encoding methods outperform baseline models on automatic metrics, as well as on human evaluations along multiple attributes.
BibTeX
@inproceedings{pasunuru2021data,
  author = {Ramakanth Pasunuru and Asli Celikyilmaz and Michel Galley and Chenyan Xiong and Yizhe Zhang and Mohit Bansal and Jianfeng Gao},
  title = {{Data Augmentation for Abstractive Query-Focused Multi-Document Summarization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=aEbjuTMVNS},
}
Figure