Improving Neural Abstractive Document Summarization with Structural Regularization

Wei Li, Xinyan Xiao, Yajuan Lyu, Yuanzhuo WangEMNLP 2018 · OpenReview

Introduction

Standard sequence-to-sequence (seq2seq) models have demonstrated significant success in short-text summarization tasks, such as headline generation. However, their performance often degrades when applied to full-length documents. Two primary issues persist: information loss, where models fail to capture the document's salient points, and repetition, where decoders produce redundant, nonsensical multi-sentence summaries by repeatedly attending to the same source content.

This research addresses these challenges by incorporating the inherent hierarchical structure of documents (words $\rightarrow$ sentences $\rightarrow$ document) into the modeling process. The authors introduce structural regularization—specifically "structural-compression" and "structural-coverage"—within a hierarchical encoder-decoder framework to guide the model toward generating concise and comprehensive summaries.

The Hierarchical Framework

The proposed model utilizes a hierarchical encoder-decoder architecture. The encoder operates at both word and sentence levels, capturing local context and global document structure. The decoder is similarly hierarchical, employing a hybrid sentence-word attention mechanism to bridge the gap between human and machine summaries.

Architecture of the hierarchical encoder-decoder with structural regularization

Structural Regularization Terms

The core innovation lies in two regularization terms added to the loss function and the decoding process:

  1. Structural-Compression: This term encourages the model to be "decisive." By applying an entropy-based penalty to sentence-level attention, the model is forced to focus each summary sentence on a few specific, salient source sentences rather than spreading attention across the entire document.
  2. Structural-Coverage: To prevent repetition, this term tracks which source sentences have already been "covered" by previous summary sentences. It penalizes the model for re-attending to previously used information, forcing the discovery of new content for subsequent sentences.

These constraints are applied during training and integrated into a novel Hierarchical Beam Search algorithm during inference, which ranks candidates based on both likelihood and structural scores.

Experimental Results

The model was evaluated on the CNN/Daily Mail dataset, a benchmark for long-form summarization. The proposed method achieved a new state-of-the-art ROUGE-1 score of 40.30, significantly outperforming the popular "Pointer-Generator + Coverage" baseline (39.53) and extractive models like SummaRuNNer (39.6).

Qualitative Analysis of Attention

The effectiveness of structural regularization is most visible in the attention patterns. As shown in the heatmaps below, the proposed method (e) closely approximates the "Gold Reference" (a) by maintaining sharp, diverse attention across different source sentences ($I_n$) for each output sentence ($O_m$). In contrast, baseline models often exhibit blurred attention or get stuck on specific source sentences, leading to redundancy.

Attention heatmaps comparing the proposed method and baselines to the gold reference

Human evaluations further confirmed these findings, with the proposed model receiving significantly higher ratings for informativeness, conciseness, and coherence compared to baselines. The model showed its greatest strength on long summaries (>150 words), where it outperformed the baseline by nearly 5 ROUGE-1 points.

Conclusion

By mathematically modeling the logical properties of human summarization—compression and coverage—this work bridges the gap between standard seq2seq models and high-quality document summaries. While the hierarchical approach introduces some computational complexity, it provides a robust framework for incorporating document structure into neural abstractive summarization. Future work may explore more complex rhetorical relations to further enhance summary coherence.

Summary updated

BibTeX
@inproceedings{li2018improving,
  author = {Wei Li and Xinyan Xiao and Yajuan Lyu and Yuanzhuo Wang},
  title = {{Improving Neural Abstractive Document Summarization with Structural Regularization}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  url = {https://openreview.net/forum?id=r14-C-GdZH},
}
Figure