Introduction
Standard sequence-to-sequence (seq2seq) models have demonstrated significant success in short-text summarization tasks, such as headline generation. However, their performance often degrades when applied to full-length documents. Two primary issues persist: information loss, where models fail to capture the document's salient points, and repetition, where decoders produce redundant, nonsensical multi-sentence summaries by repeatedly attending to the same source content.
This research addresses these challenges by incorporating the inherent hierarchical structure of documents (words $\rightarrow$ sentences $\rightarrow$ document) into the modeling process. The authors introduce structural regularization—specifically "structural-compression" and "structural-coverage"—within a hierarchical encoder-decoder framework to guide the model toward generating concise and comprehensive summaries.
The Hierarchical Framework
The proposed model utilizes a hierarchical encoder-decoder architecture. The encoder operates at both word and sentence levels, capturing local context and global document structure. The decoder is similarly hierarchical, employing a hybrid sentence-word attention mechanism to bridge the gap between human and machine summaries.
Structural Regularization Terms
The core innovation lies in two regularization terms added to the loss function and the decoding process:
- Structural-Compression: This term encourages the model to be "decisive." By applying an entropy-based penalty to sentence-level attention, the model is forced to focus each summary sentence on a few specific, salient source sentences rather than spreading attention across the entire document.
- Structural-Coverage: To prevent repetition, this term tracks which source sentences have already been "covered" by previous summary sentences. It penalizes the model for re-attending to previously used information, forcing the discovery of new content for subsequent sentences.
These constraints are applied during training and integrated into a novel Hierarchical Beam Search algorithm during inference, which ranks candidates based on both likelihood and structural scores.
Experimental Results
The model was evaluated on the CNN/Daily Mail dataset, a benchmark for long-form summarization. The proposed method achieved a new state-of-the-art ROUGE-1 score of 40.30, significantly outperforming the popular "Pointer-Generator + Coverage" baseline (39.53) and extractive models like SummaRuNNer (39.6).
Qualitative Analysis of Attention
The effectiveness of structural regularization is most visible in the attention patterns. As shown in the heatmaps below, the proposed method (e) closely approximates the "Gold Reference" (a) by maintaining sharp, diverse attention across different source sentences ($I_n$) for each output sentence ($O_m$). In contrast, baseline models often exhibit blurred attention or get stuck on specific source sentences, leading to redundancy.
Human evaluations further confirmed these findings, with the proposed model receiving significantly higher ratings for informativeness, conciseness, and coherence compared to baselines. The model showed its greatest strength on long summaries (>150 words), where it outperformed the baseline by nearly 5 ROUGE-1 points.
Conclusion
By mathematically modeling the logical properties of human summarization—compression and coverage—this work bridges the gap between standard seq2seq models and high-quality document summaries. While the hierarchical approach introduces some computational complexity, it provides a robust framework for incorporating document structure into neural abstractive summarization. Future work may explore more complex rhetorical relations to further enhance summary coherence.
Summary updated