Abstract
Extractive document summarization aims to conclude given documents by extracting some salient sentences. Often, it faces two challenges: 1) how to model the information redundancy among candidate sentences; 2) how to select the most appropriate sentences. This paper attempts to build a strong summarizer, DivSelect+CNNLM, by presenting new algorithms to optimize each of them. Concretely, it proposes CNNLM, a novel neural network language model based on a convolutional neural network (CNN), to project sentences into dense distributed representations, then models sentence redundancy by cosine similarity. Afterwards, it formulates the selection process as an optimization problem, constructing a diversified selection process (DivSelect) with the aim of selecting sentences which have high prestige while remaining dissimilar to each other. Experimental results on DUC2002 and DUC2004 benchmark data sets demonstrate the effectiveness of our approach.
Introduction
Extractive summarization—the process of identifying and extracting the most representative sentences from a document—is a cornerstone of natural language processing. However, traditional approaches often struggle with two core issues. First is Semantic Blindness: methods relying on word-frequency metrics like TF-IDF fail to recognize semantic overlaps where different words convey identical meanings. Second is the Redundancy Trap: simply selecting the highest-ranked sentences often leads to repetitive summaries. Balancing "prestige" (sentence centrality) with "diversity" (information novelty) typically relies on heuristics. This work proposes a unified mathematical framework to address both challenges.
Methodology
CNNLM: Sentence Modeling
To capture deep semantics, the authors developed an unsupervised Convolutional Neural Network Language Model (CNNLM). Unlike traditional CNNs that require task-specific labels, this model is trained as a language model to predict the next word in a sequence, ensuring the resulting sentence vectors are "unbiased" and capture general meaning.
As shown in the architecture above, the model applies convolution to extract local phrase features, followed by a max-pooling layer to identify globally significant features. These features form a dense 300-dimensional vector. Sentence similarity is then calculated using the cosine similarity between these distributed representations.
DivSelect: Diversified Sentence Selection
The selection process is formulated as a global optimization problem. The objective function $Q(C)$ seeks a subset of sentences that maximizes the difference between total prestige and a redundancy penalty:
$$\text{Objective} = \text{Total Prestige} - \text{Redundancy (Similarity weighted by Prestige)}$$
A key innovation in DivSelect is the prestige-weighted penalty. By scaling similarity by the prestige of the sentences involved, the system more aggressively penalizes the inclusion of two highly central but redundant sentences. While the authors prove this selection problem is NP-hard (equivalent to the "Densest k-Subgraph" problem), they also demonstrate that the objective function is submodular. This property of diminishing returns allows for a greedy iterative algorithm that is guaranteed to reach a near-optimal solution efficiently.
Experimental Results
The framework was evaluated on the DUC2002 and DUC2004 benchmark datasets, using one million sentences from the English Gigaword corpus for CNNLM pre-training.
Performance Comparison
DivSelect+CNNLM consistently outperformed strong baselines such as LexRank and DivRank. On DUC2002, the system achieved a ROUGE-1 score of 0.510, compared to LexRank's 0.473. On DUC2004, it reached 0.409, surpassing DivRank's 0.388.
Ablation Study
To isolate the impact of the CNNLM representations, the authors conducted an ablation study replacing CNNLM with other modeling techniques, including Vector Space Models (VSM), Latent Semantic Analysis (LSA), and Paragraph Vectors (PV).
As indicated in the table, CNNLM consistently yields the highest ROUGE-1 scores across both datasets. This confirms that the deep convolutional features provide a superior understanding of sentence semantics compared to traditional or alternative neural embedding methods.
Conclusion
The DivSelect+CNNLM framework provides a robust, mathematically grounded approach to extractive summarization. By combining unsupervised deep learning for semantic modeling with submodular optimization for selection, the system effectively balances information importance and diversity. Future work may extend this framework to query-focused summarization, where sentence prestige is dynamically weighted by relevance to a specific user query.
Summary updated