Introduction
Extractive summarization aims to select the most salient sentences from a document to form a concise summary. While pre-trained transformer models like BERT and RoBERTa have significantly advanced the field, they often struggle with document-level coherence. These models typically rely on a single [CLS] token to represent sentence meaning, which may be insufficient for capturing complex inter-sentence relationships. Furthermore, RoBERTa's lack of "Next Sentence Prediction" training makes it less inherently aware of discourse structure, often resulting in disjointed summaries.
Methodology: The NeRoBERTa Model
To address these limitations, the authors introduce NeRoBERTa (Nested RoBERTa). This model incorporates a structural "scaffold" onto the transformer encoder by nesting two types of graph structures:
- Intra-sentence (Syntactic) Graphs: For each sentence, a dependency tree is constructed. Instead of the abstract
[CLS]token, the model utilizes the "root" word of this tree to represent the sentence's core semantic content. - Inter-sentence (Discourse) Graphs: Utilizing Rhetorical Structure Theory (RST), the model builds a tree connecting sentences based on their rhetorical roles (e.g., nucleus and satellite).
- Nested Graph Attention: These structures are integrated using Graph Attention Networks (GAT), allowing information to flow hierarchically from word-level syntax to document-level discourse.
To facilitate sentence boundary recognition in RoBERTa, which lacks segment embeddings, the authors added randomly initialized segment embeddings to the encoder.
Experimental Setup
The model was evaluated on the CNN/DailyMail dataset, a standard benchmark for extractive summarization containing over 300,000 news articles.
Key Insights
- Structural Grounding: Using the syntactic root provides a more grounded representation than the
[CLS]token, ensuring the model focuses on the most structurally significant parts of a sentence. - Hierarchical Context: The nested approach enables the model to capture long-range dependencies, such as recognizing the relevance of a keyword in an early sentence to a claim made much later in the document.
- Coherence Focus: Beyond standard overlap metrics, NeRoBERTa is designed to improve the logical flow and readability of generated summaries.
Results and Discussion
NeRoBERTa demonstrates superior performance across both automatic and human evaluations.
As shown in the table above, NeRoBERTa achieved a ROUGE-1 score of 43.86, outperforming the standard RoBERTa baseline (43.55) and BERTSUM (43.25). Human evaluations further confirmed these findings, with NeRoBERTa receiving significantly higher scores for coherence (4.08) compared to the baseline (4.02). Ablation studies indicated that the combination of syntactic and discourse information is crucial, as the nested model outperformed versions using only one type of structural information.
Limitations and Future Work
The current implementation of NeRoBERTa depends on external dependency and RST parsers. Errors in these parsers can propagate through the model. Future research could explore end-to-end training of the tree-building and summarization processes. Additionally, applying this nested structure to longer documents, where discourse flow is even more critical, remains a promising direction.
Summary updated