TL;DR
SemSUM is a novel neural abstractive summarization model that utilizes semantic dependency graphs to mitigate "hallucinations" and factual errors. By explicitly encoding the relationships between subjects, verbs, and objects, it ensures that generated summaries remain faithful to the original text, outperforming standard Transformer models across multiple benchmarks.
The Challenge: Beyond Fluency to Factual Accuracy
While modern AI is remarkably proficient at generating fluent, human-like text, it often struggles with factual consistency. In abstractive summarization—where a model rewrites a long sentence into a concise summary—standard sequence-to-sequence (Seq2seq) models frequently suffer from semantic irrelevance and content deviation.
For example, given a source sentence about "rivals hoping Roger Federer wins," a standard model might output "Federer hoping to win." While the keywords are present, the model fails to grasp the underlying logic: who is doing the hoping? This occurs because standard models rely heavily on word co-occurrence and proximity rather than the formal logical structure of the sentence.
Methods: The SemSUM Architecture
The authors introduce SemSUM (Semantic Dependency Guided Summarization), which enhances the popular Transformer architecture with a specialized Graph Encoder designed to capture the "who did what to whom" of a sentence.
- Semantic Dependency Graphs: Before summarization, the input sentence is processed by a parser to create a graph where nodes represent words and edges represent semantic relations (e.g., identifying a word as the agent of a specific verb).
- The Stacked Encoder:
- Sentence Encoder: A standard Transformer encoder creates initial contextual representations of the words.
- Graph Encoder: The core innovation. It uses a Graph Attention Network (GAT) to update word representations based on their neighbors in the semantic graph. Crucially, it treats "incoming" and "outgoing" relations differently, allowing the model to distinguish between the doer and the receiver of an action.
- Layer Aggregation: To handle varying distances in semantic relations, the model uses a Bidirectional LSTM to aggregate information across different layers of the graph encoder. This allows the model to adaptively decide whether a word needs a "local" view or a "wide-range" view of the sentence structure.
- The Decoder: A standard Transformer decoder uses these semantics-aware representations to generate the final summary.
Key Insights
- Structure over Proximity: By using a graph, the model bridges "long-distance" dependencies. If a subject and its verb are separated by a long parenthetical phrase, the graph provides a direct shortcut that standard self-attention might dilute.
- Directionality Matters: The authors found that using separate mathematical mappings for "sender" and "receiver" nodes in the graph attention mechanism was vital for performance.
- Adaptive Neighborhoods: The use of a Bi-LSTM for layer aggregation recognizes that some words' meanings are clear from immediate neighbors, while others require looking several "hops" away in the semantic graph.
Key Results
SemSUM consistently outperformed strong baselines, including the standard Transformer and other state-of-the-art models, across three major benchmarks: English Gigaword, DUC 2004, and MSR-ATC.
- Quantitative Performance: On the Gigaword dataset, SemSUM achieved a ROUGE-2 score of 19.75, a significant improvement over the standard Transformer's 18.08. It also showed superior performance in BERTScore, which measures semantic similarity rather than just word overlap.
- Human Evaluation: In blind tests, human judges rated SemSUM significantly higher in faithfulness (3.58 vs. 3.03 for the Transformer). This confirms that the graph guidance successfully reduced factual distortions.
Limitations and Future Work
The primary limitation is that SemSUM depends on an external "off-the-shelf" semantic parser. If the parser creates a noisy or incorrect graph, the summarization model may inherit those errors. Future work could involve incorporating even richer structures, such as Knowledge Graphs or Abstract Meaning Representation (AMR), to provide deeper logical grounding for the model.
Summary updated