Embedding Knowledge for Document Summarization: A Survey

Xindong Wu (吴信东), Yutong Qu, Wei Emma Zhang, Jian Yang, Lingfei Wu, Xindong WuarXiv 2022 · DOI · arXiv

Introduction

While Large Language Models (LLMs) have become remarkably proficient at generating fluent text, they frequently struggle with factual consistency. In document summarization (DS), models often "hallucinate" information absent from the source text or fail to capture the logical progression of complex arguments. This stems from standard models treating documents as simple token sequences, lacking the structural understanding or "common sense" required for truthful and coherent summaries.

This survey provides a systematic framework for knowledge-aware document summarization, categorizing how auxiliary information—such as knowledge graphs, discourse trees, and topic models—is integrated into deep learning architectures to ground the generation process.

A Taxonomy of Knowledge

The authors propose a comprehensive taxonomy to classify the types of knowledge used to enhance summarization models. These categories represent the different ways researchers "pump" external facts and linguistic structures into AI systems:

  • Native Knowledge: Raw, unprocessed textual data used as auxiliary input.
  • Linguistic Knowledge: Information regarding lexis, syntax (e.g., dependency trees), and discourse (how sentences logically relate to one another).
  • Semantic Knowledge: Conceptual and factual information typically stored as (subject, predicate, object) triplets. This includes "Closed" knowledge from curated databases (like Wikidata) and "Open" knowledge extracted directly from the source text.
  • Topical Knowledge: Latent themes or salience information derived from topic models like LDA.
The taxonomy of knowledge types used in document summarization, illustrating the overlap between linguistic, semantic, and topical domains.

Embedding and Architectural Integration

To utilize this knowledge, it must be converted into mathematical representations. The survey distinguishes between Text Embeddings (tokens, sentences), Graph Embeddings (nodes, entities, edges), Topic Embeddings, and Position Embeddings. These representations are then integrated into four primary architectural types:

  1. Naive Models: Standard Deep Neural Networks (DNNs).
  2. Graph-based Models: Utilizing Graph Convolutional Networks (GCNs) or Graph Attention Networks (GATs) to process structured knowledge.
  3. Encoder-Decoder Models: Integrating knowledge directly into the attention mechanisms of Seq2Seq models.
  4. Pre-trained Language Model (PLM) based Models: Building on foundations like BERT, RoBERTa, or BART.

The Landscape of Knowledge-Aware Models

The survey maps dozens of state-of-the-art models—including BERTSUM, PEGASUS, ASGARD, and BartGraphSumm—to the specific knowledge types and embeddings they employ across various summarization tasks (Single-Document vs. Multi-Document, Abstractive vs. Extractive).

A comprehensive mapping of summarization methods to their respective knowledge types, tasks, and model architectures.

Key Insights and Future Directions

The authors identify Factual Consistency (G5) as a critical fifth goal for summarization, joining the traditional metrics of coverage, non-redundancy, readability, and relevancy. Knowledge-embedded summarizers consistently outperform "vanilla" models by grounding generation in verified entities and relations, thereby reducing hallucination rates.

However, several gaps remain for future research:

  • Knowledge Quality: Summarization performance is heavily dependent on the accuracy of the underlying Knowledge Graph or extraction tool.
  • Multi-facet Fusion: Most current models utilize only one type of knowledge; there is a significant need for architectures that can simultaneously blend linguistic, semantic, and topical data.
  • Prompting Strategies: While prompting is the current paradigm in NLP, its potential for injecting structured knowledge into summarization models remains largely under-explored.

Summary updated

BibTeX
@misc{2022embedding,
  author = {Xindong Wu (吴信东) and Yutong Qu and Wei Emma Zhang and Jian Yang and Lingfei Wu and Jia Wu and Xindong Wu},
  title = {{Embedding Knowledge for Document Summarization: A Survey}},
  year = {2022},
  eprint = {2204.11190},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.11190},
}
Figure