Document Summarization with VHTM: Variational Hierarchical Topic-Aware Mechanism

Xiyan Fu, Jun Wang, Jinghan Zhang, Jinmao Wei, Zhenglu YangAAAI 2020 · DOI

Abstract

VHTM (Variational Hierarchical Topic-Aware Mechanism) is an end-to-end summarization framework designed to infer topics and generate summaries simultaneously. By integrating multi-grained topic information—at both the document and paragraph levels—directly into a Variational Encoder-Decoder architecture, the model produces more semantically consistent and accurate summaries than traditional approaches relying on external, pre-trained topic models.

Introduction

In automatic text summarization, "topic awareness" is a critical component for ensuring that generated summaries remain focused on overarching themes rather than irrelevant details. Traditionally, researchers have utilized external models like Latent Dirichlet Allocation (LDA) to provide this guidance. However, these "off-the-shelf" approaches suffer from three major limitations: a lack of task specificity (extracting all topics instead of just the most informative ones), coarse granularity (ignoring paragraph-level shifts), and sub-optimal fusion between disparate mathematical frameworks.

Methodology

VHTM addresses these challenges by employing a Variational Encoder-Decoder (VED) framework that treats topics as latent variables learned during training.

The VHTM architecture consists of three main components: Neural Topic Inference, a Hierarchical Encoder, and a Decoder.

The architecture is composed of three primary modules:

  1. Neural Topic Inference: A VAE-based neural topic model predicts a K-dimensional topic distribution from the document's bag-of-words representation.
  2. Hierarchical Topic Mechanism:
    • Topic Embedding (Global): Document-level topic vectors are combined with BERT-enhanced dense word embeddings to disambiguate words based on global context.
  3. Topic Attention (Local): The model infers topics for individual document segments (paragraphs) and uses a specialized attention mechanism to prioritize relevant subtopics during generation.
  4. Joint Training: The system is optimized using a loss function that balances summary accuracy with topic coherence.

Experimental Setup

The model was evaluated on the CNN/Daily Mail corpus, a standard benchmark for long-form summarization containing over 280,000 document-summary pairs. Documents in this dataset average 761 tokens, while summaries average 46 tokens.

Results and Discussion

VHTM demonstrated superior performance over strong baselines, including the Pointer-Generator + Coverage model. Quantitatively, VHTM achieved a ROUGE-1 score of 40.57, ROUGE-2 of 18.05, and ROUGE-L of 37.18, significantly surpassing the pg + cov baseline (39.53, 17.28, and 36.38 respectively).

Qualitatively, visualization of the learned topics confirms the model's ability to cluster semantically related terms effectively.

Heatmap visualization showing how VHTM groups semantically related words into coherent topic clusters.

Key Insights

  • End-to-End Integration: Internalizing topic inference allows the model to tailor topics specifically for the summarization task.
  • Local Context Priority: Ablation studies indicated that paragraph-level Topic Attention provided a more substantial performance boost than document-level Topic Embedding, highlighting the importance of tracking thematic shifts in long texts.
  • BERT + Topic Synergy: Combining BERT's general representations with topic-specific embeddings refined the model's discriminative power.

Conclusion and Future Work

VHTM proves that joint learning of topics and summaries leads to more coherent outputs. Current limitations include the use of fixed paragraph segments and high computational complexity. Future research may explore dynamic document segmentation and extending this hierarchical approach to multi-document summarization tasks.

Summary updated

BibTeX
@inproceedings{fu2020document,
  author = {Xiyan Fu and Jun Wang and Jinghan Zhang and Jinmao Wei and Zhenglu Yang},
  title = {{Document Summarization with VHTM: Variational Hierarchical Topic-Aware Mechanism}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i05.6277},
}
Figure