SAE-NTM: Sentence-Aware Encoder for Neural Topic Modeling

Hao Liu, Jingsheng Gao, Suncheng Xiang, Ting Liu, Yuzhuo Fu2023 · DOI

Abstract

Incorporating external knowledge, such as pre-trained language models (PLMs), into neural topic modeling has achieved great success in recent years. However, employing PLMs for topic modeling generally ignores the maximum sequence length of PLMs and the interaction between external knowledge and bag-of-words (BOW). To this end, we propose a sentence-aware encoder for neural topic modeling, which adopts fine-grained sentence embeddings as external knowledge to entirely utilize the semantic information of input documents. We introduce sentence-aware attention for document representation, where BOW enables the model to attend on topical sentences that convey topic-related cues. Experiments on three benchmark datasets show that our framework outperforms other state-of-the-art neural topic models in topic coherence. Further, we demonstrate that the proposed approach can yield better latent document-topic features through improvement on the document classification.

Introduction

Topic modeling is a fundamental task in natural language processing aimed at discovering latent thematic structures within large text corpora. Traditionally, this has relied on the Bag-of-Words (BOW) representation, which, while effective for capturing word frequencies, fails to account for semantic context and word order. Recent advancements have sought to integrate Pre-trained Language Models (PLMs) like BERT to enrich these models with contextual semantics. However, two primary challenges persist: the strict input length limitations of PLMs (typically 512 tokens) and the representation gap between sparse BOW data and dense contextual embeddings. Standard concatenation of these features often fails to capture the intricate interactions between global word counts and local semantic information.

Methodology

The proposed SAE-NTM framework addresses these limitations through a novel Sentence-Aware Encoder (SAE) integrated into a Variational Autoencoder (VAE) architecture.

Architecture of SAE-NTM featuring the Sentence-Aware Encoder and Variational Inference module.

The SAE-NTM process involves three critical components:

  1. Sentence Encoding: Documents are decomposed into individual sentences, each processed by a Sentence-Transformer to generate dense embeddings. This approach effectively bypasses the PLM token limit while preserving fine-grained local context.
  2. BOW Guidance: Global document features are extracted from BOW data to serve as "topic cues," representing the overall thematic content.
  3. Sentence-Aware Attention: A cross-attention mechanism uses the global BOW features to weight the individual sentence embeddings. This allows the model to prioritize sentences that are most relevant to the document's overall topics, filtering out generic or noisy text.

The resulting document representation is used to infer a latent topic distribution, which the decoder then uses to reconstruct the original BOW counts.

Experimental Setup

The model was evaluated on three benchmark datasets that vary in document length and vocabulary size:

  • 20Newsgroups (20NG): A collection of approximately 18,000 newsgroup posts across 20 topics.
  • Wikitext-103 (Wiki): Over 100 million tokens extracted from Wikipedia articles, representing long-form text.
  • IMDb: A dataset of 50,000 movie reviews.

Results and Discussion

SAE-NTM was compared against several state-of-the-art models, including CTM and SCHOLAR+BAT.

Topic Coherence

The primary metric for evaluation was Normalized Pointwise Mutual Information (NPMI). SAE-NTM consistently achieved superior performance across all datasets and topic counts (K=50, K=200).

Comparison of Topic Coherence (NPMI) across three datasets for K=50 and K=200 topics.

For instance, on the Wikitext-103 dataset with K=50, SAE-NTM reached an NPMI of 0.511, surpassing the next best model (0.501). This indicates that the sentence-aware attention mechanism successfully captures more semantically coherent topics by filtering out non-topical noise.

Document Classification and Qualitative Analysis

To validate the utility of the learned latent features, the authors performed document classification. On the 20NG dataset, SAE-NTM achieved 66.1% accuracy, outperforming the best baseline (65.2%). Qualitatively, the model produced more specific and relevant topic words. For a "Music" related topic, SAE-NTM identified terms like cantata, bach, and recitative, whereas baselines often included less specific terms like painting.

Conclusion

SAE-NTM demonstrates that bridging the gap between BOW and PLMs through sentence-level attention significantly enhances neural topic modeling. By focusing on topical sentences and bypassing length constraints, the model yields more coherent topics and more descriptive document representations. Future work may explore fine-tuning the PLM during training and enhancing the decoder with factorized word embeddings.

Summary updated

Abstract

Incorporating external knowledge, such as pre-trained language models (PLMs), into neural topic modeling has achieved great success in recent years. However, employing PLMs for topic modeling generally ignores the maximum sequence length of PLMs and the interaction between external knowledge and bag-of-words (BOW). To this end, we propose a sentence-aware encoder for neural topic modeling, which adopts fine-grained sentence embeddings as external knowledge to entirely utilize the semantic information of input documents. We introduce sentence-aware attention for document representation, where BOW enables the model to attend on topical sentences that convey topic-related cues. Experiments on three benchmark datasets show that our framework outperforms other state-of-the-art neural topic models in topic coherence. Further, we demonstrate that the proposed approach can yield better latent document-topic features through improvement on the document classification.
BibTeX
@misc{liu2023sae,
  author = {Hao Liu and Jingsheng Gao and Suncheng Xiang and Ting Liu and Yuzhuo Fu},
  title = {{SAE-NTM: Sentence-Aware Encoder for Neural Topic Modeling}},
  year = {2023},
  doi = {10.18653/v1/2023.codi-1.14},
}
Figure