Enhancing Neural Topic Model with Multi-Level Supervisions from Seed Words

Yang Lin, Xin Gao, Xu Chu, Yasha Wang, Junfeng Zhao, Chao Chen

Abstract

SeededNTM is a novel neural topic model designed to improve the interpretability and accuracy of discovered themes using "seed words." By balancing word-level and document-level supervisions and employing a consistency regularizer for noise reduction, SeededNTM effectively handles word ambiguity and scales to massive datasets. Experimental results demonstrate that SeededNTM outperforms state-of-the-art baselines in both topic coherence and document classification accuracy.

Introduction

Topic modeling is a fundamental technique for automatically discovering latent themes in large text collections. However, unsupervised models frequently generate "garbage" topics that lack human interpretability or fail to align with specific user interests. To address this, "seeded" topic models use a small set of user-provided words (e.g., "apple, iphone, mac") to nudge the model toward specific topics.

Despite their utility, existing seeded neural topic models (NTMs) face two significant hurdles. First, natural language is inherently ambiguous; a seed word like "apple" may refer to a fruit or a technology company depending on the context. Most NTMs rely on a "mean-field" assumption that treats words and documents independently, struggling to resolve such polysemy. Second, document-level signals derived from seed words are often noisy, and traditional noise-reduction techniques are often too computationally expensive for large-scale applications.

Methods

SeededNTM introduces a context-dependent architecture that integrates multi-level supervisions and a robust noise-reduction mechanism.

Architecture of SeededNTM showing multi-level supervisions and the consistency regularizer.

Multi-Level Supervision and Auto-Adaptation

The model utilizes seed words to generate pseudo-labels at both the word level (local word-topic preference) and the document level (global document-topic preference). Unlike models that assume independence, SeededNTM employs a "product of experts" approach to combine inferred word-topic preferences with document-topic distributions. This allows the model to automatically "de-bias" ambiguous words based on the broader document context.

Consistency Regularizer

To handle noise without the overhead of global similarity searches, SeededNTM uses an intra-sample consistency regularizer. The model applies data augmentation by perturbing documents (e.g., masking low-importance words) and enforces consistency between the predictions for the original and augmented versions. This approach ensures robustness against misleading seed word occurrences while remaining highly scalable.

Key Results

The model was evaluated on three benchmarks: 20 Newsgroups (small), Yahoo Answers (medium), and DBpedia (large).

Performance comparison of SeededNTM against baselines on three benchmark datasets.

Topic Coherence and Classification

SeededNTM consistently achieved the highest Normalized Pointwise Mutual Information (NPMI) scores, indicating superior semantic coherence. For instance, on the 20 Newsgroups test set, it reached an NPMI of 0.335, outperforming the next best seeded model. Furthermore, the model excelled in document classification, reaching a 0.905 F1 score on the DBpedia dataset, significantly higher than the 0.865 achieved by the runner-up.

Efficiency

Beyond accuracy, SeededNTM is remarkably efficient. On the Yahoo Answers dataset, it completed training in just 12 minutes, whereas the STM baseline took 97 minutes. This efficiency stems from the intra-sample consistency regularizer, which avoids expensive nearest-neighbor computations across the entire dataset.

Conclusion

SeededNTM represents a significant advancement in guided topic modeling. By moving beyond the mean-field assumption and incorporating efficient noise reduction, it produces topics that are both semantically rich and highly accurate for classification tasks. Future work may involve integrating pre-trained language models like BERT for deeper semantic understanding and exploring multi-label seed extraction methods.

Summary updated

BibTeX
@inproceedings{lin2023enhancing,
  author = {Yang Lin and Xin Gao and Xu Chu and Yasha Wang and Junfeng Zhao and Chao Chen},
  title = {{Enhancing Neural Topic Model with Multi-Level Supervisions from Seed Words}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.845},
}
Figure