Abstract
SeededNTM is a novel neural topic model designed to improve the interpretability and accuracy of discovered themes using "seed words." By balancing word-level and document-level supervisions and employing a consistency regularizer for noise reduction, SeededNTM effectively handles word ambiguity and scales to massive datasets. Experimental results demonstrate that SeededNTM outperforms state-of-the-art baselines in both topic coherence and document classification accuracy.
Introduction
Topic modeling is a fundamental technique for automatically discovering latent themes in large text collections. However, unsupervised models frequently generate "garbage" topics that lack human interpretability or fail to align with specific user interests. To address this, "seeded" topic models use a small set of user-provided words (e.g., "apple, iphone, mac") to nudge the model toward specific topics.
Despite their utility, existing seeded neural topic models (NTMs) face two significant hurdles. First, natural language is inherently ambiguous; a seed word like "apple" may refer to a fruit or a technology company depending on the context. Most NTMs rely on a "mean-field" assumption that treats words and documents independently, struggling to resolve such polysemy. Second, document-level signals derived from seed words are often noisy, and traditional noise-reduction techniques are often too computationally expensive for large-scale applications.
Methods
SeededNTM introduces a context-dependent architecture that integrates multi-level supervisions and a robust noise-reduction mechanism.
Multi-Level Supervision and Auto-Adaptation
The model utilizes seed words to generate pseudo-labels at both the word level (local word-topic preference) and the document level (global document-topic preference). Unlike models that assume independence, SeededNTM employs a "product of experts" approach to combine inferred word-topic preferences with document-topic distributions. This allows the model to automatically "de-bias" ambiguous words based on the broader document context.
Consistency Regularizer
To handle noise without the overhead of global similarity searches, SeededNTM uses an intra-sample consistency regularizer. The model applies data augmentation by perturbing documents (e.g., masking low-importance words) and enforces consistency between the predictions for the original and augmented versions. This approach ensures robustness against misleading seed word occurrences while remaining highly scalable.
Key Results
The model was evaluated on three benchmarks: 20 Newsgroups (small), Yahoo Answers (medium), and DBpedia (large).
Topic Coherence and Classification
SeededNTM consistently achieved the highest Normalized Pointwise Mutual Information (NPMI) scores, indicating superior semantic coherence. For instance, on the 20 Newsgroups test set, it reached an NPMI of 0.335, outperforming the next best seeded model. Furthermore, the model excelled in document classification, reaching a 0.905 F1 score on the DBpedia dataset, significantly higher than the 0.865 achieved by the runner-up.
Efficiency
Beyond accuracy, SeededNTM is remarkably efficient. On the Yahoo Answers dataset, it completed training in just 12 minutes, whereas the STM baseline took 97 minutes. This efficiency stems from the intra-sample consistency regularizer, which avoids expensive nearest-neighbor computations across the entire dataset.
Conclusion
SeededNTM represents a significant advancement in guided topic modeling. By moving beyond the mean-field assumption and incorporating efficient noise reduction, it produces topics that are both semantically rich and highly accurate for classification tasks. Future work may involve integrating pre-trained language models like BERT for deeper semantic understanding and exploring multi-label seed extraction methods.
Summary updated