Neural Topic Model with Attention for Supervised Learning

Xinyi Wang, Yi YangAISTATS 2020 · OpenReview

Abstract

Topic modeling utilizing neural variational inference has shown promising results recently. Unlike traditional Bayesian topic models, neural topic models use deep neural networks to approximate the posterior distribution of latent topics. This paper introduces the Topic Attention Model (TAM), a unified neural framework that bridges the gap between unsupervised topic discovery and supervised text classification. By integrating a Neural Topic Model (NTM) with a Recurrent Neural Network (RNN) through a multi-query attention mechanism, TAM uses global topic vectors to guide the RNN’s focus, resulting in state-of-the-art prediction accuracy and more coherent document representations.

Introduction

Traditional topic models, such as Latent Dirichlet Allocation (LDA), are primarily unsupervised; they identify patterns in text without explicit knowledge of document labels. When these unsupervised topics are utilized for downstream tasks like sentiment analysis or categorization, the results are often sub-optimal because the latent topics were not informed by the labels during training.

While supervised variants of LDA exist, they often struggle with the complex, non-linear nature of modern text data. Conversely, deep learning models like Recurrent Neural Networks (RNNs) excel at sequence prediction but frequently ignore the global thematic structure of a corpus. TAM addresses this by combining the strengths of both: the global thematic awareness of topic models and the local sequence modeling of RNNs.

The TAM Architecture

TAM combines three major components into a single, jointly optimized architecture:

  1. Neural Topic Model (GSM): Based on the Variational Autoencoder (VAE) framework, this component learns document-specific topic proportions ($t$) and global topic vectors ($v$).
  2. Sequence Encoder (Bi-GRU): A bidirectional Gated Recurrent Unit processes the document word-by-word to capture local context and sequence information.
  3. Topical Attention Mechanism: This is the core innovation. Instead of using a single, generic query vector for attention, TAM uses the global topic vectors learned by the topic model as multiple queries. For every word, the model calculates how well it fits into each of the $K$ topics, weighted by the document's specific topic distribution.
The joint architecture of the Topic Attention Model (TAM), showing the integration of the topic model (left) and the RNN (right).

To ensure effective training, the authors introduce a "shifted topic mixture" ($\delta$). This prevents the model from collapsing into a single-topic focus by allowing the gradient descent process to distinguish effectively between dominant and minor topics. The entire model is trained by maximizing a joint objective: the Evidence Lower Bound (ELBO) for the topic model plus the likelihood of the supervised labels.

Experimental Results

The authors evaluated TAM across three distinct supervised tasks using the 20 Newsgroups (multi-class classification), Movie Review Dataset (regression), and Wiki10 (multi-label classification) datasets.

TAM consistently outperformed both traditional supervised topic models (like sLDA and MedLDA) and standard neural baselines. On the 20 Newsgroups dataset, TAM achieved approximately 76% accuracy, significantly outperforming sLDA (~69%) and the unsupervised GSM (~35%). In regression tasks on the Movie Review Dataset, TAM reached a predictive $R^2$ of 0.602, a massive improvement over the unsupervised neural topic model's 0.144. For multi-label classification on Wiki10, TAM achieved an $F_1$ score of 0.519, surpassing the vanilla Attention-RNN (0.491).

Interpretability and Attention Visualization

One of the primary advantages of TAM is its interpretability. Because the attention mechanism is tied to specific global topics, it is possible to visualize exactly which words the model prioritizes for specific themes within a single sentence.

Visualization of topical attention, highlighting how the model focuses on specific words like 'pharmacology' or 'drinking' based on thematic relevance.

This visualization provides a layer of transparency often missing in deep learning models. For example, the model can highlight "Medical" words versus "Political" words in the same document, showing how the global topic vectors guide the local sequence encoder.

Discussion and Insights

The research yielded several key insights:

  • Supervision Improves Modeling: Surprisingly, adding supervised labels helped the model achieve lower perplexity compared to unsupervised models. This suggests that label information helps the model discover more accurate latent structures.
  • Global vs. Local Synergy: RNNs are excellent at local context but can lose the "big picture." By using global topic vectors as attention queries, TAM forces the RNN to pay attention to words that are transitionally important to the specific themes of the document.

Conclusion

TAM represents a significant step forward in integrating structural thematic modeling with deep sequence learning. While the model currently faces challenges regarding vocabulary complexity and hyperparameter sensitivity (specifically the "shift" parameter), its ability to provide both high predictive performance and thematic interpretability makes it a powerful tool for supervised text analysis. Future work may explore why neural topic models sometimes underperform traditional LDA in specific supervised settings despite their superior document modeling capabilities.

Summary updated

Abstract

Topic modeling utilizing neural variational inference has shown promising results recently. Unlike traditional Bayesian topic models, neural topic models use deep neural network to approximate the ...
BibTeX
@inproceedings{wang2020neural,
  author = {Xinyi Wang and Yi Yang},
  title = {{Neural Topic Model with Attention for Supervised Learning}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2020},
  url = {https://openreview.net/forum?id=vDXeqW57ED},
}
Figure