TL;DR
This paper reframes topic modeling as a neighborhood aggregation problem on a word network, enabling a simpler integration of supervisory signals and pre-trained word embeddings. Their proposed model, NA-WE-SLDA, outperforms existing supervised topic models on document classification tasks.
What Problem Does This Paper Address?
Topic models, like Latent Dirichlet Allocation (LDA), are powerful tools for uncovering hidden themes in text, offering interpretability and modularity. However, extending these models to incorporate additional information, such as labels (supervisory signals) or pre-trained word embeddings, and to use nonlinear output functions, typically requires complex approximate inference procedures. This complexity hinders practitioners from easily adapting topic models to various real-world scenarios. The paper aims to simplify this process by viewing topic modeling through the lens of neighborhood aggregation, a technique commonly used in graph-based representation learning.
Methods
The core innovation lies in reinterpreting topic modeling, specifically LDA, as a neighborhood aggregation algorithm. Instead of traditional message passing, the authors view words in a document as nodes in a network. Relationships between words (co-occurrence within a document, or being the same word across documents) form edges.
- Network View of LDA: LDA is reframed as a factor graph where messages are passed between nodes (words). This perspective reveals that the message update in LDA can be seen as a product of two neighborhood aggregation operations.
- Supervised Extension: Supervisory signals (e.g., document labels) are incorporated by adding a new "neighborhood system" to the network. This system connects document-word pairs that share the same label. The influence of this supervisory signal is controlled by a learned parameter.
- Pre-trained Word Embeddings: Pre-trained word embeddings are treated as node attributes. These attributes are combined with the aggregated neighborhood information through a learned transformation and a nonlinear function.
- Unified Model (NA-WE-SLDA): The authors propose a unified neighborhood aggregation framework that combines the unsupervised topic modeling aspect (NAa, NAw), the supervised signal (NAs), and pre-trained word embeddings (ug). The message update equation becomes a weighted combination of these components, followed by normalization. This approach allows for a differentiable neural network to learn the parameters governing message passing and the influence of embeddings.
Datasets
The authors evaluate their approach on three datasets:
- Economic Watcher Survey (EWS): A multi-class dataset from Japan focusing on economic assessments, with labels ranging from "economy will get worse" to "economy will get better."
- Amazon Review Data: A dataset of Amazon reviews with ratings.
- Subjectivity Data: A dataset from Rotten Tomatoes and IMDb, classifying snippets as subjective or objective.
They use subsets of these datasets, filtering for documents with a minimum number of words and restricting vocabulary to the 3000 most frequent words.
Key Insights and Ideas
- Topic Modeling as Graph Learning: The most significant insight is the elegant reframing of topic modeling as a neighborhood aggregation problem. This perspective bridges the gap between traditional probabilistic models and modern graph representation learning techniques.
- Simplified Integration: By viewing topic models as networks, incorporating supervisory signals and pre-trained word embeddings becomes a more straightforward process of adding new edges or node attributes, rather than requiring complex inference modifications.
- Adaptive Message Passing: The proposed model learns parameters that govern how messages are passed and how different information sources (unsupervised topics, supervised labels, word embeddings) are combined, leading to a more adaptive and potentially more accurate model.
- Nonlinear Output: The ability to easily incorporate nonlinear output functions is crucial for tasks like classification, and the neighborhood aggregation framework facilitates this.
Key Results
- Classification Performance: The proposed model, NA-WE-SLDA, consistently outperforms state-of-the-art supervised LDA implementations (SLDA, MedLDA) and other topic models incorporating word embeddings (LFLDA, LFDMM) on held-out document classification tasks, particularly in terms of accuracy. For example, on the EWS dataset, NA-WE-SLDA achieves an accuracy of 0.504, surpassing SLDA (0.336) and LFLDA (0.468).
- Topic Coherence: While achieving better predictive performance, the model shows slightly inferior topic coherence compared to some other methods, suggesting a potential trade-off between interpretability and predictive power. However, the paper notes that its coherence is comparable to other models that also incorporate complex structures.
Limitations and Future Work
- Topic Coherence Trade-off: The authors acknowledge that their model exhibits slightly lower topic coherence compared to some baseline methods. They suggest that sparse message passing could potentially improve coherence.
- Future Work: The paper mentions that sparse message passing could be explored further. Additionally, the framework's flexibility suggests potential applications beyond classification, such as regression tasks. The authors also note that for the economic watcher survey, they translated Japanese words to English, which might introduce some limitations.
Summary updated