TL;DR
This paper introduces a novel neural topic modeling approach called Topics as Entity Clusters (TEC) that leverages both large language models (LLMs) and graph neural networks (GNNs) to represent entities. TEC demonstrates superior topic coherence and quality compared to existing methods, particularly by effectively integrating explicit knowledge from knowledge graphs.
What Problem Does This Paper Address?
Traditional topic modeling, like Latent Dirichlet Allocation (LDA), relies on word frequencies, which can be noisy and lead to topics that are difficult to interpret, especially in multilingual settings. While recent advancements in neural topic modeling have incorporated word embeddings, they still face challenges with data sparsity and interpretability. The authors identify a gap in purely entity-driven neural topic modeling, questioning the suitability of existing techniques for handling sparse, information-dense conceptual units like entities. They aim to develop a more robust and interpretable topic modeling approach by focusing on entities as the primary features.
Methods
The core of the proposed method, TEC, is to represent entities using a combination of implicit knowledge from LLMs and explicit knowledge from GNNs trained on a knowledge base.
- Entity Representation: Entities are represented by concatenating two types of embeddings:
- LLM Embeddings: Contextualized text representations derived from entity descriptions using a pre-trained LLM (specifically, SBERT).
- GNN Embeddings: Embeddings learned from a knowledge graph (Wikidata) using a GNN (specifically, node2vec).
- These embeddings are combined using a weighted concatenation function, allowing for adjustable contributions from each source.
- Entity Clustering: The combined entity embeddings are then clustered using K-Means to form topic centroids. This approach models topics based on the inherent structure of shared embedding spaces among entities.
- Topic Inference: Documents are represented by a weighted average of their constituent entity embeddings, weighted by term frequency. The distance of a document's embedding to topic centroids is then used to infer topic distributions.
- Reranking Top Entities: A novel inference-based method is proposed to rerank top entities within a topic. This method assigns entity frequencies from documents to their most probable topic and updates the top entities based on this, aiming to improve topic diversity and relevance.
Datasets
The authors evaluate their approach on three corpora:
- Wikipedia: A sample of preprocessed documents, each matching an entity in the vocabulary.
- CC-News: Monolingual news articles written in English.
- MLSUM: A collection of news articles in multiple languages (German, Spanish, French, Russian, and Turkish).
Wikidata was used as the knowledge base for graph embeddings, and the English Wikipedia was used as the encyclopedic corpus for entity descriptions.
Key Insights and Ideas
- Entities as Superior Topic Features: The research strongly suggests that entities, particularly when enriched with explicit knowledge from knowledge graphs, are more effective features for topic modeling than traditional word tokens, especially in sparse data settings.
- The Power of Explicit Knowledge: Combining implicit knowledge from LLMs with explicit knowledge from GNNs trained on knowledge graphs significantly improves topic coherence and quality. The explicit knowledge from the graph appears to be the more dominant factor in achieving better results.
- Language Agnosticism: Using entities as language-agnostic identifiers offers a significant advantage for multilingual topic modeling, overcoming the challenges of language bifurcation that plague word-based approaches.
- Limitations of Standard Metrics: The paper acknowledges that standard topic coherence metrics (like NPMI) can be misleading when comparing word-based and entity-based models due to differences in data sparsity and how these metrics are affected by co-occurrence statistics.
Key Results
- Quantitative Performance: TEC consistently outperformed LDA and other neural topic models (NVDM-GSM, ProdLDA, CombinedTM, WikiPDA) across various corpora and metrics (Topic Coherence - TC, Topic Quality - TQ), except for Topic Diversity (TD) where LDA sometimes performed better. For instance, on the WIKIPEDIA corpus with 300 topics, TEC achieved a TC of 0.31 (±0.01) and TQ of 0.30 (±0.01), significantly higher than other models.
- Impact of Knowledge Sources: Experiments showed that increasing the weight of graph embeddings (higher α values) consistently improved TEC's performance, indicating the crucial role of explicit knowledge. Purely LLM-based TEC (α=0) performed worse than models leveraging graph embeddings.
- Qualitative Results: Visual inspection of topics generated by TEC revealed highly interpretable and coherent topics, especially when compared to LDA trained on entities, which produced unclear topics with noisy terms.
Limitations and Future Work
- Entity Assumption: TEC assumes that documents contain entities, which might not always be true. The authors suggest exploring self-supervised models to generate word embeddings that can be integrated into the shared embedding space for entity-rich applications like news articles.
- GNN Depth: While node2vec was effective, the authors believe that using deeper, multilayer GNNs could potentially learn more complex relationships between entities, further improving results.
- Metric Alignment: The paper highlights the ongoing issue that automatic coherency metrics may not perfectly align with human judgment of topic quality. They suggest future work involving human evaluation methods like intrusion tests.
- Knowledge Base Updates: Updating the knowledge base requires retraining the model, and the current approach doesn't guarantee a direct relationship between old and new topics, which could be a hindrance for some applications.
Summary updated