GINopic: Topic Modeling with Graph Isomorphism Network

Suman Adhya, Debarshi Kumar SanyalNAACL 2024 · DOI

TL;DR

This paper introduces GINopic, a novel topic modeling framework that leverages Graph Isomorphism Networks (GINs) to better capture complex word dependencies within documents. By representing documents as graphs and using GINs for representation learning, GINopic achieves superior topic coherence and diversity compared to existing methods, leading to more meaningful and accurate topic discovery.

What Problem Does This Paper Address?

Organizing and understanding the vast amount of text data generated daily is a significant challenge. Topic modeling aims to uncover the underlying themes within documents, but traditional and even recent neural approaches often overlook the intricate relationships between words. While some methods attempt to incorporate word co-occurrences or dependencies, they can be computationally expensive, limited in scope (e.g., only considering words within a small window), or fail to capture the full complexity of these relationships. This paper addresses the need for a topic modeling approach that can effectively model these complex, inter-word dependencies to improve the quality and interpretability of discovered topics.

Methods

GINopic's core innovation lies in its use of Graph Isomorphism Networks (GINs) for topic modeling. The process involves:

  1. Graph Construction: Each document is transformed into a weighted, undirected graph. Nodes represent words, and edges are weighted based on the cosine similarity of their word embeddings. A threshold parameter ($\delta$) is used to prune weak connections, balancing graph density and computational complexity.
  2. Graph Representation Learning: A GIN is employed to learn a unique representation for each document graph. GINs are chosen for their proven ability to capture graph isomorphism and their theoretical power in distinguishing between different graph structures, effectively encoding the topological information of the word dependency graph.
  3. VAE Framework: The learned graph representation is concatenated with a traditional TF-IDF representation of the document. This combined representation is then fed into an encoder-decoder architecture based on a Variational Autoencoder (VAE). The VAE learns a latent topic distribution for each document, allowing for topic generation and reconstruction.

The use of GINs is novel in the context of topic modeling, offering a more powerful way to process graph-structured data compared to standard graph convolutional networks.

Datasets

The paper evaluates GINopic on five publicly available datasets:

  • 20NewsGroups (20NG): A collection of 16,309 pre-processed documents from 20 different newsgroups.
  • BBC News (BBC): 2,225 news articles categorized into five classes: tech, business, entertainment, sports, and politics.
  • SearchSnippets (SS): Derived from web search transactions across eight domains.
  • Biomedicine (Bio): Data from the BioASQ challenge.
  • StackOverflow (SO): A large dataset of question titles from Kaggle.com.

These datasets are standard benchmarks for topic modeling research.

Key Insights and Ideas

  • GINs Excel at Capturing Word Dependencies: The study demonstrates that GINs are highly effective at learning representations from document graphs, outperforming other GNN architectures (GAT, GraphSAGE, GCN) in topic coherence. This highlights the power of GINs in understanding the structural relationships between words.
  • Graph Construction Threshold is Crucial: The choice of the graph construction threshold ($\delta$) significantly impacts both model performance and training time. Tuning this parameter allows for a trade-off between achieving higher coherence scores and reducing computational overhead.
  • GINopic Generates More Coherent Topics: Both quantitative metrics (NPMI, CV) and qualitative analysis show that GINopic produces topics with more strongly correlated words, indicating a deeper understanding of semantic relationships within documents.
  • Latent Space Disentanglement: Visualizations of the latent space show that GINopic can effectively disentangle document clusters, with clearer separation observed on datasets with fewer distinct labels.

Key Results

  • Quantitative Evaluation: GINopic consistently achieved the highest topic coherence scores (NPMI and CV) across most of the evaluated datasets, often outperforming strong baselines like CombinedTM and ZeroShotTM. For the SO dataset, it ranked second in NPMI but led in CV.
  • Diversity Metrics: GINopic also demonstrated strong performance in topic diversity, achieving the highest wI-M and wI-C scores on most datasets.
  • Extrinsic Evaluation: In document classification tasks, GINopic generally achieved higher accuracy than other models across datasets, except for 20NG where it was closely edged out by GNTM.
  • Qualitative Evaluation: Manual inspection of topics revealed that GINopic consistently generated more correlated words within topics, as evidenced by a higher number of bolded terms in the representative topics.
  • Sensitivity Analysis: The study showed that GIN outperformed other GNNs, and optimal threshold values for graph construction were identified for each dataset, leading to improved coherence and reduced training time.

Limitations and Future Work

  • Graph Construction: The current graph construction relies solely on word similarity. Future work could explore incorporating other types of word dependencies, such as syntactic or semantic relationships, to create more multifaceted document graphs.
  • Computational Cost: While GINs offer advantages, the computational cost of graph construction can still be a factor, especially for very large vocabularies.
  • Extrinsic Evaluation: While GINopic performed well in document classification, further exploration of its performance on other downstream tasks could be beneficial.

The authors suggest exploring alternative graph construction methods and integrating diverse word dependency information as promising future directions.

Summary updated

BibTeX
@inproceedings{adhya2024ginopic,
  author = {Suman Adhya and Debarshi Kumar Sanyal},
  title = {{GINopic: Topic Modeling with Graph Isomorphism Network}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.342},
}
Figure