Topic Modeling With Topological Data Analysis

Ciarán Byrne, Danijela Horak, Karo Moilanen, Amandla MabonaEMNLP 2022 · OpenReview

Abstract

Recent unsupervised topic modelling approaches that use clustering techniques on word, token or document embeddings can extract coherent topics. A common limitation of such approaches is that they reveal nothing about inter-topic relationships which are essential in many real-world application domains. We present an unsupervised topic modelling method which harnesses Topological Data Analysis (TDA) to extract a topological skeleton of the manifold upon which contextualised word embeddings lie. We demonstrate that our approach, which performs on par with a recent baseline, is able to construct a network of coherent topics together with meaningful relationships between them.

Introduction

Traditional topic modeling methods, such as Latent Dirichlet Allocation (LDA) or modern clustering-based approaches like BERTopic, often suffer from a "silo" problem. While effective at identifying discrete clusters of words, they treat topics as isolated entities, failing to capture the inherent interconnectedness of human discourse. In reality, semantic themes exist on a continuum; for instance, "space exploration" naturally bridges "physics" and "military technology." By forcing topics into disjoint clusters, the global context and the underlying relationships between themes are lost. This research addresses this gap by proposing a method to map the "manifold" of meaning using Topological Data Analysis (TDA).

Methodology: The Mapper Pipeline

The proposed approach treats topic modeling as a geometric problem, utilizing the Mapper algorithm to construct a topological skeleton of language data.

  1. Embedding and Projection: Contextualized word embeddings are generated using a pre-trained bert-base-uncased model. These high-dimensional embeddings are then projected into a 2D space using UMAP.
  2. The Mapper Algorithm: This core TDA component divides the projected space into overlapping "bins." Within each bin, the original high-dimensional points are clustered using HDBSCAN.
  3. Graph Construction: Nodes in the final graph represent these clusters. An edge is drawn between two nodes if their corresponding clusters share common data points due to bin overlap.
  4. Stability Analysis: Mapper is sensitive to hyperparameters like bin size and overlap. To ensure reliability, the authors introduce two metrics—Matching Distance (for cluster stability) and Spectral Distance (for graph topology stability)—to identify the most stable graph structure.
Stability analysis of the Mapper algorithm across different hyperparameter configurations (number of bins and overlap).
  1. Community Detection: Finally, the Label Propagation algorithm is applied to the graph to group fine-grained topics into broader, high-level themes.

Key Insights and Results

The TDA approach views topics as regions on a continuous manifold rather than discrete buckets. The resulting "topological skeleton" acts as a map, revealing "continents" of related discourse and the "bridges" that connect them.

The topological skeleton of the embedding space, showing interconnected topics and semantic 'continents'.

Performance and Interpretability

On the 20 Newsgroups benchmark, the Mapper + BERT approach achieved a Topic Coherence (NPMI) of 0.059, significantly outperforming baselines like Top2Vec (0.0002) and BERTopic (-0.008). Qualitatively, the model excels at word-sense disambiguation. For example, it can distinguish between different senses of the word "program" (e.g., software vs. radio show) based on their distinct positions and connections within the graph.

Structural Discoveries

The research also found that the topological structure often aligns with linguistic features, such as parts of speech. Clear regions in the graph are dominated by specific categories like nouns or verbs, suggesting the embedding space captures both syntax and semantics. Furthermore, the method allows for hierarchical discovery, enabling researchers to "zoom out" from hundreds of micro-topics to broader themes without losing structural integrity.

Conclusion and Future Work

While the TDA-based approach provides a powerful tool for mapping semantic relationships, it remains sensitive to hyperparameters and the inherent biases of the underlying transformer models. Future work aims to automate the semantic interpretation of edges—identifying whether a connection represents a "part-of" or "is-a" relationship—potentially through integration with formal ontologies.

Summary updated

Abstract

Recent unsupervised topic modelling ap-proaches that use clustering techniques onword, token or document embeddings can ex-tract coherent topics. A common limitationof such approaches is that they reveal noth-ing about inter-topic relationships which areessential in many real-world application do-mains. We present an unsupervised topic mod-elling method which harnesses TopologicalData Analysis (TDA) to extract a topologicalskeleton of the manifold upon which contextu-alised word embeddings lie. We demonstratethat our approach, which performs on par witha recent baseline, is able to construct a networkof coherent topics together with meaningfulrelationships between them.
BibTeX
@inproceedings{byrne2022topic,
  author = {Ciarán Byrne and Danijela Horak and Karo Moilanen and Amandla Mabona},
  title = {{Topic Modeling With Topological Data Analysis}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=sW3U6WnLuG},
}
Figure