Abstract
Recent unsupervised topic modelling approaches that use clustering techniques on word, token or document embeddings can extract coherent topics. A common limitation of such approaches is that they reveal nothing about inter-topic relationships which are essential in many real-world application domains. We present an unsupervised topic modelling method which harnesses Topological Data Analysis (TDA) to extract a topological skeleton of the manifold upon which contextualised word embeddings lie. We demonstrate that our approach, which performs on par with a recent baseline, is able to construct a network of coherent topics together with meaningful relationships between them.
Introduction
Traditional topic modeling methods, such as Latent Dirichlet Allocation (LDA) or modern clustering-based approaches like BERTopic, often suffer from a "silo" problem. While effective at identifying discrete clusters of words, they treat topics as isolated entities, failing to capture the inherent interconnectedness of human discourse. In reality, semantic themes exist on a continuum; for instance, "space exploration" naturally bridges "physics" and "military technology." By forcing topics into disjoint clusters, the global context and the underlying relationships between themes are lost. This research addresses this gap by proposing a method to map the "manifold" of meaning using Topological Data Analysis (TDA).
Methodology: The Mapper Pipeline
The proposed approach treats topic modeling as a geometric problem, utilizing the Mapper algorithm to construct a topological skeleton of language data.
- Embedding and Projection: Contextualized word embeddings are generated using a pre-trained
bert-base-uncasedmodel. These high-dimensional embeddings are then projected into a 2D space using UMAP. - The Mapper Algorithm: This core TDA component divides the projected space into overlapping "bins." Within each bin, the original high-dimensional points are clustered using HDBSCAN.
- Graph Construction: Nodes in the final graph represent these clusters. An edge is drawn between two nodes if their corresponding clusters share common data points due to bin overlap.
- Stability Analysis: Mapper is sensitive to hyperparameters like bin size and overlap. To ensure reliability, the authors introduce two metrics—Matching Distance (for cluster stability) and Spectral Distance (for graph topology stability)—to identify the most stable graph structure.
- Community Detection: Finally, the Label Propagation algorithm is applied to the graph to group fine-grained topics into broader, high-level themes.
Key Insights and Results
The TDA approach views topics as regions on a continuous manifold rather than discrete buckets. The resulting "topological skeleton" acts as a map, revealing "continents" of related discourse and the "bridges" that connect them.
Performance and Interpretability
On the 20 Newsgroups benchmark, the Mapper + BERT approach achieved a Topic Coherence (NPMI) of 0.059, significantly outperforming baselines like Top2Vec (0.0002) and BERTopic (-0.008). Qualitatively, the model excels at word-sense disambiguation. For example, it can distinguish between different senses of the word "program" (e.g., software vs. radio show) based on their distinct positions and connections within the graph.
Structural Discoveries
The research also found that the topological structure often aligns with linguistic features, such as parts of speech. Clear regions in the graph are dominated by specific categories like nouns or verbs, suggesting the embedding space captures both syntax and semantics. Furthermore, the method allows for hierarchical discovery, enabling researchers to "zoom out" from hundreds of micro-topics to broader themes without losing structural integrity.
Conclusion and Future Work
While the TDA-based approach provides a powerful tool for mapping semantic relationships, it remains sensitive to hyperparameters and the inherent biases of the underlying transformer models. Future work aims to automate the semantic interpretation of edges—identifying whether a connection represents a "part-of" or "is-a" relationship—potentially through integration with formal ontologies.
Summary updated