For decades, discovering the underlying themes in large collections of text relied on counting words. Traditional probabilistic techniques, most notably Latent Dirichlet Allocation (LDA), treat documents as a "bag of words." As demonstrated by Jana Diesner and surveyed by Ehsan-Ul Haq et al., these classical methods assume that documents are mixtures of latent topics, where each topic is a statistical distribution over a fixed vocabulary. If words like "interest," "rate," and "inflation" frequently appear together in the same documents, a probabilistic model groups them into a thematic category.
However, the bag-of-words paradigm suffers from fundamental limitations: it ignores word order, strips away context, and struggles with polysemy (the same word having multiple meanings). Furthermore, traditional models break down when applied to short, informal texts where word co-occurrence matrices become sparse and uninformative.
To overcome these challenges, a powerful paradigm has emerged: embedding-based topic modeling. Instead of counting words globally, this approach maps text into a continuous, high-dimensional vector space where geometric proximity reflects semantic similarity. Topics are no longer abstract probability distributions; they are dense clusters within a geometric space.
The Geometric Intuition
To understand why embedding-based topic modeling works, imagine a massive multi-dimensional map where every sentence or document is assigned a specific coordinate.
In a traditional keyword-based system, the sentences "The financial institution closed its doors" and "The bank by the river was muddy" might be grouped together because they share the word "bank." Conversely, "The market crashed" and "Stocks plummeted" share no words and would be treated as entirely unrelated.
Pre-trained language models change this dynamic. By processing text through deep neural architectures, these models generate dense vector representations (embeddings) that capture contextual meaning. Early foundations for this shift utilized paragraph vectors, as explored by Wengong Jin, or clustered high-dimensional word vectors to track evolving debates, as seen in the work on Finding Social Media Trolls: Dynamic Keyword Selection Methods for Rapidly-Evolving Online Debates.
In modern pipelines, when text is converted into these dense vectors, sentences with similar meanings are placed close to one another, regardless of whether they share exact keywords. Topic discovery then transforms from a complex statistical estimation problem into a geometric search for dense regions in this embedding space.
The Modern Modular Pipeline
The most prominent realization of this paradigm is the modular pipeline popularized by frameworks like BERTopic, which has been widely adopted to analyze complex text corpora ranging from scientific literature (Daniele Atzeni et al.) to clinical datasets (Martin Orr et al.). The process operates in four distinct phases:
[Raw Text]
│
▼
[1. Document Embeddings] <-- Contextual language models (e.g., BERT)
│
▼
[2. Dimensionality Reduction] <-- UMAP (Preserves local & global structure)
│
▼
[3. Density-Based Clustering] <-- HDBSCAN (Identifies dense clusters & filters noise)
│
▼
[4. Class-based TF-IDF] <-- Extracts topic-representative keywords
1. Document Embedding
First, documents are converted into high-dimensional vectors using a transformer-based language model. As Salim Salmi et al. demonstrate, this ensures that short, informal text snippets are captured via semantic context rather than raw keyword frequency.
2. Dimensionality Reduction
Raw transformer embeddings typically span hundreds of dimensions (e.g., 768 dimensions for standard BERT models). In such high-dimensional spaces, distance metrics lose their discriminatory power because vectors appear almost equidistant from one another—a phenomenon known as the curse of dimensionality.
To fix this, pipelines employ algorithms like Uniform Manifold Approximation and Projection (UMAP). UMAP compresses the vectors into a much lower-dimensional space (often 5 to 10 dimensions) while explicitly preserving both the local neighbor relationships and the broader global structure of the data.
3. Density-Based Clustering
Once the space is compressed, a clustering algorithm identifies the topics. While some implementations use centroid-based algorithms like K-means (Chuanlin Ning et al.), density-based algorithms like HDBSCAN or DBSCAN are generally preferred.
As Trenton Ford and Utkucan Balci show, density-based clustering does not force every single document into a category. Instead, it looks for tightly packed "mountains" of semantic similarity to define topics, while safely categorizing sparse, unrelated text as "noise." This is vital for real-world datasets where many documents may be off-topic or irrelevant.
4. Topic Representation via c-TF-IDF
Clustering tells us which documents belong together, but it does not inherently explain why. To make the clusters human-interpretable, the pipeline applies a modified version of a classic NLP metric: Class-based Term Frequency-Inverse Document Frequency (c-TF-IDF).
Traditional TF-IDF measures how important a word is to a single document relative to a collection of documents. In contrast, c-TF-IDF treats all the documents within a single cluster as if they were one giant mega-document. As detailed by Malvina Bozhidarova et al., c-TF-IDF calculates the frequency of a word within a specific cluster divided by the frequency of that word across the entire corpus. This surfaces the most distinctive, representative keywords for each geometric cluster, yielding highly interpretable topic descriptions.
Why the Embedding Approach Excels
The shift from probabilistic modeling to embedding clustering succeeds because it decouples semantic understanding from topic extraction.
In traditional models like LDA, the algorithm must simultaneously learn the meaning of words and the distribution of topics. Embedding-based models bypass this by leveraging pre-trained language models that already understand syntax, grammar, and semantic relationships from billions of words of training data.
This separation of concerns makes embedding-based topic modeling uniquely suited for tracking highly dynamic, unstructured, and short-form text. For instance, Felix Drinkall utilized unsupervised embedding clusters of social media posts to forecast localized public health trends, capturing shifts in public discourse that static keyword lexicons missed entirely. Similarly, Yukun Jiang et al. successfully applied transformer-based topic modeling to separate core content from chaotic, high-volume livestream chat streams.
While alternative deep-learning frameworks exist—such as Neural Topic Models (NTMs) that use variational autoencoders to reconstruct document profiles (Xiaobao Wu et al.)—the modular embedding-clustering pipeline remains highly popular. By translating the abstract problem of "thematic meaning" into the concrete mathematics of spatial density, it provides an intuitive, robust, and highly scalable window into massive text datasets.