ProSiT! Latent Variable Discovery with PROgressive SImilarity Thresholds

Tommaso Fornaciari, Dirk Hovy, Federico BianchiarXiv 2022 · DOI · arXiv

Introduction

Traditional topic modeling techniques, such as Latent Dirichlet Allocation (LDA) and K-Means clustering, have long been the standard for discovering latent structures in document collections. However, these methods suffer from two significant drawbacks: stochasticity and the requirement to pre-specify the number of topics ($k$). Stochasticity leads to inconsistent results across different runs, while the "K problem" forces researchers to guess the underlying structure of their data. ProSiT (PROgressive SImilarity Thresholds) addresses these issues by providing a deterministic, data-driven approach to latent variable discovery that eliminates the need for random initialization.

Methodology

ProSiT employs a two-step process that integrates hierarchical clustering principles with topic modeling to achieve reproducible results.

Step 1: Topic Discovery via Progressive Similarity Thresholds

The algorithm initializes by treating each document as an individual potential topic. It then iteratively merges these points based on their cosine similarity. To prevent the "mean of means" problem—where clusters prematurely collapse into a single global center—ProSiT introduces a Progressive Similarity Threshold (CST). This threshold follows a hyperbolic curve, governed by a parameter $\alpha$, which increases the similarity requirement as the iterations progress.

Progressive Similarity Threshold (CST) curves for different alpha values over training epochs.

This dynamic thresholding allows for "fuzzy" membership early in the process, where documents can contribute to multiple centroids, before enforcing stricter boundaries as the model converges on stable topic centers. This mechanism ensures that distinct topic clusters emerge naturally from the data.

Step 2: Descriptor Selection

After identifying topic centroids, ProSiT describes them using a novel multi-class application of Information Gain (IG). By identifying words that are most informative for a specific topic relative to all others, the model ensures that the resulting descriptors are highly distinctive and interpretable, avoiding the generic word lists often produced by traditional models.

Experimental Setup

The performance of ProSiT was evaluated across four diverse benchmark datasets: Reuters and Google News (long-form documents), and Wiki20K and Tweets2011 (short-form text). These datasets were chosen to test the model's robustness across different document lengths and vocabularies. ProSiT was compared against a heavy-hitting lineup of baselines, including LDA, Contextualized Topic Models (CTM), and neural models like N-PLDA.

Key Results

ProSiT demonstrated superior performance in terms of topic coherence and distinctiveness. In datasets like Reuters and Wiki20K, ProSiT's coherence scores were competitive with or exceeded those of state-of-the-art neural models.

Comparison of coherence scores across different models and topic counts.

Beyond coherence, ProSiT consistently produced more distinct topics, as measured by Topic Dissimilarity and Specificity. This indicates that the discovered topics are well-separated and non-redundant. Furthermore, the model is highly efficient; training on the 20,000-document Wiki20K dataset took approximately three minutes on a standard consumer GPU (GTX 1080 Ti), making it practical for large-scale applications.

Conclusion

ProSiT offers a robust alternative to traditional topic models by providing deterministic and reproducible results. By eliminating random initialization and the need to guess the number of topics, it serves as a reliable tool for sensitive applications in social science and legal analysis where transparency is paramount. Future work will focus on direct comparisons with popular frameworks like BERTopic and further optimizing hyperparameter selection through automated grid searches.

Summary updated

Abstract

The most common ways to explore latent document dimensions are topic models and clustering methods. However, topic models have several drawbacks: e.g., they require us to choose the number of latent dimensions a priori, and the results are stochastic. Most clustering methods have the same issues and lack flexibility in various ways, such as not accounting for the influence of different topics on single documents, forcing word-descriptors to belong to a single topic (hard-clustering) or necessarily relying on word representations. We propose PROgressive SImilarity Thresholds - ProSiT, a deterministic and interpretable method, agnostic to the input format, that finds the optimal number of latent dimensions and only has two hyper-parameters, which can be set efficiently via grid search. We compare this method with a wide range of topic models and clustering methods on four benchmark data sets. In most setting, ProSiT matches or outperforms the other methods in terms six metrics of topic coherence and distinctiveness, producing replicable, deterministic results.
BibTeX
@misc{fornaciari2022prosit,
  author = {Tommaso Fornaciari and Dirk Hovy and Federico Bianchi},
  title = {{ProSiT! Latent Variable Discovery with PROgressive SImilarity Thresholds}},
  year = {2022},
  eprint = {2210.14763},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.14763},
}
Figure