Abstract
Short text topic modeling faces significant challenges due to data sparsity, where limited word co-occurrences hinder traditional methods like Latent Dirichlet Allocation (LDA). This paper presents the Topic-Semantic Contrastive Topic Model (TSCTM), a framework that leverages contrastive learning to enhance topic discovery in short texts. By employing vector quantization to identify semantically similar and dissimilar pairs, TSCTM learns robust topic distributions. Experimental results across multiple benchmark datasets demonstrate that TSCTM significantly outperforms state-of-the-art baselines in topic quality, clustering, and classification tasks.
1. Introduction
Traditional topic models rely heavily on the "bag-of-words" assumption, identifying topics through frequent word co-occurrences within documents. While effective for long-form content, this approach fails for short texts such as tweets or news headlines, which often contain fewer than ten words. This phenomenon, known as data sparsity, results in insufficient information for models to capture meaningful patterns. Existing solutions often use data augmentation or document clustering but fail to fully exploit relational information between samples, often leading to inconsistent topic assignments for semantically similar documents.
2. Methodology: TSCTM Framework
The TSCTM framework is built upon Neural Topic Models, specifically Variational Autoencoders (VAEs). The core innovation is the Topic-Semantic Contrastive (TSC) objective, which regularizes the latent space by pulling semantically related representations together and pushing unrelated ones apart.
As illustrated in Figure 1, the process involves encoding text into a latent representation ($h$) and a corresponding topic distribution ($\theta$). To determine positive and negative pairs without human labels, the model utilizes Vector Quantization (VQ). VQ maps continuous topic distributions to discrete "topic embeddings." Two texts are considered a positive pair if they map to the same discrete embedding index, and a negative pair otherwise.
The model then applies an InfoNCE-style contrastive loss. This strategy prevents the common pitfall of "random negatives" in standard contrastive learning, where related documents in a batch might be incorrectly pushed apart. This framework is flexible, supporting both self-supervised learning and explicit data augmentation (e.g., BERT-based synonym replacement).
3. Experimental Results
The authors evaluated TSCTM on three short-text datasets: TagMyNews, AG News, and Google News. The model was compared against state-of-the-art baselines, including NQTM and ProdLDA.
3.1 Topic Quality and Diversity
TSCTM consistently achieved superior Topic Coherence ($C_V$) and Topic Diversity ($TU$). For instance, on the TagMyNews dataset with $K=50$ topics, TSCTM reached a coherence score of 0.445, surpassing NQTM's 0.432.
3.2 Semantic Similarity Performance
A qualitative comparison highlights TSCTM's ability to capture semantic relationships more effectively than previous models. Figure 2 shows examples of short texts and their corresponding similarity scores. While the baseline NQTM often assigns high similarity scores to unrelated texts (e.g., $x^{(i)}$ and $x^{(j)}$), TSCTM correctly identifies high similarity for semantically related pairs ($x^{(i)}$ and $x^{(\ell)}$) and low similarity for unrelated ones.
3.3 Downstream Tasks
In clustering and classification, TSCTM showed substantial improvements. On TagMyNews, it achieved a clustering purity of 0.610, a significant increase over NQTM (0.595) and ProdLDA (0.260). Visualizations using t-SNE further confirm that TSCTM produces more distinct and well-separated topic clusters compared to the "smeared" distributions of its predecessors.
4. Conclusion
TSCTM addresses the data sparsity problem in short text topic modeling by introducing a topic-semantic contrastive learning objective. By using vector quantization to guide the contrastive process, the model effectively regularizes the latent space, leading to higher quality topics and better performance in downstream applications. Future work may explore more sophisticated sentence-level augmentations and the integration of metadata to further enhance model robustness.
Summary updated