Mitigating Data Sparsity for Short Text Topic Modeling by Topic-Semantic Contrastive Learning

Xiaobao Wu, Anh Tuan Luu, Xinshuai Dong

Abstract

Short text topic modeling faces significant challenges due to data sparsity, where limited word co-occurrences hinder traditional methods like Latent Dirichlet Allocation (LDA). This paper presents the Topic-Semantic Contrastive Topic Model (TSCTM), a framework that leverages contrastive learning to enhance topic discovery in short texts. By employing vector quantization to identify semantically similar and dissimilar pairs, TSCTM learns robust topic distributions. Experimental results across multiple benchmark datasets demonstrate that TSCTM significantly outperforms state-of-the-art baselines in topic quality, clustering, and classification tasks.

1. Introduction

Traditional topic models rely heavily on the "bag-of-words" assumption, identifying topics through frequent word co-occurrences within documents. While effective for long-form content, this approach fails for short texts such as tweets or news headlines, which often contain fewer than ten words. This phenomenon, known as data sparsity, results in insufficient information for models to capture meaningful patterns. Existing solutions often use data augmentation or document clustering but fail to fully exploit relational information between samples, often leading to inconsistent topic assignments for semantically similar documents.

2. Methodology: TSCTM Framework

The TSCTM framework is built upon Neural Topic Models, specifically Variational Autoencoders (VAEs). The core innovation is the Topic-Semantic Contrastive (TSC) objective, which regularizes the latent space by pulling semantically related representations together and pushing unrelated ones apart.

As illustrated in Figure 1, the process involves encoding text into a latent representation ($h$) and a corresponding topic distribution ($\theta$). To determine positive and negative pairs without human labels, the model utilizes Vector Quantization (VQ). VQ maps continuous topic distributions to discrete "topic embeddings." Two texts are considered a positive pair if they map to the same discrete embedding index, and a negative pair otherwise.

Conceptual diagram of the TSCTM framework showing the relationship between latent representations and topic distributions.

The model then applies an InfoNCE-style contrastive loss. This strategy prevents the common pitfall of "random negatives" in standard contrastive learning, where related documents in a batch might be incorrectly pushed apart. This framework is flexible, supporting both self-supervised learning and explicit data augmentation (e.g., BERT-based synonym replacement).

3. Experimental Results

The authors evaluated TSCTM on three short-text datasets: TagMyNews, AG News, and Google News. The model was compared against state-of-the-art baselines, including NQTM and ProdLDA.

3.1 Topic Quality and Diversity

TSCTM consistently achieved superior Topic Coherence ($C_V$) and Topic Diversity ($TU$). For instance, on the TagMyNews dataset with $K=50$ topics, TSCTM reached a coherence score of 0.445, surpassing NQTM's 0.432.

3.2 Semantic Similarity Performance

A qualitative comparison highlights TSCTM's ability to capture semantic relationships more effectively than previous models. Figure 2 shows examples of short texts and their corresponding similarity scores. While the baseline NQTM often assigns high similarity scores to unrelated texts (e.g., $x^{(i)}$ and $x^{(j)}$), TSCTM correctly identifies high similarity for semantically related pairs ($x^{(i)}$ and $x^{(\ell)}$) and low similarity for unrelated ones.

Comparison of short text examples and similarity scores between NQTM and TSCTM.

3.3 Downstream Tasks

In clustering and classification, TSCTM showed substantial improvements. On TagMyNews, it achieved a clustering purity of 0.610, a significant increase over NQTM (0.595) and ProdLDA (0.260). Visualizations using t-SNE further confirm that TSCTM produces more distinct and well-separated topic clusters compared to the "smeared" distributions of its predecessors.

4. Conclusion

TSCTM addresses the data sparsity problem in short text topic modeling by introducing a topic-semantic contrastive learning objective. By using vector quantization to guide the contrastive process, the model effectively regularizes the latent space, leading to higher quality topics and better performance in downstream applications. Future work may explore more sophisticated sentence-level augmentations and the integration of metadata to further enhance model robustness.

Summary updated

Abstract

To overcome the data sparsity issue in short text topic modeling, existing methods commonly rely on data augmentation or the data characteristic of short texts to introduce more word co-occurrence information. However, most of them do not make full use of the augmented data or the data characteristic: they insufficiently learn the relations among samples in data, leading to dissimilar topic distributions of semantically similar text pairs. To better address data sparsity, in this paper we propose a novel short text topic modeling framework, Topic-Semantic Contrastive Topic Model (TSCTM). To sufficiently model the relations among samples, we employ a new contrastive learning method with efficient positive and negative sampling strategies based on topic semantics. This contrastive learning method refines the representations, enriches the learning signals, and thus mitigates the sparsity issue. Extensive experimental results show that our TSCTM outperforms state-of-the-art baselines regardless of the data augmentation availability, producing high-quality topics and topic distributions.
BibTeX
@inproceedings{wu2022mitigating,
  author = {Xiaobao Wu and Anh Tuan Luu and Xinshuai Dong},
  title = {{Mitigating Data Sparsity for Short Text Topic Modeling by Topic-Semantic Contrastive Learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=iXKUKG25Pc},
}
Figure