Mitigating Data Sparsity for Short Text Topic Modeling by Topic-Semantic Contrastive Learning

Xiaobao Wu, Anh Tuan Luu, Xinshuai Dong

Abstract

To overcome the data sparsity issue in short text topic modeling, existing methods commonly rely on data augmentation or the data characteristic of short texts to introduce more word co-occurrence information. However, most of them do not make full use of the augmented data or the data characteristic: they insufficiently learn the relations among samples in data, leading to dissimilar topic distributions of semantically similar text pairs. To better address data sparsity, in this paper we propose a novel short text topic modeling framework, Topic-Semantic Contrastive Topic Model (TSCTM). To sufficiently model the relations among samples, we employ a new contrastive learning method with efficient positive and negative sampling strategies based on topic semantics. This contrastive learning method refines the representations, enriches the learning signals, and thus mitigates the sparsity issue. Extensive experimental results show that our TSCTM outperforms state-of-the-art baselines regardless of the data augmentation availability, producing high-quality topics and topic distributions.

Introduction

Traditional topic modeling techniques, such as Latent Dirichlet Allocation (LDA), have long served as the standard for extracting thematic structures from large document corpora. These models rely heavily on word co-occurrence patterns; if two words frequently appear within the same document, they are statistically likely to belong to the same topic. However, this reliance becomes a significant liability when applied to short texts, such as tweets, news headlines, or product reviews.

In these contexts, the "data sparsity" problem arises: a 10-word snippet lacks the depth to establish robust co-occurrence signals. While previous attempts to solve this have utilized data augmentation (e.g., adding synonyms) or heuristic clustering, these methods often fail to capture the nuanced semantic relationships between different documents. This frequently results in topic distributions that are inconsistent or redundant. To address these limitations, we introduce the Topic-Semantic Contrastive Topic Model (TSCTM), a framework designed to refine latent representations through intelligent contrastive learning.

Methodology

TSCTM is built upon a Variational Autoencoder (VAE) architecture, integrated with a novel Topic-Semantic Contrastive (TSC) learning objective. The framework focuses on three core components to mitigate sparsity:

1. Vector Quantization (VQ) for Sampling

Standard contrastive learning often suffers from the "false negative" trap, where two different documents discussing the same topic are mistakenly treated as a negative pair and pushed apart in the latent space. TSCTM avoids this by using Vector Quantization. The model maps a document's topic distribution to the nearest "topic embedding" in a codebook.

  • Positive Pairs: Documents that map to the same topic embedding are identified as semantically similar.
  • Negative Pairs: Documents mapping to different embeddings are treated as dissimilar. This strategy ensures that the model learns to group documents based on shared semantics rather than just identical word usage.

2. Dual-Objective Learning

The model is trained to minimize two losses simultaneously:

  • Reconstruction Loss: A standard VAE objective that ensures the model can accurately reconstruct the original text from its latent representation.
  • Contrastive Loss (TSC): A regularizing objective that pulls semantically similar documents together while pushing dissimilar ones away, effectively "cleaning" the latent space.

3. Flexibility with Augmentation

TSCTM is designed to be agnostic to the presence of external data. It can function as a standalone model or incorporate data augmentation (such as WordNet synonyms). When augmentation is used, the TSC objective ensures that the original text and its augmented counterpart maintain nearly identical topic distributions, a feat previous models struggled to achieve.

Experimental Evaluation

We evaluated TSCTM against state-of-the-art baselines, including NQTM and CLNTM, across three benchmark datasets:

  • TagMyNews title: 31,223 news titles categorized into 7 classes.
  • AG News: A subset of 8,000 articles across 4 categories.
  • Google News: 11,066 snippets across 152 granular categories.

Quantitative Results

TSCTM demonstrated superior performance across all primary metrics:

  • Topic Coherence ($C_V$): On the TagMyNews dataset (50 topics), TSCTM achieved a coherence score of 0.514, significantly outperforming the next best baseline (0.458).
  • Topic Diversity ($TU$): The model achieved a perfect diversity score of 1.000 on the Google News dataset, indicating that every generated topic was unique and distinct.
  • Downstream Tasks: The topic distributions produced by TSCTM led to higher accuracy in text clustering (Purity/NMI) and classification tasks compared to all other tested models.

Qualitative Analysis

Qualitative reviews revealed that TSCTM effectively solves the "redundancy" problem. While baseline models often produced multiple, overlapping topics for a single subject (e.g., three separate topics for a single news event), TSCTM successfully consolidated these into a single, coherent theme with distinct keywords.

Conclusion

TSCTM represents a significant advancement in short text topic modeling by addressing the data sparsity problem through semantic awareness. By utilizing Vector Quantization to guide contrastive learning, the model produces more coherent, diverse, and useful topic representations. Future work will explore the integration of metadata—such as hashtags, authors, and sentiment—to further enhance the granularity of topic discovery.

Summary updated

Abstract

To overcome the data sparsity issue in short text topic modeling, existing methods commonly rely on data augmentation or the data characteristic of short texts to introduce more word co-occurrence information. However, most of them do not make full use of the augmented data or the data characteristic: they insufficiently learn the relations among samples in data, leading to dissimilar topic distributions of semantically similar text pairs. To better address data sparsity, in this paper we propose a novel short text topic modeling framework, Topic-Semantic Contrastive Topic Model (TSCTM). To sufficiently model the relations among samples, we employ a new contrastive learning method with efficient positive and negative sampling strategies based on topic semantics. This contrastive learning method refines the representations, enriches the learning signals, and thus mitigates the sparsity issue. Extensive experimental results show that our TSCTM outperforms state-of-the-art baselines regardless of the data augmentation availability, producing high-quality topics and topic distributions.
BibTeX
@inproceedings{wu2022mitigating,
  author = {Xiaobao Wu and Anh Tuan Luu and Xinshuai Dong},
  title = {{Mitigating Data Sparsity for Short Text Topic Modeling by Topic-Semantic Contrastive Learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=iXKUKG25Pc},
}
Figure