Platform-Invariant Topic Modeling via Contrastive Learning to Mitigate Platform-Induced Bias

Minseo Koo, Doeun Kim, Sungwon Han, Sungkyu ParkEMNLP 2024 · DOI

Introduction

Social media platforms like X (formerly Twitter), Reddit, and YouTube are rich sources of public opinion. However, aggregating data across these platforms introduces "platform-induced bias." Standard topic models often conflate platform-specific semantic traits (writing styles) and syntactic traits (jargon like "RT" or "subreddit") with the actual topics of discussion. This paper introduces PITopic, a neural topic model designed to extract consistent, high-quality topics by neutralizing these platform-specific signatures.

The Challenge of Platform-Induced Bias

Traditional models like LDA or BERTopic frequently cluster content based on source-specific quirks. For instance, YouTube comments might be grouped together due to the prevalence of words like "subscribe" rather than their thematic content. The authors demonstrate the severity of this issue by showing that a simple classifier can predict a post's origin platform with 96% accuracy, indicating that latent platform signatures are dominant features in raw social media text.

The PITopic Framework

PITopic addresses these biases through two main innovations: platform-invariant contrastive learning and jargon-filtered reconstruction.

The PITopic architecture featuring platform-invariant contrastive learning and jargon-filtered BoW reconstruction.

Platform-Invariant Contrastive Learning

The model utilizes a frozen BERT backbone to encode text. It employs a contrastive objective consisting of an Alignment Loss, which ensures semantically similar texts have similar topic distributions, and a Distribution Loss, which regularizes the topic distribution toward a Dirichlet prior. By processing batches from only one platform at a time, the model is forced to identify features that satisfy the topic prior without relying on platform-specific signals.

Jargon Filtering for BoW Reconstruction

To prevent the decoder from learning platform-specific associations, PITopic identifies jargon using a modified class-based TF-IDF (c-TF-IDF) score. Words frequent on one platform but rare on others (e.g., "remindme" on Reddit) are removed from the target Bag-of-Words (BoW) reconstruction. This prevents the model from using platform-specific slang as a shortcut for topic identification.

Experimental Setup

The model was evaluated using 1.5 million posts related to "ChatGPT" collected from X, Reddit, and YouTube between December 2022 and March 2023.

Distribution of content across platforms over the study period.

Additionally, a synthetic benchmark was created using the 20NewsGroup, BBC, and New York Times datasets to test the model's ability to ignore source-specific styles while identifying shared topics like Tech, Sports, and Politics.

Key Results and Insights

PITopic outperformed seven state-of-the-art baselines across multiple metrics:

  • Topic Diversity: Achieved a score of 0.973, significantly higher than LDA (0.695) or Top2Vec (0.567).
  • Topic Coherence (UCI): On real-world data, PITopic reached an average UCI score of 3.559, nearly a full point higher than BERTopic (2.606).
  • Bias Reduction: A critical finding was the reduction in Mutual Information (MI) between topics and platforms. By minimizing MI, PITopic ensures that topic assignments are mathematically independent of the source platform.

Qualitatively, PITopic produced cleaner thematic lists (e.g., "neural, explainable, redefinition") compared to models that included platform-specific terms like "video" or "subreddit."

Conclusion and Future Work

PITopic demonstrates that platform-specific jargon and style are not merely noise but powerful anchors that bias traditional topic models. By leveraging contrastive learning and targeted jargon filtering, researchers can uncover authentic cross-platform discourse. Future work will focus on automating jargon threshold selection and exploring fine-tuning strategies for the BERT backbone to potentially yield even richer topic embeddings.

Summary updated

BibTeX
@inproceedings{koo2024platform,
  author = {Minseo Koo and Doeun Kim and Sungwon Han and Sungkyu Park},
  title = {{Platform-Invariant Topic Modeling via Contrastive Learning to Mitigate Platform-Induced Bias}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.650},
}
Figure