Enabling Cross-Platform Comparison of Online Communities Using Content and Opinion Similarity

Prasanna Lakkur Subramanyam, Jeng-Yu Chou, Kevin Nam, Brian LevineEMNLP 2024 · DOI

Introduction

Measuring the similarity between online communities is essential for content recommendation and tracking the spread of misinformation. However, traditional methods often suffer from "dimensionality blindness," conflating topical content with ideological sentiment. Furthermore, many models rely on platform-specific network data (like user overlap), making cross-platform comparisons difficult. This research introduces a framework that treats Content (what is discussed) and Opinion (how it is discussed) as independent axes, enabling automated, cross-platform comparisons without manual seeding.

Methodology

The framework utilizes two primary metrics to decouple community characteristics:

1. Content Similarity: Emb-PSR

The Embedding-based Post Similarity Ratio (Emb-PSR) measures topical overlap. By using SentenceBERT to create vector embeddings of post titles, the system establishes a "self-similarity" baseline for a community. It then calculates the fraction of posts in a target community that fall within this baseline, effectively quantifying how much of the topical space is shared.

2. Opinion Similarity: BOTS

The Binary Opinion Transfer Similarity (BOTS) captures a community's ideological "vibe." The researchers fine-tuned a Llama-2-7B model (using LoRA) to predict whether a comment would be upvoted or downvoted within a specific community. If a model trained on Community A's preferences accurately predicts reactions in Community B, the two share a similar worldview. This text-based approach works across any platform with a feedback mechanism (e.g., Reddit, GAB).

Experimental Results

The researchers evaluated these methods using extensive datasets from Reddit (2018–2023) and GAB.

Decoupling Content and Opinion

A key finding is that communities can discuss identical topics while holding diametrically opposed views. The visualization below demonstrates how subreddits are distributed across these two axes. For instance, highly political subreddits often cluster in high content similarity but vary significantly in opinion similarity, highlighting the existence of ideological echo chambers.

Figure 1: Visualization of subreddits across Content Similarity (Emb-PSR) and Opinion Similarity (BOTS) axes.

Performance Benchmarks

The proposed metrics significantly outperform existing baselines. Emb-PSR achieved a higher "mean" score in identifying topically related categories compared to the industry-standard Community2Vec (C2V). Furthermore, BOTS demonstrated a significantly higher correlation (0.51) with external political bias ratings (AllSides) than previous partisanship-based distance metrics or C2V.

Figure 5: Comparison of BOTS and Emb-PSR against baseline methods on political bias correlation and group identification.

Key Insights

  • Fine-Tuning is Essential: Base LLMs (even with 5-shot prompting) struggle to capture community-specific nuances. Fine-tuning on local "approval" signals is necessary to model a community's unique "persona."
  • Cross-Platform Utility: By relying on text and binary feedback rather than user IDs, the framework successfully compared GAB's discourse to Reddit subreddits, identifying that GAB shares content with r/neoliberal but aligns in opinion with r/AskThe_Donald.

Limitations and Future Work

While powerful, the current BOTS implementation is computationally expensive, requiring individual fine-tuned models for each community. Additionally, the framework is currently text-only, potentially missing context in meme-heavy communities. Future research aims to incorporate multimodal data to better capture the full spectrum of modern social media discourse.

Summary updated

BibTeX
@inproceedings{subramanyam2024enabling,
  author = {Prasanna Lakkur Subramanyam and Jeng-Yu Chou and Kevin Nam and Brian Levine},
  title = {{Enabling Cross-Platform Comparison of Online Communities Using Content and Opinion Similarity}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.586},
}
Figure