Characterizing the Language of Online Communities and its Relation to Community Reception

Trang Tran, Mari Ostendorf

TL;DR

This paper explores how language style and topic contribute to identifying online communities and influencing how content is received. They found that language style, particularly using a hybrid word and part-of-speech model, is a better indicator of community identity than topic alone. Crucially, content that matches a community's language style is more likely to be well-received (higher karma).

What Problem Does This Paper Address?

Online communities, like Reddit, are rich environments for studying human communication. While we intuitively understand that communities form around shared interests (topics), the role of how people communicate (language style) in defining these communities and influencing their dynamics is less understood. Specifically, this paper asks:

  1. Can language style, independent of topic, help us identify which online community a piece of text belongs to?
  2. Does matching a community's language style, or its topic, lead to better reception (e.g., upvotes) of that content?

Understanding these questions has implications for everything from recommender systems and content moderation to detecting newcomers and even generating more engaging online content.

Methods

The authors employ two main approaches to model language:

  • Topic Modeling: They use Latent Dirichlet Allocation (LDA) to identify underlying themes within discussions. They train both 100- and 200-dimensional LDA models.
  • Style Modeling: This is where the novelty lies. They experiment with several n-gram language models to capture linguistic style, balancing word usage with grammatical structure (Part-of-Speech or POS tags). They compare:
  • word_only: A standard model using only words.
  • tag_only: A model using only POS tags.
  • hyb-15k and hyb-500.30: Hybrid models that combine a limited set of frequent words with POS tags. The hyb-500.30 model is particularly interesting as it uses a mix of general and community-specific words to capture nuanced community jargon.

They use these models for two main tasks:

  1. Community Classification: Can the models predict which subreddit a discussion thread or a user's comments belong to?
  2. Community Feedback Correlation: Is there a relationship between how well a piece of content (thread or comment) matches a community's style or topic, and its reception (karma)?

Datasets

The primary dataset used is from Reddit, specifically focusing on discussions within various subreddits. They selected eight diverse subreddits and created an artificial "merged_others" subreddit to act as a catch-all for normalization. Data was collected from January 1, 2014, to January 31, 2015. For training, they filtered for discussion threads with at least 100 comments and excluded individual comments with non-positive karma to focus on content that is generally accepted.

Key Insights and Ideas

  • Style Trumps Topic for Identity: Surprisingly, language style is a more powerful identifier of an online community than its topic, even for communities with very specific themes. This suggests that how people talk is as, if not more, defining than what they talk about.
  • Hybrid Style Models Shine: The hybrid word-POS models, especially hyb-500.30, outperform both pure word-based and pure POS-based models in classifying communities. This highlights the importance of capturing both lexical content (jargon, specific terms) and grammatical structure.
  • Style Drives Reception: Content that aligns with a community's language style is more likely to receive positive feedback (higher karma). This is a significant finding, suggesting that "fitting in" linguistically is rewarded.
  • Topic's Role in Reception is Negative (or Neutral): Interestingly, topic similarity showed a mostly negative or neutral correlation with community reception. This could imply that while topics define communities, discussions that deviate slightly from the core topic but maintain stylistic coherence might be more engaging, or that topic similarity is simply less important than style for engagement.
  • Specialization is Key for Success: Users who are highly successful (high k-indices, a measure of consistent positive reception) tend to participate in fewer subreddits. This mirrors offline expertise – it's rare to be a true expert in many diverse fields.

Key Results

  • Community Classification Accuracy: The hyb-500.30 style model achieved the highest average accuracy for classifying both discussion threads (86.5%) and authors (51.0%), significantly outperforming topic models and simpler style models.
  • Feedback Correlation: For thread-level feedback, the hyb-500.30 style model consistently showed a positive, statistically significant correlation between stylistic similarity and karma. For user-level feedback, the hyb-500.30 style model also showed the strongest positive correlation with user k-indices.
  • Confusable Communities: Communities with similar themes (e.g., politics and worldnews, askmen and askwomen) were more difficult to distinguish, even with advanced models.

Limitations and Future Work

  • Separating Style and Topic: The authors acknowledge the inherent difficulty in completely separating style and genre from topic, as these factors can be intertwined.
  • Data Filtering: Excluding comments with non-positive karma might bias the results towards content that is already well-received, potentially missing nuances in how less popular content is perceived.
  • Future Directions: The paper suggests implications for detecting newcomers, predicting post popularity, and generating more contextually appropriate language. Further research could explore these applications more deeply.

Summary updated

Abstract

This work investigates style and topic aspects of language in online communities: looking at both utility as an identifier of the community and correlation with community reception of content. Style is characterized using a hybrid word and part-of-speech tag n-gram language model, while topic is represented using Latent Dirichlet Allocation. Experiments with several Reddit forums show that style is a better indicator of community identity than topic, even for communities organized around specific topics. Further, there is a positive correlation between the community reception to a contribution and the style similarity to that community, but not so for topic similarity.
BibTeX
@inproceedings{tran2016characterizing,
  author = {Trang Tran and Mari Ostendorf},
  title = {{Characterizing the Language of Online Communities and its Relation to Community Reception}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  doi = {10.18653/v1/d16-1108},
}
Figure