Content Quality Vs. Attention Allocation: An LLM-Based Case Study in a Peer-To-Peer Mental Health Network

Teng Ye, Huijun Yan, Xusheng Huang, Connor Grogan, Walter Yuan, Qiaozhu Mei, Matthew O. JacksonSSRN 2024 · DOI

TL;DR

This paper investigates how online platforms, specifically a peer-to-peer mental health network, prioritize content, finding that timeliness often trumps quality. Using LLMs to assess comment quality across multiple dimensions, they demonstrate that re-ranking comments by quality, rather than just upvotes and recency, could significantly improve the support offered to users.

What Problem Does This Paper Address?

In online communities, especially those offering support for sensitive topics like mental health, users rely on crowdsourced responses. However, platforms typically use ranking algorithms that favor recency and upvotes, which can inadvertently promote less helpful or even harmful content. This is particularly problematic in mental health contexts where nuanced aspects like empathy, cultural understanding, and actionable advice are crucial for effective support, but are difficult to measure with traditional methods. The paper addresses the gap in understanding how these ranking mechanisms align with actual content quality and explores how to better direct user attention to genuinely valuable responses.

Methods

The researchers employed Large Language Models (LLMs), specifically OpenAI's ChatGPT-4, to evaluate the quality of millions of comments. They developed a multi-dimensional quality assessment framework, rating each comment on seven criteria: topical alignment, lexical precision, empathy expression, encouragement level, actionable suggestions, personal resonance, and cultural alignment. These scores were then averaged to create a composite quality score. To analyze attention allocation, they compared the quality of comments ranked by the platform's existing algorithm (based on upvotes and recency) against a hypothetical scenario where comments were ranked solely by their LLM-derived composite quality score. They also investigated the impact of comment timeliness on upvotes and explored simple re-ranking strategies to mitigate biases.

Datasets

The study utilized a dataset of over 96,000 Reddit posts and more than 3.15 million comments from 11 subreddits focused on mental health advice and help-seeking. The data was collected between June 2005 and December 2023, including timestamps and upvote counts for each comment. No new benchmarks were introduced; the focus was on analyzing existing platform data.

Key Insights and Ideas

  • Timeliness Over Quality: The most striking finding is the significant mismatch between content quality and attention allocation. Platforms often prioritize early responses, which, despite potentially lower quality, gain more visibility and upvotes.
  • LLMs as Quality Assessors: LLMs offer a scalable and nuanced way to assess the multi-dimensional quality of online content, going beyond simple topical relevance to capture crucial emotional and personal aspects.
  • The "Early Bird" Effect is Real, But Not the Whole Story: While early comments get more visibility and upvotes, the data suggests an additional "rich-gets-richer" dynamic where voters also tend to pay more attention to already higher-ranked comments, reinforcing the advantage of early posts.
  • Simple Re-ranking Can Make a Big Difference: Even basic re-ranking strategies, like randomly breaking ties in upvote counts or favoring newer comments, can significantly improve the quality of top-ranked content.

Key Results

  • Re-ranking comments by LLM-rated quality scores could improve the quality of the top-ranked comment by up to 39%.
  • For the top 10 comments, hypothetical quality-based ranking could still improve average quality by 16.4% compared to the platform's ranking.
  • The quality of comments generally declines as their rank percentile increases, but the top 10% of comments often have lower quality than the next decile, a pattern that contradicts the probabilistic ranking principle.
  • Emotional aspects of comments (empathy, encouragement, personal resonance) are rated lower than topical aspects (alignment, precision) and are most improved by quality-based re-ranking.
  • A large proportion of comments are posted within the first hour of the original post, with a sharp drop-off after 24 hours.
  • Even after accounting for timeliness, the first 30% of comments receive a disproportionately higher share of upvotes.

Limitations and Future Work

The authors acknowledge that their study is limited to a single domain (mental health) and a single platform (Reddit). They suggest that future work should test these findings on other platforms and domains. Additionally, they note that other factors, such as responder reputation, social network structure, and platform-specific promotion mechanisms, could influence rankings and interact with comment quality in complex ways. Future research should explore how these factors affect comment quality and user attention, and investigate more sophisticated ranking strategies that balance timeliness with quality and diversity. The study also highlights the need to understand how different attention allocation strategies impact help-seekers' outcomes and content creator behavior.

Summary updated

BibTeX
@misc{ye2024content,
  author = {Teng Ye and Huijun Yan and Xusheng Huang and Connor Grogan and Walter Yuan and Qiaozhu Mei and Matthew O. Jackson},
  title = {{Content Quality Vs. Attention Allocation: An LLM-Based Case Study in a Peer-To-Peer Mental Health Network}},
  year = {2024},
  doi = {10.2139/ssrn.5013979},
  howpublished = {SSRN},
}
Figure