TL;DR
This paper introduces a new dataset of helpful and unhelpful comments from online support forums and investigates the linguistic features that distinguish them. They find that helpfulness is strongly correlated with user empathy and that specific language patterns, like positive and understanding phrasing, predict helpfulness.
What Problem Does This Paper Address?
Online forums, like Reddit's r/Advice, are popular platforms for individuals seeking advice on various life issues. While many users offer support, not all contributions are equally helpful. Distinguishing helpful comments from unhelpful ones is crucial for both human users and AI dialogue agents aiming to provide effective assistance. Existing research has qualitatively identified characteristics of helpfulness, but a systematic analysis of the language features that predict helpfulness has been lacking. This paper aims to bridge this gap by creating a labeled dataset and analyzing the linguistic markers of helpfulness.
Methods
The authors developed a novel dataset by leveraging Reddit's r/Advice subreddit, where users can explicitly mark comments as "helpful." They collected posts and their associated comments, labeling comments marked as "helpful" by the original poster as helpful, and other comments on the same posts as unhelpful. This criterion was chosen to ensure that the "helpful" label was actively provided by the person seeking advice.
To understand the relationship between helpfulness and empathy, they administered the Empathy Quotient (EQ) questionnaire to a subset of users and correlated their EQ scores with their average helpfulness rating across their comments.
Finally, they performed a linguistic analysis to identify features associated with helpfulness. This involved training baseline machine learning models to predict helpfulness and extracting significant predictors from a logistic regression model. They also conducted a manual annotation of a subset of comments to categorize helpfulness features.
Datasets
- Newly Introduced Dataset: A dataset of helpful and unhelpful comments from the r/Advice subreddit. This dataset was created by leveraging the "helped" marker provided by Reddit users. Comments marked as "helpful" were labeled as such, while other comments on the same posts were labeled as "unhelpful." The dataset consists of 24,964 posts and 92,477 comments, with 41,146 identified as helpful.
- Empathy Quotient (EQ) Questionnaire: A short form of the EQ questionnaire was used to measure user empathy.
Key Insights and Ideas
- Helpfulness and Empathy are Linked: There's a moderate but significant correlation between a user's overall helpfulness on the platform and their measured empathy score. This suggests that individuals who are more empathetic tend to be more helpful in online support contexts.
- Language Predicts Helpfulness: Specific linguistic patterns are strong indicators of helpfulness. Positive, friendly, and optimistic language is associated with helpful comments, while negative or dismissive language is linked to unhelpful ones.
- Understanding vs. Directing: Comments that show an attempt to understand the poster's perspective (e.g., by asking clarifying questions or using phrases like "it sounds like") are more likely to be perceived as helpful. Conversely, overly directive or patronizing language is less effective.
- Machine Learning Challenges: While humans can easily discern helpfulness, current machine learning models struggle to achieve high accuracy in predicting it, indicating the complexity of capturing nuanced social cues in language.
Key Results
- A moderate correlation (r=0.359) was found between user empathy scores and their average helpfulness.
- Helpfulness correlated most strongly with cognitive empathy, suggesting that understanding the poster's thought process is key.
- Linguistic analysis identified "positive and friendly words" and "words addressing the post author directly" as significant positive predictors of helpfulness.
- "Words indicating negative emotions" and "words that patronize the problem" were identified as negative predictors.
- Baseline models achieved a micro-F1 score of around 65-69%, which the authors note is lower than human judgment, highlighting the difficulty for machines.
Limitations and Future Work
- Dataset Size and Scope: While the dataset is substantial, it's limited to the r/Advice subreddit, which might not generalize to all online support forums.
- Machine Learning Performance: The relatively low performance of baseline models suggests that current techniques are not fully capturing the nuances of helpfulness.
- Future Work: The authors suggest exploring knowledge-enhanced models to better understand contextual semantic understanding. They also plan to release their dataset to facilitate further research in this area, aiming to improve automated dialogue agents and human support providers.
Summary updated