TL;DR
This paper investigates what makes an answer "high quality" on ResearchGate, an academic Q&A platform. They found that while traditional academic metrics like authority (RG score) and timely responses matter, the content of the answer, specifically its factual nature and provision of resources, is crucial. Interestingly, social niceties actually detract from perceived quality.
What Problem Does This Paper Address?
Online Question and Answer (Q&A) platforms have become ubiquitous, but understanding what constitutes a "good" answer varies significantly between general platforms (like Yahoo! Answers) and specialized academic ones (like ResearchGate). While previous research has explored answer quality on general Q&A sites, there's a notable gap in understanding these dynamics within the academic context. Academic Q&A sites host complex questions requiring specialized knowledge, and the users are scholars themselves. This paper aims to bridge this gap by identifying the characteristics that influence peer-judged answer quality on ResearchGate and developing predictive models for it.
Methods
The researchers employed a multi-faceted approach:
- Feature Engineering: They extracted two types of features from answers:
- Web-captured features: These are readily available data points from the platform, including the responder's RG score (a measure of academic authority), their institution's RG score, publication impact points, answer length, and response time.
- Human-coded features: These were derived from a manual content analysis of the answers, focusing on aspects like the presence of social elements (greetings, politeness), consensus building, providing factual information, offering resources, referring to other researchers, giving opinions, and sharing personal experiences.
- Quality Scoring: Answer quality was defined by peer judgment, specifically the number of "upvotes" an answer received. Answers with 3+ upvotes were considered "high quality," 1-2 votes "medium," and zero votes "low quality."
- Predictive Modeling: They constructed and compared three classification models (Naive Bayes, SVM, and Multiple Regression) to predict answer quality using the extracted features. The performance of these models was evaluated using metrics like accuracy, precision, recall, and AUC scores.
- Contrastive Analysis: They further analyzed the predictive power of different feature sets: human-coded features alone, web-captured features alone, and a combination of both.
Datasets
The study utilized a dataset of 1021 answers from ResearchGate Q&A across three disciplines: Library and Information Science (LIS), History of Art, and Astrophysics. This dataset included posts ranging from September 2009 to November 2013 (for LIS) and more recent data for the other disciplines. This forms a new benchmark dataset for academic Q&A quality analysis.
Key Insights and Ideas
- Authority and Timeliness Still Matter: Similar to general Q&A sites, a responder's academic authority (measured by RG score) and how quickly they answer positively correlate with perceived quality.
- Content is King (for Academics): Providing factual information, offering resources, and sharing opinions or personal experiences are strong indicators of high-quality answers on ResearchGate. This suggests academic users prioritize substantive content.
- Social Niceties Can Be Detrimental: Surprisingly, answers containing social elements like greetings or polite phrases were more likely to be rated as low quality. This highlights a key difference from general Q&A platforms where social interaction might be more valued.
- Longer Answers Can Be Better: Contrary to some findings on general Q&A sites, longer answers on ResearchGate were associated with higher quality. This implies academic users appreciate detailed and comprehensive responses.
- SVM Outperforms Other Models: The Support Vector Machine (SVM) algorithm, particularly with an RBF kernel, demonstrated superior performance in predicting answer quality compared to Naive Bayes and Multiple Regression.
Key Results
- Significant Correlations: The study found significant correlations between answer quality and several features, including response time (negative correlation), answer length (positive correlation), RG scores (positive correlation), and the presence of factual information, resources, opinions, and personal experiences (positive correlations). Social elements showed a negative correlation.
- Predictive Power: The optimized SVM model achieved an accuracy of 62.684% in predicting answer quality.
- Feature Portfolio Performance: The combination of web-captured and human-coded features yielded the best prediction accuracy (62.684%), outperforming models that used only web-captured features (59.158%) or only human-coded features (53.183%). This underscores the importance of a holistic feature set.
Limitations and Future Work
The authors acknowledge a key limitation: the insufficient quantity of high-quality answers in their dataset, which impacted the precision and recall for the "high quality" category in their SVM models. This also made it challenging to fully predict high-quality answers.
Future work suggested includes:
- User Studies: Conducting user studies to directly understand how users perceive the importance of different features for answer quality.
- Larger Datasets: Expanding the dataset size and diversity to improve prediction accuracy.
- More Features: Incorporating additional features, such as the contributor's specific field of expertise and their scientific contributions, to enhance predictive models.
- Cross-Platform Analysis: Testing the developed model on other professional Q&A sites (like Quora or Stack Overflow) to see if the findings generalize and to explore platform-specific nuances.
Summary updated