A Study of Update Request Comments in Stack Overflow Answer Posts

Mohammad Sadegh Sheikhaei, Yuan Tian, Shaowei Wang

TL;DR

This paper tackles the challenge of identifying "update request comments" (URCs) on Stack Overflow, which signal issues in answers and can lead to improvements. The authors developed a machine learning model that can automatically detect these URCs with high accuracy, paving the way for better community awareness and maintenance of answer quality.

What Problem Does This Paper Address?

Stack Overflow is a massive Q&A platform where developers seek solutions to programming problems. While answers are crucial, they can become outdated or contain errors. "Update request comments" (URCs) are comments that point out these issues, prompting answerers to revise their posts. However, many URCs go unaddressed, potentially due to their low visibility. The current community-vote mechanism doesn't effectively highlight these critical comments. This paper aims to understand the prevalence and handling of URCs and proposes an automated method to detect them, making them more visible and actionable.

Methods

The researchers employed a multi-faceted approach:

  • Manual Analysis: They began by manually analyzing a large dataset of Stack Overflow comments to understand URC characteristics, their addressing patterns, and the roles of users involved.
  • Feature Engineering: Based on their manual analysis, they extracted a rich set of features from comments, posts, users, and time-related aspects. These features aim to capture nuances that distinguish URCs from other comments.
  • Machine Learning Models: They trained and evaluated several supervised learning models, including classic ones like Random Forest and Logistic Regression, and advanced deep learning models like CNN and a BERT-based Transformer (Multimodal-Toolkit).
  • Cross-Programming Language Evaluation: To ensure generalizability, models trained on Java comments were tested on JavaScript and Python comments.

Datasets

  • Java Comments Dataset: A newly created dataset of 1,221 annotated comments from 384 Stack Overflow answer posts tagged with "Java." This dataset includes labels for whether a comment is a URC, where it's addressed (post body, next comment, or both), and the addressing comment ID.
  • JavaScript Test Dataset: 377 comments from JavaScript questions.
  • Python Test Dataset: 289 comments from Python questions.

Key Insights and Ideas

  • URCs are prevalent but often ignored: Roughly half of all comments are URCs, yet a significant portion (36.5%) remain unaddressed even after a year.
  • Community voting isn't enough: Comment scores are often zero, making URCs invisible to the community.
  • Answer owners are key, but not always enough: While answer owners address most URCs (80.1%), the sheer volume of unaddressed URCs suggests a need for broader community involvement.
  • Context matters: Features beyond just the text of a comment (like author role, post score, and comment order) are crucial for identifying URCs.
  • Cross-language prediction is feasible: Models trained on Java data perform well on JavaScript and Python data, suggesting that URC characteristics are somewhat language-independent.

Key Results

  • High Detection Accuracy: The proposed automated URC detection models achieved an average accuracy of around 90% on JavaScript and Python comments. The BERT-based Multimodal-Toolkit model, using both text and extracted features, reached an accuracy of 90% and an AUC of 0.96.
  • Feature Importance: User features (like the commenter's role) and comment features (like comment score and order) were found to be highly influential in detecting URCs.
  • Addressing Speed: A significant portion of URCs (55.3%) are addressed within 24 hours, but a substantial number still remain unaddressed.

Limitations and Future Work

  • External Validity: The study was primarily conducted on Java questions and tested on JavaScript and Python. While results suggest generalizability, findings might differ for other topics or less popular programming languages.
  • Annotation Bias: The reliance on manually labeled data introduces potential bias. However, the authors mitigated this by having two annotators independently label a subset of the data, achieving high inter-rater agreement.
  • Future Work: The authors plan to increase the number of annotated comments, investigate specific types of URCs that are more likely to be addressed, and analyze URCs addressed by non-answer owners to understand their motivations and roles. They also suggest that more advanced models might be needed to better capture comment history for identifying unaddressed URCs.

Summary updated

Abstract

Comments play an important role in updating Stack Overflow (SO) posts. They are used to point out a problem (e.g., obsolete answer and buggy code) in a SO answer or ask for more details about a proposed answer. We refer to this type of comment as update request comments (URCs), which may trigger an update to the answer post and thus improve its quality. In this study, we manually analyze a set of 384 sampled SO answer posts and their associated 1,221 comments to investigate the prevalence of URCs and how URCs are addressed. We find that around half of the analyzed comments are URCs. While 55.3% of URCs are addressed within 24 hours, 36.5% of URCs remain unaddressed after a year. Moreover, we find that the current community-vote mechanism could not differentiate URCs from non-URCs. Thus many URCs might not be aware by users who can address the issue or improve the answer quality. As a first step to enhance the awareness of URCs and support future research on URCs, we investigate the feasibility of URC detection by proposing a set of features extracted from different aspects of SO comments and using them to build supervised classifiers that can automatically identify URCs. Our experiments on 377 and 289 comments posted on answers to JavaScript and Python questions show that the proposed URC classifier can achieve an accuracy of 90% and an AUC of 0.96, on average.
BibTeX
@misc{sheikhaei2023study,
  author = {Mohammad Sadegh Sheikhaei and Yuan Tian and Shaowei Wang},
  title = {{A Study of Update Request Comments in Stack Overflow Answer Posts}},
  year = {2023},
  eprint = {2304.07848},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.07848},
}
Figure