Code2Que: a tool for improving question titles from mined code snippets in stack overflow

Zhipeng Gao, Xin Xia, David Lo, John C. Grundy, Yuan-Fang Li

TL;DR

Code2Que is a web-based tool designed to help developers write better Stack Overflow questions by automatically generating descriptive titles directly from code snippets. By combining a deep learning sequence-to-sequence model with specialized "copy" and "coverage" mechanisms, it produces titles that are clearer and more likely to elicit expert help than the original low-quality posts.

What Problem Does This Paper Address?

Stack Overflow (SO) is a lifeline for developers, but its effectiveness depends entirely on the quality of the questions asked. Poorly phrased, vague, or incomplete titles often fail to attract experts, leaving developers without answers. While previous research has focused on identifying these low-quality questions, the authors of Code2Que aim to fix them. They address the "cold start" problem for less experienced users who may struggle to articulate their technical issues in English or lack the specific terminology needed to describe a bug in their code.

Methods

The authors developed a two-stage framework consisting of offline learning and online recommendation.

  1. Offline Learning: The system was trained on over a million high-quality <code snippet, question> pairs. The core architecture is a deep sequence-to-sequence (seq2seq) model. To handle the unique challenges of source code, they added three critical enhancements:
  2. Attention Mechanism: Helps the model focus on the most relevant parts of a long code snippet.
  3. Copy Mechanism: This is vital for programming. It allows the model to "copy" rare tokens (like specific function names or variables) directly from the code into the title, rather than trying to guess them from a limited vocabulary.
  4. Coverage Mechanism: Prevents the model from getting stuck in a loop and repeating the same words (e.g., "how to how to").
  5. Online Recommendation: When a user pastes code into the tool, the trained model generates a new title. Simultaneously, the tool converts the code into a vector embedding to retrieve the top five most similar existing questions from Stack Overflow, providing immediate context and potential solutions.
Overview of the Code2Que approach, showing the transition from offline training to online recommendation.

Datasets

The researchers utilized the Stack Overflow data dump from September 2019. They focused on five major languages: Python, Java, JavaScript, C#, and SQL.

  • Training Set: They collected over 1 million pairs where the question had a community score of at least 1 (indicating "good" quality).
  • Evaluation Set: For the user study, they sampled 50 "low-quality" pairs from Python and Java datasets—specifically those marked as lacking clarity—to see if Code2Que could improve them.

Key Insights and Ideas

  • Code as Language: The researchers treat code as an ordered sequence of tokens, allowing them to leverage advanced Natural Language Processing (NLP) techniques typically used for translation.
  • The Power of Copying: In technical Q&A, the specific name of a library or function is often the most important word in the title. Standard NLP models often fail here because these names are "out-of-vocabulary." The Copy Mechanism (illustrated in Figure 2) is the "aha moment" that allows the tool to correctly include specific methods like get_client_ip in the generated title.
  • Multi-Metric Evaluation: Instead of just using automated metrics like BLEU (which can be misleading for titles), they used human evaluators to judge Clearness (is it easy to read?), Fitness (does it match the code?), and Willingness to Respond (would an expert actually click this?).
Example showing how different model enhancements (Attention, Coverage, Copy) affect the quality of the generated question.

Key Results

In a head-to-head comparison against the original human-written "low-quality" titles, Code2Que was a clear winner:

  • Python: The tool's titles "won" on Clearness (52.4%), Fitness (55.2%), and Willingness to Respond (61.2%).
  • Java: While slightly lower, the tool still outperformed humans across all metrics, winning roughly 43-49% of the time and tying in many others.
  • Social Engineering: The high "Willingness to Respond" score suggests that the tool is particularly good at making questions look "professional" enough to attract help from the community.
Table showing the results of the human evaluation, comparing Code2Que generated titles against original human titles.

Limitations and Future Work

The authors acknowledge that Code2Que currently only looks at the code snippet itself. It does not yet consider the body text of the post or the tags provided by the user, which could provide valuable context. Furthermore, while the tool significantly improves many titles, it isn't perfect; some generated titles still lack the nuance of a high-scoring human post.

Future iterations aim to incorporate more "context information" beyond just the code to further refine the accuracy and relevance of the generated questions. You can see the current interface in Figure 3.

The Code2Que homepage, demonstrating the user interface for inputting code and receiving generated/retrieved questions.

Summary updated

Abstract

Stack Overflow is one of the most popular technical Q&A sites used by software developers. Seeking help from Stack Overflow has become an essential part of software developers’ daily work for solving programming-related questions. Although the Stack Overflow community has provided quality assurance guidelines to help users write better questions, we observed that a significant number of questions submitted to Stack Overflow are of low quality. In this paper, we introduce a new web-based tool, Code2Que, which can help developers in writing higher quality questions for a given code snippet. Code2Que consists of two main stages: offline learning and online recommendation. In the offline learning phase, we first collect a set of good quality ⟨code snippet, question⟩ pairs as training samples. We then train our model on these training samples via a deep sequence-to-sequence approach, enhanced with an attention mechanism, a copy mechanism and a coverage mechanism. In the online recommendation phase, for a given code snippet, we use the offline trained model to generate question titles to assist less experienced developers in writing questions more effectively. To evaluate Code2Que, we first sampled 50 low quality ⟨code snippet, question⟩ pairs from the Python and Java datasets on Stack Overflow. Then we conducted a user study to evaluate the question titles generated by our approach as compared to human-written ones using three metrics: Clearness, Fitness and Willingness to Respond. Our experimental results show that for a large number of low-quality questions in Stack Overflow, Code2Que can improve the question titles in terms of Clearness, Fitness and Willingness measures.
BibTeX
@inproceedings{gao2021code,
  author = {Zhipeng Gao and Xin Xia and David Lo and John C. Grundy and Yuan-Fang Li},
  title = {{Code2Que: a tool for improving question titles from mined code snippets in stack overflow}},
  booktitle = {ESEC/SIGSOFT FSE},
  year = {2021},
  doi = {10.1145/3468264.3473114},
}

Loading related content...

Figure