Abstract
When drafting question posts for Stack Overflow, developers may not accurately summarize the core problems in the question titles, which can cause these questions to not get timely help. Therefore, improving the quality of question titles has attracted the wide attention of researchers. An initial study aimed to automatically generate the titles by only analyzing the code snippets in the question body. However, this study ignored the helpful information in their corresponding problem descriptions. Therefore, we propose an approach SOTitle+ by considering bi-modal information (i.e., the code snippets and the problem descriptions) in the question body. Then we formalize the title generation for different programming languages as separate but related tasks and utilize multi-task learning to solve these tasks. Later we fine-tune the pre-trained language model CodeT5 to automatically generate the titles. Unfortunately, the inconsistent inputs and optimization objectives between the pre-training task and our investigated task may make fine-tuning hard to fully explore the knowledge of the pre-trained model. To solve this issue, SOTitle+ further prompt-tunes CodeT5 with hybrid prompts (i.e., mixture of hard and soft prompts). To verify the effectiveness of SOTitle+, we construct a large-scale high-quality corpus from recent data dumps shared by Stack Overflow. Our corpus includes 179,119 high-quality question posts for six popular programming languages. Experimental results show that SOTitle+ can significantly outperform four state-of-the-art baselines in both automatic evaluation and human evaluation. Our work indicates that considering bi-modal information and prompt learning in Stack Overflow title generation is a promising exploration direction.
Introduction: The Challenge of Title Generation
Stack Overflow (SO) is a critical resource for developers, yet its utility depends heavily on the quality of question titles. Vague or inaccurate titles often lead to delayed responses or no help at all. Previous attempts at automated title generation, such as the Code2Que model, focused exclusively on code snippets. However, this approach is fundamentally limited because the same code block can be used to ask vastly different questions depending on the developer's intent.
As shown in Figure 1, a single Python function for retrieving a client IP address might be part of a question about where to implement the code or how to get the "real" IP. Without the natural language "problem description," a model cannot capture the user's true intent. This "Aha!" moment highlights that code-only models have a fundamental ceiling.
The SOTitle+ Framework
To address these limitations, we propose SOTitle+, a novel approach that leverages bi-modal information and advanced prompt-tuning. The framework utilizes several advanced NLP techniques to bridge the gap between code and intent.
The SOTitle+ architecture (Figure 4) consists of three main phases:
- Corpus Construction: We extracted 179,119 high-quality posts across six languages (Python, Java, JavaScript, C#, HTML, and PHP) using heuristic rules (e.g., score ≥ 10, accepted answer, and presence of code snippets).
- Model Construction: We utilized Multi-task Learning to share knowledge across programming languages, which is particularly helpful for "low-resource" languages. The core engine is CodeT5, optimized using Hybrid Prompt Learning.
- Model Prediction: The model generates succinct, platform-appropriate titles by combining hard prompts (structural context) and soft prompts (tunable vectors).
Hybrid Prompt Learning
Standard fine-tuning often fails to fully exploit pre-trained knowledge because the task objective (generation) differs from the pre-training objective (masked word prediction). SOTitle+ reformulates title generation as a "fill-in-the-blank" task (Masked Span Prediction) using:
- Hard Prompts: Fixed, human-readable strings like "The problem description is:" and "The code snippet is:" to provide structural context.
- Soft Prompts: Tunable, continuous vectors (embeddings) that the model optimizes during training to bridge the gap between its pre-trained state and the specific nuances of Stack Overflow data.
Key Results and Insights
SOTitle+ was evaluated against four state-of-the-art baselines (Code2Que, SOTitle, CCBERT, and M3NSCT5) and zero-shot ChatGPT (GPT-3.5).
- Performance Gains: SOTitle+ significantly outperformed all baselines. In Python title generation, it improved the ROUGE-L score by 18.04% over the previous best model (CCBERT).
- Human Evaluation: Six developers rated SOTitle+ higher in similarity to ground truth, naturalness (fluency), and informativeness.
- Comparison with LLMs: In zero-shot scenarios for Ruby and Go, SOTitle+ outperformed ChatGPT. While ChatGPT's titles were grammatically correct, they were often overly wordy; SOTitle+ was better at capturing the specific technical essence required for a succinct SO title.
- Ablation Study: Removing the problem description caused the model's performance to plummet by over 50%, confirming that the bi-modal approach is essential.
Conclusion
SOTitle+ demonstrates that combining code snippets with natural language descriptions is vital for understanding developer intent on Stack Overflow. By employing hybrid prompt-tuning on CodeT5, the model effectively extracts pre-trained knowledge to generate high-quality, concise titles. While challenges remain regarding semantic noise and evaluation metrics, SOTitle+ represents a significant step forward in automated developer support.
Summary updated