TL;DR
This paper investigates whether Large Language Models (LLMs) can effectively transform noisy app store reviews into high-quality user stories for agile development. The study finds that LLMs, particularly with few-shot prompting, can generate user stories that are comparable to, and sometimes even exceed, those written by human experts in terms of structure and readability, though challenges remain in ensuring story independence and uniqueness.
What Problem Does This Paper Address?
Software development teams heavily rely on user feedback, especially from app stores, to inform requirements and improve their products. However, these reviews are often unstructured, informal, and contain mixed concerns, making them difficult to analyze at scale. While automated methods exist, they often fail to produce "backlog-ready" user stories – clear, concise, and independent requirements essential for agile workflows. This paper addresses the gap in understanding how well modern LLMs can bridge this divide, specifically by generating human-like user stories directly from raw app reviews and comparing their quality to human-authored stories.
Methods
The researchers employed a multi-faceted approach to evaluate LLM-generated user stories:
- LLM Generation: Three prominent LLMs (GPT-3.5 Turbo, Gemini 2.0 Flash, and Mistral 7B Instruct) were used to generate user stories from app reviews.
- Prompting Strategies: The study tested three prompting techniques:
- Zero-shot: LLMs received only instructions.
- One-shot: Instructions plus one example review-story pair.
-
Two-shot: Instructions plus two example review-story pairs. This was done to understand how different levels of guidance affect output quality.
-
Automated Evaluation: A RoBERTa-based classifier, fine-tuned on the UStAI dataset, was used to assess user stories against nine Quality User Story (QUS) dimensions. This provided a scalable, objective measure of quality.
- Human Evaluation: An expert evaluation using the RUST framework (Readability, Understandability, Specifiability, Technical Aspects) was conducted by experienced agile practitioners. This provided a qualitative assessment of the generated stories.
Datasets
- Mini-BAR Corpus: Approximately 6,000 English-language reviews from three health-related applications (Garmin Connect, Huawei Health, and Samsung Health) were used. After preprocessing (emoji retention, length filtering), a curated set of 1,000 reviews was selected for LLM generation.
- UStAI Dataset: This dataset, containing around 3,000 LLM-generated stories from AI research abstracts, was used to fine-tune the RoBERTa classifier for automated quality assessment.
- Expert-Written User Stories: 183 user stories were manually created by two agile experts from a subset of the app reviews to serve as a human baseline for comparison.
Key Insights and Ideas
- Few-Shot Learning is Key: Providing LLMs with even a few examples (one-shot or two-shot prompting) significantly improves their ability to generate structured, human-like user stories compared to zero-shot prompting. This suggests that LLMs benefit greatly from contextual guidance.
- LLMs Can Mimic Human Quality: For many quality dimensions, LLM-generated stories match or even surpass human-written stories, particularly in areas like readability and understandability. This indicates LLMs are becoming powerful tools for initial requirement drafting.
- Independence and Uniqueness Remain a Hurdle: A consistent challenge for LLMs is generating user stories that are truly independent and unique. They tend to produce repetitive or semantically overlapping stories, which can hinder the creation of a robust and modular agile backlog.
- Model Choice Matters, but Prompting Matters More: While different LLMs show varying strengths (e.g., Gemini 2.0's structural consistency), the choice of prompting strategy appears to have a more significant impact on overall user story quality.
Key Results
- Structural Adherence: Gemini 2.0 Flash showed exceptional adherence to the canonical user story format ("As a [role], I want [feature] so that [benefit]") even in zero-shot settings, with further improvement in few-shot scenarios. GPT-3.5 Turbo also showed significant improvement with few-shot prompting, moving from ~50% deviation to near-perfect adherence. Mistral 7B, however, performed best with zero-shot, suggesting smaller models might overfit to examples.
- Automated Quality Scores: LLMs, especially with few-shot prompting, achieved higher scores than human-written stories in criteria like atomicity (0.82-0.83 vs. 0.76) and minimality (0.93-0.94 vs. 0.83). However, they lagged in independence (0.26-0.28 vs. 0.32) and uniqueness (0.29-0.31 vs. 0.33).
- Human Evaluation (RUST Scores): Top-performing LLMs (Gemini 2.0 Two-Shot, GPT-3.5 Turbo Two-Shot) achieved overall RUST scores comparable to or slightly higher than agile experts (e.g., Gemini 2.0 Two-Shot: 4.53 vs. Agile Expert: 4.39). This was consistent across readability, understandability, specifiability, and technical aspects.
Limitations and Future Work
- Internal Validity: Human evaluation is subjective, and while efforts were made to mitigate bias, personal judgment could still influence results. Variations in prompt design and LLM settings could also impact performance.
- External Validity: The dataset was limited to 1,000 English reviews from three health-related apps, potentially limiting generalizability to other languages, domains, or LLM architectures. There's also a possibility of LLMs recognizing and reproducing content they were trained on.
- Construct Validity: The automated and human evaluation metrics might not fully capture all real-world aspects of user story quality, such as ease of maintenance or impact on team collaboration.
- Future Work: The authors suggest expanding human assessment with more raters and diverse domains, exploring cross-domain and cross-lingual generation, integrating LLMs with agile tools, and investigating advanced prompting strategies to address the remaining challenges of independence and uniqueness.
Summary updated