A Case Study Investigating the Role of Generative AI in Quality Evaluations of Epics in Agile Software Development

Werner Geyer, Jessica He, Daita Sarkar, Michelle Brachman, Chris Hammond, Jennifer Heins, Zahra Ashktorab, Carlos Rosemberg, Charlie Hill

Abstract

In Agile software development, the quality of high-level requirement documents, known as "Epics," is critical to project success. However, these documents are frequently vague or incomplete, leading to significant "requirement debt." This paper investigates the efficacy of using Large Language Models (LLMs) to automatically evaluate Epic quality. Through the development of a structured rubric and a tool called Epic Evaluator, researchers at IBM conducted a case study with 17 product managers. The results indicate that while AI provides valuable, actionable feedback that aligns with human judgment, its effectiveness is currently limited by a lack of domain-specific context and an inability to account for the document's lifecycle stage.

1. Introduction

In Agile methodologies, an "Epic" serves as the bridge between a broad product vision and the granular user stories executed by engineering teams. Despite their importance, Epics in fast-paced industrial environments are often poorly defined. This lack of clarity results in "requirement debt," which manifests as delivery delays, wasted effort, and increased costs.

While peer reviews are the gold standard for quality assurance, Product Managers (PMs) often lack the bandwidth for rigorous manual evaluations. This study explores whether Generative AI can serve as an "AI-as-a-Judge," providing instant, high-quality feedback to help PMs refine requirements before they reach development teams.

2. Methodology

The researchers employed a three-phase approach to test the viability of AI-driven evaluations:

2.1 Rubric Development

Working with subject matter experts, the team established a quality rubric consisting of eight essential elements:

  1. Title
  2. Problem Statement
  3. Product Outcome & Instrumentation
  4. User Stories
  5. Requirements
  6. Assumptions
  7. Non-functional Requirements
  8. Out-of-Scope

2.2 Tool Construction: Epic Evaluator

The team developed Epic Evaluator, a Python-based application using the Gradio framework. The backend utilizes Llama-3-70B-instruct (via IBM watsonx.ai). Rather than providing a simple numerical score, the tool identifies the presence of rubric elements, assigns a rating (High, Medium, or Low), explains the reasoning behind the rating, and offers concrete, actionable recommendations for improvement.

2.3 User Study

The evaluation involved 17 IBM product managers in semi-structured interviews and concept tests. Participants submitted their own real-world Epics to the tool and critiqued the AI’s performance, utility, and accuracy.

3. Key Insights and Discussion

3.1 Actionability Over Accuracy

A primary finding was that PMs valued actionable recommendations over numerical scores. Participants viewed the AI not as a final arbiter of quality, but as a "sparring partner" or automated peer reviewer. The ability of the LLM to suggest specific additions or clarifications was cited as its most useful feature.

3.2 The "Domain Gap" and Semantic Quality

LLMs struggle with the "semantic" quality of requirements due to a lack of internal organizational context. For instance, the model might flag internal project codenames or specialized industry terminology as "jargon" or "unclear" because it lacks access to the company's specific vocabulary. This highlights a gap between structural completeness and contextual accuracy.

3.3 Stage-Awareness

Epics are living documents that evolve over time. Participants noted that an early-stage "placeholder" Epic should not be held to the same standard as one ready for development. Current AI evaluations tend to be static; future iterations must be "stage-aware" to avoid penalizing documents that are intentionally incomplete during the ideation phase.

4. Results

  • Human-AI Alignment: 15 out of 17 participants (88%) agreed with the AI’s assessments and recommendations for their specific Epics.
  • Trust: Participants reported a moderate-to-high level of trust in the system, with an average rating of 3.5 out of 5.
  • Efficiency: PMs estimated that a high-quality Epic could save up to five days of downstream work by preventing communication breakdowns. They viewed the AI as a significant accelerant in reaching that "high-quality" state.
  • Model Performance: Iterative testing identified Llama-3-70B as the optimal model for this use case, providing the best balance of stability and contextual relevance within the researchers' environment.

5. Limitations and Future Work

As an industry case study within IBM, the results may reflect specific corporate cultures and Agile practices. Furthermore, the study used a static testing environment where participants viewed screenshots rather than interacting with a live tool.

Future research will focus on integrating Retrieval-Augmented Generation (RAG) to provide the LLM with domain-specific knowledge, such as product roadmaps and stakeholder lists. This is expected to bridge the "domain gap" and improve the semantic accuracy of the feedback. Additionally, the researchers plan to expand the study to include perspectives from developers and designers to see how AI-evaluated Epics impact their workflow.

Summary updated

BibTeX
@inproceedings{geyer2025case,
  author = {Werner Geyer and Jessica He and Daita Sarkar and Michelle Brachman and Chris Hammond and Jennifer Heins and Zahra Ashktorab and Carlos Rosemberg and Charlie Hill},
  title = {{A Case Study Investigating the Role of Generative AI in Quality Evaluations of Epics in Agile Software Development}},
  booktitle = {CHIWORK},
  year = {2025},
  doi = {10.1145/3729176.3729200},
}
Figure