TL;DR
This paper presents a case study of an AI-based software defect prediction model implemented in a large telecommunications company. The model successfully predicted a significant portion of code defects, leading to reduced testing effort and improved efficiency, demonstrating the practical value of AI in software development.
What Problem Does This Paper Address?
Software defects are costly, especially when discovered late in the development lifecycle or after deployment. Testing, a critical but time-consuming phase, accounts for a substantial portion of development costs. The challenge lies in efficiently identifying defect-prone areas of code to focus testing efforts, thereby saving time and resources. Traditional methods like code reviews and inspections, while valuable, are labor-intensive and may not catch all defects. This paper addresses the need for more effective and efficient defect detection strategies by exploring the application of AI-based defect prediction models in a real-world industrial setting.
Methods
The authors developed a learning-based defect prediction model using a Naïve Bayes classifier. This approach involves a training phase where the model learns patterns from historical software metrics and defect data from past projects, and a testing phase where it predicts defect-proneness in new projects. The model utilizes static code attributes (e.g., lines of code, complexity) as input variables. A key aspect of their implementation was the development of an open-source tool called "Prest" for metrics extraction and defect prediction. They also incorporated a "version history flag" as a novel metric, indicating recent file activity, to help reduce false alarms.
Datasets
The study utilized data from a telecommunications company's internal projects. Specifically, they collected static code metrics and churn metrics from 9 projects across 10 releases. This dataset was matched with pre-release defects at the file level. The training set was derived from previous releases of a project, and the test set from the current release. The authors mention that this dataset was donated to a public data repository, PROMISE (2007).
Key Insights and Ideas
- AI as a Complement, Not a Replacement: The study suggests that AI-based defect predictors are most effective when used in conjunction with existing processes like code reviews and inspections, rather than as a complete replacement.
- The Power of Historical Data: The success of defect prediction hinges on the availability and quality of historical data (software metrics and defect records) for training the models.
- Reducing False Alarms is Crucial: While high detection rates are desirable, a high rate of false alarms (predicting a defect-free module as defective) can lead to wasted testing effort. The introduction of the "version history flag" metric was a practical step to mitigate this.
- Practical Implementation Challenges: Matching defects to specific files and ensuring accurate historical data logging are significant hurdles in real-world defect prediction.
- Process Change vs. AI Tooling: The paper compares the impact of a process change (Team Software Process - TSP) with the implementation of an AI defect predictor, suggesting that while process changes are beneficial, AI tools can offer targeted improvements in efficiency.
Key Results
- The implemented AI-based defect predictor achieved an average detection rate (PD) of 87% across 9 releases, with an average false alarm rate (PF) of 26%.
- This resulted in a significant reduction in inspection effort, with the model highlighting only 25% of files that contained 88% of defects, leading to an estimated 72% gain in inspection effort compared to random testing.
- In a pilot project, the model detected 75% of defects during the "coding" phase without extensive manual effort.
- The company observed a decrease in post-release failures due to code defects from 59% to 32% (a 44% decrease) after implementing the defect prediction model.
- The model helped reduce the effort in person-hours from 1.25 to 1.1 (an 11.2% decrease).
Limitations and Future Work
- Data Granularity: The authors note that matching defects at the file level was more beneficial than at the package level for reducing inspection efforts.
- False Alarms: While reduced, false alarms remain a concern, potentially leading to wasted effort on defect-free modules.
- Future Work: The company plans to improve predictions by incorporating additional metrics from version management systems. They also aim to use these predictions to compute defect rates for each release and compare them against reliability thresholds, potentially leading to delayed or canceled releases if risks are too high. Continuous calibration of the model with new data and parameters is also planned.
Summary updated