Abstract
In crowdsourced testing, developers often struggle with a high volume of redundant and low-quality test reports. This paper proposes DivClass, a novel prioritization method that integrates a diversity strategy with text classification. By employing an asymmetric similarity computation, DivClass effectively identifies redundant reports, including those containing multiple bugs. Experimental results across five real-world datasets demonstrate that DivClass achieves an Average Percentage of Fault Detected (APFD) of 0.8887, marking a 14.12% improvement over the state-of-the-art DivRisk method.
The Crowdsourced Testing Challenge
Crowdsourced testing leverages a global pool of workers to test software across diverse environments. While this increases coverage, it introduces a significant bottleneck: the manual inspection of test reports. Developers must sift through thousands of submissions, many of which are duplicates or "noise."
The primary difficulty lies in the redundancy of reports. Traditional prioritization methods often fail to account for the "multi-bug" paradox, where a single comprehensive report might be incorrectly flagged as a duplicate of a simpler one, or vice versa, leading to inefficient bug discovery.
Methodology: The DivClass Approach
DivClass addresses these issues through a three-stage pipeline:
- NLP Preprocessing: The system utilizes the Language Technology Platform (LTP) for Chinese word segmentation, removes stop words, and performs synonym replacement. This ensures that semantic variations (e.g., "select" vs. "choose") do not hinder similarity detection.
- Asymmetric Similarity Computation: Unlike standard symmetric metrics (like Jaccard), DivClass uses an asymmetric formula. This recognizes that a detailed "multi-bug" report contains the information of a "single-bug" report, but the reverse is not true. This allows the system to better manage overlapping information.
- Prioritization Strategy:
- Classification: Reports are grouped into classes using propagation-based classification. If a report is similar to an existing class, it is merged; otherwise, it forms a new class.
- Diversity Selection: The algorithm selects the most "risky" (keyword-dense) report from each distinct class first. This ensures developers are exposed to a wide variety of unique bugs early in the process.
Experimental Evaluation
The researchers evaluated DivClass on five crowdsourced datasets from mobile applications: Justforfun, SE-1800, iShopping, CloudMusic, and UBook. These datasets contained 1,728 reports in total.
Key Results
- Detection Speed: DivClass achieved an average APFD of 0.8887, outperforming the previous state-of-the-art, DivRisk, by 14.12%.
- Metric Impact: The asymmetric similarity strategy alone contributed a 4.82% increase in APFD compared to symmetric alternatives.
- Efficiency: In the UBook dataset, DivClass required the inspection of only 34 reports to find all bugs, compared to 228 for DivRisk—a massive reduction in developer effort.
Conclusion
DivClass significantly streamlines the bug-reporting process in crowdsourced testing. By combining classification with a diversity-focused prioritization strategy and a novel asymmetric similarity metric, it ensures that unique and critical bugs are identified faster. Future work will focus on adapting the NLP pipeline for other languages and exploring deep-learning techniques for even more robust report classification.
Summary updated