Crowdsourced Test Report Prioritization Based on Text Classification

Yuxuan Yang, Xin Chen

Abstract

In crowdsourced testing, developers often struggle with a high volume of redundant and low-quality test reports. This paper proposes DivClass, a novel prioritization method that integrates a diversity strategy with text classification. By employing an asymmetric similarity computation, DivClass effectively identifies redundant reports, including those containing multiple bugs. Experimental results across five real-world datasets demonstrate that DivClass achieves an Average Percentage of Fault Detected (APFD) of 0.8887, marking a 14.12% improvement over the state-of-the-art DivRisk method.

The Crowdsourced Testing Challenge

Crowdsourced testing leverages a global pool of workers to test software across diverse environments. While this increases coverage, it introduces a significant bottleneck: the manual inspection of test reports. Developers must sift through thousands of submissions, many of which are duplicates or "noise."

The standard workflow of crowdsourced testing, from task assignment to report collection.

The primary difficulty lies in the redundancy of reports. Traditional prioritization methods often fail to account for the "multi-bug" paradox, where a single comprehensive report might be incorrectly flagged as a duplicate of a simpler one, or vice versa, leading to inefficient bug discovery.

Methodology: The DivClass Approach

DivClass addresses these issues through a three-stage pipeline:

  1. NLP Preprocessing: The system utilizes the Language Technology Platform (LTP) for Chinese word segmentation, removes stop words, and performs synonym replacement. This ensures that semantic variations (e.g., "select" vs. "choose") do not hinder similarity detection.
  2. Asymmetric Similarity Computation: Unlike standard symmetric metrics (like Jaccard), DivClass uses an asymmetric formula. This recognizes that a detailed "multi-bug" report contains the information of a "single-bug" report, but the reverse is not true. This allows the system to better manage overlapping information.
  3. Prioritization Strategy:
    • Classification: Reports are grouped into classes using propagation-based classification. If a report is similar to an existing class, it is merged; otherwise, it forms a new class.
  4. Diversity Selection: The algorithm selects the most "risky" (keyword-dense) report from each distinct class first. This ensures developers are exposed to a wide variety of unique bugs early in the process.

Experimental Evaluation

The researchers evaluated DivClass on five crowdsourced datasets from mobile applications: Justforfun, SE-1800, iShopping, CloudMusic, and UBook. These datasets contained 1,728 reports in total.

Key Results

  • Detection Speed: DivClass achieved an average APFD of 0.8887, outperforming the previous state-of-the-art, DivRisk, by 14.12%.
  • Metric Impact: The asymmetric similarity strategy alone contributed a 4.82% increase in APFD compared to symmetric alternatives.
  • Efficiency: In the UBook dataset, DivClass required the inspection of only 34 reports to find all bugs, compared to 228 for DivRisk—a massive reduction in developer effort.

Conclusion

DivClass significantly streamlines the bug-reporting process in crowdsourced testing. By combining classification with a diversity-focused prioritization strategy and a novel asymmetric similarity metric, it ensures that unique and critical bugs are identified faster. Future work will focus on adapting the NLP pipeline for other languages and exploring deep-learning techniques for even more robust report classification.

Summary updated

Abstract

In crowdsourced testing, crowd workers from different places help developers conduct testing and submit test reports for the observed abnormal behaviors. Developers manually inspect each test report and make an initial decision for the potential bug. However, due to the poor quality, test reports are handled extremely slowly. Meanwhile, due to the limitation of resources, some test reports are not handled at all. Therefore, some researchers attempt to resolve the problem of test report prioritization and have proposed many methods. However, these methods do not consider the impact of duplicate test reports. In this paper, we focus on the problem of test report prioritization and present a new method named DivClass by combining a diversity strategy and a classification strategy. First, we leverage Natural Language Processing (NLP) techniques to preprocess crowdsourced test reports. Then, we build a similarity matrix by introducing an asymmetric similarity computation strategy. Finally, we combine the diversity strategy and the classification strategy to determine the inspection order of test reports. To validate the effectiveness of DivClass, experiments are conducted on five crowdsourced test report datasets. Experimental results show that DivClass achieves 0.8887 in terms of APFD (Average Percentage of Fault Detected) and improves the state-of-the-art technique DivRisk by 14.12% on average. The asymmetric similarity computation strategy can improve DivClass by 4.82% in terms of APFD on average. In addition, empirical results show that DivClass can greatly reduce the number of inspected test reports.
BibTeX
@inproceedings{yang2021crowdsourced,
  author = {Yuxuan Yang and Xin Chen},
  title = {{Crowdsourced Test Report Prioritization Based on Text Classification}},
  booktitle = {IEEE Access},
  year = {2021},
  doi = {10.1109/access.2021.3128726},
}
Figure