Using Knowledge Transfer and Rough Set to Predict the Severity of Android Test Reports via Text Mining

Shikai Guo, Rong Chen, Hui Li

TL;DR

This paper tackles the challenge of automatically predicting the severity of Android bug reports, which are often numerous and lack explicit severity labels. The authors propose a novel Knowledge Transfer Classification (KTC) approach that leverages bug reports from other open-source projects to train a model for Android reports. They also introduce an Importance Degree Reduction (IDR) strategy using rough sets to refine the extracted keywords, improving prediction accuracy.

What Problem Does This Paper Address?

Crowdsourced software testing, while cost-effective and efficient, generates a massive volume of bug reports. Manually assessing the severity of these reports is a time-consuming and labor-intensive bottleneck for development teams. A key challenge is that Android test reports, unlike those in some other projects, are not consistently labeled with severity. This lack of labeled data makes it difficult to train traditional supervised machine learning models. Existing methods often require manual labeling, which is impractical at scale. The paper aims to bridge this gap by developing an automated system that can accurately predict bug severity, thereby streamlining the bug triage process.

Methods

The core of the proposed solution is a Knowledge Transfer Classification (KTC) approach. This method addresses the lack of labeled Android bug reports by transferring knowledge from well-labeled bug repositories of other open-source projects (like Eclipse and Mozilla). The process involves several key steps:

  1. Text Preprocessing: Standard Natural Language Processing (NLP) techniques are applied, including:
  2. Tokenization: Breaking down text into individual words or tokens.
  3. Stop-word Removal: Eliminating common words (e.g., "the," "in") that don't carry much semantic meaning for classification.
  4. Stemming: Reducing words to their root form (e.g., "computerized," "computerize" to "computer") to consolidate variations.
  5. Importance Degree Reduction (IDR): This is a novel contribution. After extracting keywords, the authors use a rough set-based strategy to identify the most "characteristic" keywords. This process aims to:
  6. Reduce the dimensionality of the data by filtering out noisy or less informative keywords.
  7. Improve the quality of the training data and, consequently, the predictive model.
  8. The IDR strategy focuses on the "importance degree" of keywords in relation to bug severity.
  9. Keyword Vector Modeling: The refined keywords are used to create a vector representation for each bug report. A report is represented as a vector where each dimension corresponds to a keyword, and the value is 1 if the keyword is present and 0 otherwise.
  10. Classification: Several standard machine learning classifiers are then applied to these keyword vectors to predict the severity of the Android test reports. The paper evaluates Naïve Bayes (NB), K-Nearest Neighbor (KNN), Support Vector Machines (SVM), and Decision Tree (J48).

Datasets

The study utilizes two types of datasets:

  • Training Data: Bug reports from two major open-source projects: Eclipse and Mozilla. These datasets are chosen because they are known to have manually labeled severity information, making them suitable for knowledge transfer.
  • Evaluation Data: 600 randomly selected Android test reports from the Android bug tracker system. These reports were manually labeled with severity for the purpose of this study, as the original system lacks this information.

Key Insights and Ideas

  • Knowledge Transfer is Viable: The study demonstrates that knowledge learned from bug reports in one software ecosystem (Eclipse, Mozilla) can be effectively transferred to predict bug severity in another (Android). This is crucial for domains where labeled data is scarce.
  • Rough Sets for Noise Reduction: The IDR strategy, leveraging rough set theory, proves effective in identifying and reducing noise from keyword extraction. This is particularly important given the varied language and potential inconsistencies in crowdsourced bug reports.
  • Keyword Importance Matters: Not all keywords are created equal. The IDR method highlights the value of focusing on keywords that have a strong correlation with bug severity, leading to more robust models.
  • NB Classifier Shines: Among the tested classifiers, Naïve Bayes, when combined with the IDR approach, consistently performed best across various components and projects.

Key Results

The experiments show that the proposed KTC approach with IDR generally outperforms standard classification methods without knowledge transfer or noise reduction.

  • Accuracy Improvement: For example, the average accuracy of NB classification for Eclipse reports was 0.725, which improved to 0.758 with the IDR + NB approach. Similar improvements were observed for Mozilla.
  • Classifier Performance: The NB classifier, particularly with the IDR strategy, showed the highest accuracy in predicting Android bug report severity. SVM and KNN also showed good performance, while J48 was less effective.
  • Area Under Curve (AUC): AUC values, a robust metric for imbalanced datasets, also indicated improvements with the proposed approach, with the IDR + NB method achieving an AUC of approximately 0.758 for Eclipse and 0.778 for Mozilla components.

Limitations and Future Work

  • Conclusion Validity: The authors acknowledge that bug reports submitted by the community might not always have perfectly accurate severity labels, even after manual relabeling. They also note that the "normal" severity category was omitted due to potential unreliability.
  • Internal Validity: The approach primarily uses the "short summary descriptions" of bug reports. Future work could explore incorporating other features (e.g., product, component) to potentially enhance model performance.
  • Construct Validity: The assumption that component-specific terminology improves prediction might be challenged if users interpret component fields differently. The study mitigated this by selecting components with a significant number of reports.
  • External Validity: The study focused on Eclipse and Mozilla. While these are diverse projects, the results might not generalize perfectly to all software projects or environments. The authors suggest investigating other cases with fewer bug reports but more rigorous review processes.
  • Future Work: The authors propose exploring additional data sources, more thorough preprocessing of longer bug report descriptions, and investigating topic modeling for severity labeling. They also plan to investigate cases with fewer bug reports but more conscious review.

Summary updated

Abstract

Crowdsourcing is an appealing and economic solution to software application testing because of its ability to reach a large international audience. Meanwhile, crowdsourced testing could have brought a lot of bug reports. Thus, in crowdsourced software testing, the inspection of a large number of test reports is an enormous but essential software maintenance task. Therefore, automatic prediction of the severity of crowdsourced test reports is important because of their high numbers and large proportion of noise. Most existing approaches to this problem utilize supervised machine learning techniques, which often require users to manually label a large number of training data. However, Android test reports are not labeled with their severity level, and manual labeling is time-consuming and labor-intensive. To address the above problems, we propose a Knowledge Transfer Classification (KTC) approach based on text mining and machine learning methods to predict the severity of test reports. Our approach obtains training data from bug repositories and uses knowledge transfer to predict the severity of Android test reports. In addition, our approach uses an Importance Degree Reduction (IDR) strategy based on rough set to extract characteristic keywords to obtain more accurate reduction results. The results of several experiments indicate that our approach is beneficial for predicting the severity of android test reports.
BibTeX
@inproceedings{guo2017using,
  author = {Shikai Guo and Rong Chen and Hui Li},
  title = {{Using Knowledge Transfer and Rough Set to Predict the Severity of Android Test Reports via Text Mining}},
  booktitle = {Symmetry},
  year = {2017},
  url = {https://openreview.net/forum?id=RHXYrEy9d4},
}
Figure