Predicting Crashing Releases of Mobile Applications

Xin Xia, Emad Shihab, Yasutaka Kamei, David LoESEM 2016 · DOI

TL;DR

Mobile app quality is crucial, and crashing releases can severely impact user experience and app success. This paper introduces a machine learning approach to predict which app releases are likely to crash before they are deployed, using factors mined from commit logs and code. Their method significantly outperforms random prediction, with text-based features from commit messages proving to be the most effective predictors.

What Problem Does This Paper Address?

In today's competitive mobile app market, app quality is paramount. Users have little tolerance for bugs, especially crashes, which can lead to uninstalls and negative reviews. While traditional software development has methods for defect prediction, mobile apps present unique challenges: rapid release cycles, smaller development teams, and their increasing use in critical daily functions. Identifying "crashing releases" – those that cause the app to crash – early on is vital for developers to maintain user trust and ensure app success. However, existing methods for predicting software defects might not directly translate to the specific context of mobile app releases. This paper aims to fill this gap by developing a predictive model specifically for crashing mobile app releases.

Methods

The core of this research lies in a machine learning approach to predict crashing releases. The authors extract 20 distinct factors from mobile app repositories, categorizing them into six dimensions: complexity, time, code, diffusion, commit, and text.

  • Feature Engineering: Factors are derived from commit logs and code analysis. For instance, "complexity" includes cyclomatic complexity, "time" considers days since the last release, "code" looks at lines added/deleted and code size, "diffusion" examines how spread out changes are across the codebase, and "commit" counts the number of commits and bug-fixing commits.
  • Textual Features: A significant focus is placed on the "text" dimension. Commit messages are tokenized, stop words removed, and then processed using five different classifiers (Fuzzy Set, Naive Bayes, Naive Bayes Multinomial, Discriminative Naive Bayes Multinomial, and Complement Naive Bayes) to generate scores indicating the likelihood of a release being a crashing release based on the commit message content.
  • Prediction Model: A Naive Bayes classifier is used as the primary prediction model. This choice is motivated by its effectiveness in text classification and its ability to handle the textual features generated. The authors also compare Naive Bayes against other classifiers like Decision Trees (C4.5), K-Nearest Neighbors (kNN), and Random Forests.
  • Handling Imbalance: Recognizing that crashing releases are a minority class, the authors employ both over-sampling and under-sampling techniques on the training data to mitigate the class imbalance problem.

Datasets

The study utilizes data from 10 open-source mobile applications hosted on the F-Droid repository. These applications collectively represent 2,638 releases. The authors manually identified crashing releases by searching commit logs for keywords like "crash," "crashed," "crashing," and "crashes," followed by manual verification to reduce false positives and negatives.

Key Insights and Ideas

  • Text is King (for predicting crashes): The most surprising and impactful finding is that textual features derived from commit messages are the strongest predictors of crashing releases. This suggests that the language developers use to describe changes often contains subtle clues about potential instability.
  • Commit Message Nuances Matter: The authors found that terms related to UI issues ("screen," "loading," "UI," "button," "view") and localization ("dialog," "translation," "icon") frequently appeared in commit messages for crashing releases, highlighting the impact of these areas on app stability.
  • Crashing Releases are a Real Problem: The motivation section provides compelling evidence that crashing releases are a significant concern for developers, who often rush to fix them. This underscores the practical relevance of the research.
  • Naive Bayes Shines: While other classifiers were tested, Naive Bayes consistently performed well, especially when considering its efficiency and effectiveness with textual data.

Key Results

  • Significant Improvement: The proposed Naive Bayes model, using the extracted factors, achieved an average F1-score of 0.30 and an AUC score of 0.64 across the 10 applications. This represents a substantial improvement over random prediction (50% better F1, 28% better AUC) and majority prediction (30% better AUC).
  • Textual Factors Dominate: Factors from the "text" dimension consistently showed higher importance (IncNodePurity) and effect scores, indicating their strong predictive power.
  • Effect of Factors is Positive: All 20 studied factors showed a non-negative effect on the likelihood of a crashing release, meaning higher values of these factors generally increase the probability of a crash.
  • Performance on Imbalanced Data: The approach showed particularly strong improvements on highly imbalanced datasets (where crashing releases were a very small percentage), suggesting its robustness in real-world scenarios.

Limitations and Future Work

  • Keyword-Based Identification: The method for identifying crashing releases relies on specific keywords, which might lead to false positives or negatives. While manual verification was performed, it's not a perfect solution.
  • Generalizability: The study analyzed 10 applications. To further validate the findings, analyzing a larger and more diverse set of mobile applications is recommended.
  • Improving F1-Score: While recall is good, the F1-score could be higher. Future work could involve exploring additional factors or developing more sophisticated prediction algorithms.
  • Practical Deployment: The authors suggest deploying the tool in practice to gather real-world feedback and further refine the approach.
  • Bug Localization Integration: Investigating how bug localization techniques can be integrated to aid developers in fixing crashing releases is a promising avenue.
  • Developer Perception: Conducting empirical studies on developers' perceptions of crash release prediction is also suggested.

Summary updated

BibTeX
@inproceedings{xia2016predicting,
  author = {Xin Xia and Emad Shihab and Yasutaka Kamei and David Lo and Xinyu Wang},
  title = {{Predicting Crashing Releases of Mobile Applications}},
  booktitle = {ESEM},
  year = {2016},
  doi = {10.1145/2961111.2962606},
}
Figure