Simpler Hyperparameter Optimization for Software Analytics: Why, How, When?

Amritanshu Agrawal, Xueqi Yang, Rishabh Agrawal, Xipeng Shen, Tim Menzies

TL;DR

This paper introduces DODGE, a simple and fast hyperparameter optimization (HPO) technique that significantly speeds up software analytics tasks. They demonstrate that DODGE excels on "intrinsically simple" software engineering datasets, characterized by low intrinsic dimensionality, while struggling with more complex, non-SE datasets. This work provides a practical guide for choosing the right HPO method based on data complexity.

What Problem Does This Paper Address?

Software analytics relies heavily on data mining techniques, which often require extensive tuning of hyperparameters to achieve optimal performance. This hyperparameter optimization (HPO) process can be computationally very expensive, leading to long wait times and hindering rapid iteration and reproducibility. The authors aim to address this by proposing and evaluating a simpler, faster HPO method, DODGE, and to understand when it's appropriate to use such a simplified approach versus more complex, time-consuming methods. They highlight that the computational cost of HPO can be a significant bottleneck, leading to a "less CPU means more science" philosophy.

Methods

The core of the paper revolves around evaluating DODGE, a hyperparameter optimization technique that "dodges" redundant hyperparameter settings. DODGE works by exploring the hyperparameter space and deprecating settings that yield similar performance to already-seen configurations. This is achieved by modeling the hyperparameter space as a tree and adjusting weights based on performance outcomes, effectively pruning less promising branches.

The authors compare DODGE against several baseline methods, including:

  • Genetic Algorithms (GA) and Differential Evolution (DE), which are common HPO techniques.
  • Grid Search, a more exhaustive but slower method.
  • SMOTE + DE tuning, a state-of-the-art approach for defect prediction.
  • Standard data miners like CART, Random Forests, Logistic Regression, and KNN.

They also introduce a method to quantify intrinsic dimensionality of a dataset using Levina et al.'s fractal-based approach. This metric is proposed as a predictor for when DODGE will perform well.

Datasets

The study utilizes a diverse set of 120 datasets across five domains:

  1. Defect Prediction: 10 datasets from previous work, with metrics like average method complexity, lines of code, etc.
  2. Issue Tracking: 6 datasets from NASA projects, focusing on predicting issue severity.
  3. Github Issue Close Time: 63 datasets related to predicting how long it takes to resolve issues.
  4. Bad Smell Detection: 4 datasets for identifying code smells.
  5. Non-SE Problems: 37 datasets from the UCI Machine Learning Repository, covering various domains like physical, financial, and computer vision.

The authors note that the SE datasets are generally characterized by lower intrinsic dimensionality compared to the non-SE datasets.

Key Insights and Ideas

  • Intrinsic Dimensionality as a Predictor: The most significant insight is the strong correlation between a dataset's "intrinsic dimensionality" and DODGE's performance. Datasets with low intrinsic dimensionality (around 3) are where DODGE shines, while higher dimensionality (above 8) leads to poor performance.
  • "Less CPU Means More Science": The paper champions the idea that simplifying computationally intensive tasks like HPO frees up resources for more scientific exploration and experimentation.
  • DODGE's Speed Advantage: DODGE achieves comparable or better results than more complex methods in a fraction of the time, often terminating after only 30 evaluations compared to thousands or millions for others.
  • SE Data is Often "Simple": A key finding is that many common software engineering datasets are intrinsically low-dimensional, making DODGE a highly applicable and efficient tool for a wide range of SE analytics tasks.

Key Results

  • Defect Prediction: DODGE performed statistically better than prior state-of-the-art methods in 16 out of 20 results across 10 defect prediction datasets.
  • Text Mining: DODGE outperformed other methods in 5 out of 6 text mining datasets.
  • Issue Lifetime Estimation: DODGE showed statistically better performance than prior work in 75% of the 63 datasets.
  • Bad Code Smell Detection: DODGE performed as well as or better than other methods in all 4 datasets.
  • Non-SE Problems: DODGE performed poorly on the 37 non-SE datasets, succeeding in only 6 out of 31 comparisons. This stark contrast highlights the importance of intrinsic dimensionality.
  • Speed: DODGE was orders of magnitude faster than traditional methods, often completing its search in around 30 evaluations.

Limitations and Future Work

  • Limited Scope of Comparison: The authors acknowledge that they did not compare DODGE against all possible hyperparameter optimization methods or learners, as this would be infeasible. They focused on baselines found in prior SE literature.
  • Data Bias: While they used a large number of datasets, they acknowledge that sampling bias is always a potential threat. They recommend future work to test their findings on even more diverse SE and non-SE datasets.
  • Exploration of Sparsely Populated Regions: The paper identifies a "performance plateau" for DODGE in the intrinsic dimensionality range of 3.1 to 8, which they attribute to a sparsely populated region in their data. Future work should explore this region with more data.
  • Regression Tasks and Multi-Objective Optimization: The current study focused on classification tasks. Future work should explore DODGE's applicability to regression tasks and problems with more than two performance goals (e.g., optimizing precision, recall, and false alarm simultaneously).
  • Reducing Learner Set: The authors suggest reducing the list of learners used in Table 1 for faster experimentation.
  • Deprecating Worse Results: An extension to DODGE could involve deprecating tunings that lead to similar and worse results, potentially pruning larger parts of the output space.
  • Deep Learning for Complex Data: The paper notes that for intrinsically complex data (like source code translators), deep learning methods are likely more effective, and DODGE would not be recommended in such cases.

Summary updated

Abstract

How to make software analytics simpler and faster? One method is to match the complexity of analysis to the intrinsic complexity of the data being explored. For example, hyperparameter optimizers find the control settings for data miners that improve for improving the predictions generated via software analytics. Sometimes, very fast hyperparameter optimization can be achieved by just DODGE-ing away from things tried before. But when is it wise to use DODGE and when must we use more complex (and much slower) optimizers? To answer this, we applied hyperparameter optimization to 120 SE data sets that explored bad smell detection, predicting Github ssue close time, bug report analysis, defect prediction, and dozens of other non-SE problems. We find that DODGE works best for data sets with low "intrinsic dimensionality" (D = 3) and very poorly for higher-dimensional data (D over 8). Nearly all the SE data seen here was intrinsically low-dimensional, indicating that DODGE is applicable for many SE analytics tasks.
BibTeX
@article{agrawal2022simpler,
  author = {Amritanshu Agrawal and Xueqi Yang and Rishabh Agrawal and Xipeng Shen and Tim Menzies},
  title = {{Simpler Hyperparameter Optimization for Software Analytics: Why, How, When?}},
  journal = {IEEE Trans. Software Eng.},
  year = {2022},
  doi = {10.1109/tse.2021.3073242},
}
Figure