Simpler Hyperparameter Optimization for Software Analytics: Why, How, When?

Amritanshu Agrawal, Xueqi Yang, Rishabh Agrawal, Rahul Yedida, Xipeng Shen, Tim Menzies

This paper, published in IEEE Transactions on Software Engineering, challenges the "bigger is better" trend in machine learning. While the AI community often races toward more complex hyperparameter optimization (HPO) algorithms, researchers demonstrate that for Software Engineering (SE) data, a "dumb" and fast approach isn't just sufficient—it’s often superior.

The Problem: The High Cost of Optimization

In software analytics, practitioners use machine learning to predict bugs, estimate issue resolution times, or detect "code smells." To achieve peak performance, they must tune "hyperparameters"—the configuration settings that control how a learner behaves.

Standard HPO methods, such as Grid Search or Genetic Algorithms, are computationally expensive, often requiring thousands or millions of evaluations. This creates a bottleneck where data scientists wait hours for results. The authors ask: Can we make software analytics simpler and faster without sacrificing accuracy?

Methods: The DODGE Approach

The authors evaluate a simplified HPO tool called DODGE. Unlike complex optimizers that build sophisticated probabilistic models of the search space, DODGE relies on two simple ideas:

  1. The RELAX Heuristic: It assumes that small differences in performance (less than a threshold $\epsilon$) are noise and should be ignored.
  2. Dodging: It models the hyperparameter space as a tree. If a specific setting leads to a performance score similar to one already seen, DODGE "deprecates" that branch and moves on.

To understand why this works, the authors use Levina’s Intrinsic Dimensionality calculator. This method estimates the true number of independent variables in a dataset. If a dataset has 100 columns but an intrinsic dimensionality of 3, most of that data is redundant, and the search space for optimization is much smaller than it appears.

Datasets and Scope

The study is massive in scope, utilizing 120 datasets across five domains, including NASA projects and GitHub issue tracking. A primary focus was SE Defect Prediction, utilizing well-known open-source projects.

Overview of the software engineering defect prediction datasets used in the study.

Key Insights: Why Simplicity Wins

  • SE Data is "Simple": Nearly all SE data studied had an intrinsic dimensionality ($\mu_D$) of approximately 3. In contrast, non-SE data from the UCI repository was much more complex ($\mu_D > 8$).
  • Complexity is Overkill: Because SE data is low-dimensional, complex optimizers like Tree-structured Parzen Estimators (TPE) spend time modeling complexities that don't exist in the data.
  • The Threshold of Complexity: DODGE works brilliantly when $\mu_D \le 3$. However, when the intrinsic dimensionality rises above 8, DODGE fails nearly half the time, and more complex optimizers become necessary.

Results and Performance

In defect prediction, DODGE performed statistically better than or equal to the prior state-of-the-art in 16 out of 20 trials. Crucially, it ran orders of magnitude faster than traditional methods. While grid search might take billions of calls to a data miner, DODGE achieves comparable results in a tiny fraction of the time, typically terminating after just 30 evaluations.

Comparison of optimized and default classifiers across various models, showing the significant performance gains achievable through optimization.

Conclusion

The findings suggest that for most software engineering tasks, the search space is much smaller than the number of features suggests. By "dodging" redundant configurations, practitioners can achieve state-of-the-art performance in seconds rather than hours. While complex optimizers remain necessary for high-dimensional data (like those found in the UCI repository), the "simple" approach is the right tool for the software analytics trade.

Summary updated

Abstract

How to make software analytics simpler and faster? One method is to match the complexity of analysis to the intrinsic complexity of the data being explored. For example, hyperparameter optimizers find the control settings for data miners that improve for improving the predictions generated via software analytics. Sometimes, very fast hyperparameter optimization can be achieved by just DODGE-ing away from things tried before. But when is it wise to use DODGE and when must we use more complex (and much slower) optimizers? To answer this, we applied hyperparameter optimization to 120 SE data sets that explored bad smell detection, predicting Github ssue close time, bug report analysis, defect prediction, and dozens of other non-SE problems. We find that DODGE works best for data sets with low "intrinsic dimensionality" (D = 3) and very poorly for higher-dimensional data (D over 8). Nearly all the SE data seen here was intrinsically low-dimensional, indicating that DODGE is applicable for many SE analytics tasks.
BibTeX
@article{agrawal2022simpler,
  author = {Amritanshu Agrawal and Xueqi Yang and Rishabh Agrawal and Rahul Yedida and Xipeng Shen and Tim Menzies},
  title = {{Simpler Hyperparameter Optimization for Software Analytics: Why, How, When?}},
  journal = {IEEE Trans. Software Eng.},
  year = {2022},
  doi = {10.1109/tse.2021.3073242},
}
Figure