Does class size matter? An in-depth assessment of the effect of class size in software defect prediction

Amjed Tahir, Kwabena Ebo Bennin, Xun Xiao, Stephen G. MacDonell

TL;DR

This paper dives deep into whether class size truly matters in predicting software defects. It finds that while class size's impact isn't as universally significant as previously thought, it consistently mediates the relationship between the Coupling Between Objects (CBO) metric and defects, and can moderate the effect of Fan-out. The study advocates for a more nuanced approach, urging practitioners to investigate class size's specific impact rather than blindly controlling for it.

What Problem Does This Paper Address?

For decades, software engineering research has acknowledged that class size might influence defect prediction models. The common assumption is that larger classes are more prone to defects. However, there's been ongoing debate about whether class size's effect is a direct predictor of defects, or if it confounds the relationship between other important code metrics and defects. This means class size might be the real driver behind a metric's apparent predictive power, rather than the metric itself. Previous studies have offered conflicting advice: some suggest always controlling for class size, while others question its consistent importance. This paper aims to settle this by providing a thorough, in-depth analysis of class size's direct, indirect (mediation), and interaction (moderation) effects on software defects, using robust statistical methods.

Methods

The core of this study lies in its use of mediation and moderation analysis, specifically employing bootstrapping-based regression models. This is a more advanced approach than older methods like the Sobel test or causal-steps, offering better statistical power and reliability.

  • Mediation Analysis: This technique investigates if the effect of an independent variable (an OO metric) on a dependent variable (defects) is transmitted through a third variable (class size). In essence, it asks: "Does the metric affect defects because it affects class size, which then affects defects?"
  • Moderation Analysis: This method examines if a third variable (class size) changes the strength or direction of the relationship between an independent variable (an OO metric) and a dependent variable (defects). It asks: "Does the relationship between the metric and defects become stronger or weaker depending on the class size?"

The authors applied these analyses to both count models (predicting the number of defects) and binary models (predicting defect-proneness – whether a module is defective or not). For count models, they used Poisson regression, and for binary models, logistic regression. Bootstrapping with 5000 resamples was used to estimate confidence intervals for the indirect effects.

Datasets

The study utilized two publicly available datasets:

  • DAMB (D'Ambros et al., 2012): This dataset comprises 5 large open-source projects from the Eclipse and Apache ecosystems.
  • JURE (Jureczko and Madeyski, 2010): This dataset includes 18 systems, a mix of open-source and proprietary projects.

In total, 23 systems were analyzed, containing approximately 11.4k files and 2.5 million lines of code. All systems are Java-based. The authors note that the defect distributions in these datasets are not normally distributed, which informed their choice of statistical models.

Key Insights and Ideas

  • Class Size Isn't a Universal Culprit: Contrary to some prior assumptions, class size doesn't consistently and significantly mediate or moderate the relationship between all OO metrics and defects. This challenges the blanket recommendation to always control for size.
  • CBO is Consistently Affected by Size: The Coupling Between Objects (CBO) metric stands out. Class size consistently shows a significant mediation effect on the relationship between CBO and both the number of defects and defect-proneness across many systems. This suggests that CBO's predictive power might be heavily influenced by the size of the class it's applied to.
  • Fan-out's Relationship with Size is Nuanced: Class size shows a potential moderation effect on the relationship between the Fan-out metric and defects. This means class size can influence how strongly Fan-out predicts defects.
  • System-Specific Effects: The impact of class size varies significantly across different systems. What holds true for one project might not for another, highlighting the need for context-specific analysis.
  • Bootstrapping is Key: The study champions bootstrapping-based mediation and moderation analysis as more robust tools for understanding these complex relationships in software engineering.

Key Results

  • Mediation:
    • Count Models (Number of Defects): Class size significantly mediated the relationship between CBO and defects in 13 out of 23 systems. It also showed significant mediation for Fan-out in 10 systems. For other metrics like RFC, WMC, LCOM, and Fan-in, the evidence was inconsistent or weak.
  • Binary Models (Defect-Proneness): Similar to count models, CBO showed a significant mediation effect in 13 systems. Fan-out also showed some mediation, but less consistently than CBO.
  • Moderation:
    • Count Models: The moderation effect of class size was most consistently observed for RFC and Fan-out metrics across both datasets, though the evidence was mixed.
  • Binary Models: The moderation effect was less consistent. Only Fan-out showed a somewhat consistent moderation effect across systems.

Limitations and Future Work

  • Dataset Bias: The study primarily used Java-based open-source projects. While efforts were made to include diverse systems, the findings might not generalize to other programming languages, proprietary software, or different development methodologies.
  • Lack of Contextual Data: Factors like the specific domain of the program, the development team's experience, or the project's timeline were not investigated, but could influence the observed effects.
  • Inconclusive Results for Many Metrics: While CBO and Fan-out showed some consistent effects, many other metrics exhibited highly inconsistent results across systems, making it difficult to draw universal conclusions.

The authors recommend that researchers and practitioners:

  1. Examine class size's impact individually: Instead of blindly controlling for size, investigate its specific mediation or moderation effects for the data at hand.
  2. Explore other indirect variables: Future studies should investigate other potential mediators or moderators in defect prediction models.
  3. Utilize robust statistical methods: Employ techniques like bootstrapping for mediation/moderation analysis.

They also suggest that the statistical procedures used in this paper can be applied to other areas of software engineering, such as cost estimation and change prediction.

Summary updated

Abstract

In the past 20 years, defect prediction studies have generally acknowledged the effect of class size on software prediction performance. To quantify the relationship between object-oriented (OO) metrics and defects, modelling has to take into account the direct, and potentially indirect, effects of class size on defects. However, some studies have shown that size cannot be simply controlled or ignored, when building prediction models. As such, there remains a question whether, and when, to control for class size. This study provides a new in-depth examination of the impact of class size on the relationship between OO metrics and software defects or defect-proneness. We assess the impact of class size on the number of defects and defect-proneness in software systems by employing a regression-based mediation (with bootstrapping) and moderation analysis to investigate the direct and indirect effect of class size in count and binary defect prediction. Our results show that the size effect is not always significant for all metrics. Of the seven OO metrics we investigated, size consistently has significant mediation impact only on the relationship between Coupling Between Objects (CBO) and defects/defect-proneness, and a potential moderation impact on the relationship between Fan-out and defects/defect-proneness. Other metrics show mixed results, in that they are significant for some systems but not for others. Based on our results we make three recommendations. One, we encourage researchers and practitioners to examine the impact of class size for the specific data they have in hand and through the use of the proposed statistical mediation/moderation procedures. Two, we encourage empirical studies to investigate the indirect effect of possible additional variables in their models when relevant. Three, the statistical procedures adopted in this study could be used in other empirical software engineering research to investigate the influence of potential mediators/moderators.
BibTeX
@inproceedings{tahir2021does,
  author = {Amjed Tahir and Kwabena Ebo Bennin and Xun Xiao and Stephen G. MacDonell},
  title = {{Does class size matter? An in-depth assessment of the effect of class size in software defect prediction}},
  booktitle = {Empir. Softw. Eng},
  year = {2021},
  url = {https://openreview.net/forum?id=mVJc2fKQay},
}
Figure