An empirical investigation of modularity metrics for indicating architectural technical debt

Zengyang Li, Peng Liang, Paris Avgeriou, Nicolas Guelfi, Apostolos AmpatzoglouQoSA 2014 · DOI

TL;DR

This paper investigates whether software modularity metrics can serve as reliable indicators of architectural technical debt (ATD), a common problem in software development that compromises system quality. The authors found that two specific metrics, the Index of Package Changing Impact (IPCI) and the Index of Package Goal Focus (IPGF), show a significant negative correlation with a proxy for ATD, making them promising alternatives for identifying debt when traditional methods are unavailable.

What Problem Does This Paper Address?

Software development often incurs "technical debt," which arises from design decisions that prioritize short-term gains over long-term system health. This debt can manifest in various forms, including architectural technical debt (ATD), which impacts maintainability and evolvability. A common way to measure ATD is by calculating the average number of modified components per commit (ANMCC). However, ANMCC is difficult to compute because it requires access to commit history, which is not always available for legacy systems or projects using reused components. This paper addresses the challenge of finding an alternative, readily available indicator for ATD. The authors propose using software modularity metrics, which can be calculated directly from source code, as a substitute for ANMCC.

Methods

The researchers conducted a holistic multiple case study involving thirteen open-source C# projects. Their approach involved:

  • Data Collection: For each project, they collected source code from two selected releases and their corresponding commit records.
  • Metric Calculation:
    • ANMCC: Calculated using commit history to represent ATD.
  • Modularity Metrics: Six specific modularity metrics (IIPU, IIPE, IPCI, IIPUD, IIPED, IPGF) were calculated from the source code of the earlier release of each project pair. These metrics aim to quantify different aspects of a system's modularity.
  • Statistical Analysis:
    • Normality Test: Shapiro-Wilk's tests were performed to determine if the data followed a normal distribution, guiding the choice of correlation test.
  • Correlation Analysis: Spearman's rank correlation coefficient was used to assess the relationship between the modularity metrics and the normalized ANMCC. A one-tailed test was employed due to a directional hypothesis (higher modularity implies lower ANMCC).
  • Comparative Analysis: The Hotelling-Williams test was used to compare the strength of the correlations between the most promising modularity metrics and ANMCC.

Datasets

The study utilized a dataset of thirteen C# open-source software (OSS) projects obtained from GitHub. For each project, data was collected from two selected releases. The selection criteria for projects and releases were designed to ensure representativeness and the availability of sufficient data for analysis. The paper provides detailed tables listing the selected projects, their release information, and the calculated modularity metrics and normalized ANMCC values.

Key Insights and Ideas

  • Modularity as a Proxy for ATD: The core idea is that well-modularized code should be easier to change, leading to fewer component modifications per commit (lower ANMCC). Therefore, modularity metrics can indirectly indicate the presence of ATD.
  • The Importance of Dependency Types: The study highlights that modularity metrics considering both "Use" and "Extend" dependencies (like IPCI and IPGF) are more effective in correlating with ATD indicators than those considering only one type of dependency. This suggests that a comprehensive view of inter-component relationships is crucial.
  • IPCI and IPGF Emerge as Strong Indicators: The research identified IPCI (Index of Package Changing Impact) and IPGF (Index of Package Goal Focus) as the most promising modularity metrics for indicating ATD.
  • IPCI Outperforms IPGF: IPCI demonstrated a stronger negative correlation with ANMCC than IPGF, suggesting it's a more accurate substitute for ANMCC when commit history is unavailable.
  • Practical Implications for IDEs: The findings suggest that integrating IPCI and IPGF calculations directly into Integrated Development Environments (IDEs) could provide developers and architects with continuous, automated feedback on architectural technical debt.

Key Results

  • Significant Correlations: The study found significant negative correlations between ANMCC and two modularity metrics:
  • IPCI: rho = -0.828, p-value = 0.0001 (highly significant)
    • IPGF: rho = -0.522, p-value = 0.0341 (significant)
  • Insignificant Correlations: The other four modularity metrics (IIPU, IIPE, IIPUD, IIPED) did not show significant correlations with ANMCC.
  • IPCI's Superiority: The Hotelling-Williams test confirmed that IPCI has a significantly stronger correlation with ANMCC than IPGF, making it the preferred metric for ATD indication.
  • Practical Utility: IPCI and IPGF, being calculable from source code, offer a valuable alternative to ANMCC, which relies on commit history. They can help architects and project managers assess and monitor ATD without needing complete historical data.

Limitations and Future Work

  • Construct Validity: The authors acknowledge that the chosen modularity metrics are only valid as substitutes for ANMCC over relatively short periods, as significant system evolution can impact their accuracy. They mitigated this by carefully selecting release pairs.
  • External Validity: The study was limited to C# projects. Generalizing these findings to other programming languages (like Java) requires further investigation.
  • Future Work:
    • Validate the findings with projects written in other languages (e.g., Java).
  • Explore new or adapt existing modularity metrics, potentially incorporating complex network analysis, to improve ATD prediction accuracy or reduce calculation effort.
  • Develop IDE plugins to facilitate the automatic calculation and reporting of IPCI and IPGF for practical ATD management.

Summary updated

BibTeX
@inproceedings{li2014empirical,
  author = {Zengyang Li and Peng Liang and Paris Avgeriou and Nicolas Guelfi and Apostolos Ampatzoglou},
  title = {{An empirical investigation of modularity metrics for indicating architectural technical debt}},
  booktitle = {QoSA},
  year = {2014},
  doi = {10.1145/2602576.2602581},
}
Figure