A Longitudinal Study of Static Analysis Warning Evolution and the Effects of PMD on Software Quality in Apache Open Source Projects

Alexander Trautsch, Steffen Herbold, Jens Grabowski

Introduction

Automated Static Analysis Tools (ASATs), such as PMD, Checkstyle, and FindBugs, are staples in modern software development. By scanning source code for potential bugs, "code smells," and style violations without executing the program, these tools aim to improve maintainability and reduce defects. Despite their widespread adoption, empirical evidence regarding their long-term impact on software evolution remains sparse. This study, originally published in Empirical Software Engineering, presents a longitudinal analysis of nearly two decades of code history across 54 Apache Software Foundation projects to determine how static analysis warnings evolve and whether the integration of these tools into the build process yields tangible improvements in software quality.

Methodology

The researchers utilized the SmartSHARK infrastructure to perform a large-scale retrospective analysis. The study's scope included:

  • Scale: 112,266 commits spanning up to 17 years of history.
  • Dataset: A convenience sample of 54 mature, Java-based Apache projects (e.g., commons-math, struts, jspwiki) that utilize Maven and active issue trackers.
  • Tooling: The study focused on PMD, applying 193 rules across 61 different PMD releases. This "time-aware" approach ensured that code was only evaluated against rules that existed at the time of the commit.
  • Metrics: The primary metric for analysis was Warning Density, defined as the number of warnings per 1,000 Logical Lines of Code (kLLoC). To assess external quality, the researchers modeled Defect Density as a proxy.
  • Statistical Analysis: Linear regression was used to identify longitudinal trends, while the Mann-Whitney-U test compared projects with PMD integrated into their build process against those without.

Evolution of Static Analysis Warnings

The analysis revealed a general trend of improvement across the Apache ecosystem. In 43 out of the 54 projects studied, warning density declined over time, with an average removal rate of 3.5 warnings per kLLoC per year.

Interestingly, the study found that the vast majority of warning removals pertained to coding style—specifically naming conventions and the use of braces. This suggests that the primary practical application of ASATs in these projects is the standardization of code appearance. Furthermore, "minor" and "major" warnings were resolved at a significantly higher rate (37% reduction) than "critical" warnings (27%). The researchers hypothesize that critical warnings may be ignored more frequently because they are often associated with complex logic and are more prone to being perceived as false positives.

The Impact of Build Process Integration

A central question of the study was whether formally integrating PMD into the build configuration (e.g., via a Maven pom.xml) changed developer behavior. The results presented a nuanced "Build Process Paradox":

  1. Warning Removal Rates: Surprisingly, the presence of PMD in the build process did not lead to a statistically significant increase in the rate at which warnings were fixed. Developers appeared to address warnings at a consistent pace regardless of whether the tool was an "official" part of the build.
  2. Long-term Trends: Despite the lack of change in the rate of removal, 19 out of 24 projects showed a positive long-term impact on overall warning density following the introduction of PMD into their build files.

Correlation with Software Quality

The most significant finding of the research relates to the correlation between tool usage and defect density. Years in which PMD was integrated into the project's build process exhibited a lower defect density compared to years without such integration. Specifically, the median reduction was approximately 0.78 fewer defects per 1,000 LLoC.

This suggests that while developers may not be obsessively clearing every style warning flagged by the tool, the mere presence of an ASAT in the formal workflow may foster a culture of "cleaner" coding or provide enough ambient feedback to prevent the introduction of actual bugs.

Limitations and Conclusion

While the study provides robust evidence for the benefits of ASATs, several limitations apply. The focus on a single tool (PMD) may not generalize to bytecode-based tools like FindBugs. Additionally, the study cannot account for developers using ASATs locally within their IDEs without committing the configuration to the repository. Finally, the relationship between PMD usage and lower defects is correlational; it remains possible that higher-quality development teams are simply more predisposed to adopting static analysis tools.

In conclusion, this 17-year longitudinal study demonstrates that static analysis warnings generally decrease as projects mature. More importantly, the formal integration of tools like PMD into the build process is associated with a measurable reduction in software defects, highlighting the value of ASATs as a component of a comprehensive quality assurance strategy. Future research should explore effort-aware models to help developers prioritize the most impactful warnings.

Summary updated

Abstract

Automated static analysis tools (ASATs) have become a major part of the software development workflow. Acting on the generated warnings, i.e., changing the code indicated in the warning, should be part of, at latest, the code review phase. Despite this being a best practice in software development, there is still a lack of empirical research regarding the usage of ASATs in the wild. In this work, we want to study ASAT warning trends in software via the example of PMD as an ASAT and its usage in open source projects. We analyzed the commit history of 54 projects (with 112,266 commits in total), taking into account 193 PMD rules and 61 PMD releases. We investigate trends of ASAT warnings over up to 17 years for the selected study subjects regarding changes of warning types, short and long term impact of ASAT use, and changes in warning severities. We found that large global changes in ASAT warnings are mostly due to coding style changes regarding braces and naming conventions. We also found that, surprisingly, the influence of the presence of PMD in the build process of the project on warning removal trends for the number of warnings per lines of code is small and not statistically significant. Regardless, if we consider defect density as a proxy for external quality, we see a positive effect if PMD is present in the build configuration of our study subjects.
BibTeX
@inproceedings{trautsch2021longitudinal,
  author = {Alexander Trautsch and Steffen Herbold and Jens Grabowski},
  title = {{A Longitudinal Study of Static Analysis Warning Evolution and the Effects of PMD on Software Quality in Apache Open Source Projects}},
  booktitle = {Software Engineering},
  year = {2021},
  url = {https://openreview.net/forum?id=JFCRaJO5sam},
}
Figure