TL;DR
This paper introduces GoSVI, a novel tool and the first large-scale dataset for analyzing Semantic Versioning (SemVer) compliance in the Go ecosystem. They found that while most Go library upgrades follow SemVer, a significant portion of non-major upgrades still introduce breaking changes, impacting a third of downstream client programs.
What Problem Does This Paper Address?
Modern software development heavily relies on third-party libraries (TPLs) to accelerate development. However, upgrading these TPLs can introduce "breaking changes" – modifications that are incompatible with existing client code, leading to compilation errors or runtime issues. Semantic Versioning (SemVer) aims to standardize versioning to signal compatibility (e.g., major version bumps for breaking changes, minor/patch for compatible ones). Despite SemVer's widespread adoption, not all libraries adhere to it strictly. While studies exist for ecosystems like Java and JavaScript, the Go ecosystem, which is rapidly growing in popularity, lacked comprehensive analysis and tools to detect these breaking changes and their impact. This gap leaves Go developers uncertain about library upgrades, potentially hindering progress or introducing unexpected bugs.
Methods
The authors developed GoSVI (Go Semantic Versioning Insight), a tool designed to automatically detect breaking changes in Go modules and analyze their impact on client programs. The process involves:
- Extracting exported objects (interfaces, functions, structs) from TPLs.
- Utilizing and enhancing the
go apidifftool to identify differences between two versions of a TPL. They optimizedgo apidiffto provide more detailed reports on breaking changes, including module, package, type, condition, and message. - Parsing client program Abstract Syntax Trees (ASTs) to resolve identifiers and determine their types.
- Comparing the types of breaking changes in TPLs with the types of identifiers used in client programs to assess impact.
They also constructed a large-scale dataset by crawling GitHub repositories, filtering for quality, and extracting dependency graphs using go.mod files, storing this information in a Neo4j graph database.
Datasets
The paper introduces a novel, large-scale dataset for the Go ecosystem:
- 124K Third-Party Libraries (TPLs) and their versions.
- 532K Client Programs that depend on these TPLs.
- This dataset was constructed by crawling GitHub repositories with more than five stars, cleaning invalid/duplicate entries, and extracting dependency graphs.
Key Insights and Ideas
- SemVer is a work in progress in Go: While many Go library upgrades claim to follow SemVer, a significant portion of non-major upgrades (which shouldn't break things) still do. This suggests a need for better developer education or tooling.
- Breaking changes have a disproportionate impact: Even though only a small percentage of identified breaking changes are used by client programs, a much larger fraction of client programs are affected. This highlights the cascading effect of dependencies and the importance of understanding the broader ecosystem impact.
- Go's module system is relatively new: The study period starts from 2018/09, coinciding with the introduction of Go modules. This means the analysis captures the early adoption and evolution of SemVer practices within this newer module-based system.
- "Remove" is the most common breaking action: Across all breaking changes, removing code elements (like functions or types) is the most frequent culprit. This is a key area for developers to watch out for.
Key Results
- 86.3% of library upgrades follow SemVer compliance.
- 28.6% of non-major upgrades (minor and patch) introduce breaking changes.
- "Remove in function type" is the most common breaking change condition (36.2%).
- The tendency to comply with SemVer has significantly improved over time, from 63.7% in 2018/09 to 92.2% in 2023/03.
- Only 3.1% of breaking changes are directly used by client programs, but 33.3% of downstream client programs may be affected by these changes.
- The most common breaking changes affecting client programs are "add interface method in interface type" (45.4%) and "value change in basic (const) type" (37.2%).
Limitations and Future Work
- Internal Validity:
- The dataset collection starts from 2018/09, missing SemVer's earlier history (since 2013).
- The filtering of certain directories (like
/cmd,/internal) might miss some relevant breaking changes if developers don't strictly adhere to the standard Go project layout. - The accuracy of
go apidiffwas verified on a small sample (91% precision), but a comprehensive benchmark is lacking. - External Validity:
- The study relies on GitHub repositories, which may not perfectly represent the entire Go ecosystem.
- Future Work:
- Detect semantic breaking changes (beyond syntactic ones) in the Go ecosystem.
- Compare breaking change patterns across different package ecosystems (e.g., Npm, Maven) to gain broader insights.
Summary updated