A Large-Scale Empirical Study on Semantic Versioning in Golang Ecosystem

Wenke Li, Feng Wu, Cai Fu, Fan Zhou

TL;DR

This paper introduces GoSVI, a novel tool and the first large-scale dataset for analyzing Semantic Versioning (SemVer) compliance in the Go ecosystem. They found that while most Go library upgrades follow SemVer, a significant portion of non-major upgrades still introduce breaking changes, impacting a third of downstream client programs.

What Problem Does This Paper Address?

Modern software development heavily relies on third-party libraries (TPLs) to accelerate development. However, upgrading these TPLs can introduce "breaking changes" – modifications that are incompatible with existing client code, leading to compilation errors or runtime issues. Semantic Versioning (SemVer) aims to standardize versioning to signal compatibility (e.g., major version bumps for breaking changes, minor/patch for compatible ones). Despite SemVer's widespread adoption, not all libraries adhere to it strictly. While studies exist for ecosystems like Java and JavaScript, the Go ecosystem, which is rapidly growing in popularity, lacked comprehensive analysis and tools to detect these breaking changes and their impact. This gap leaves Go developers uncertain about library upgrades, potentially hindering progress or introducing unexpected bugs.

Methods

The authors developed GoSVI (Go Semantic Versioning Insight), a tool designed to automatically detect breaking changes in Go modules and analyze their impact on client programs. The process involves:

  1. Extracting exported objects (interfaces, functions, structs) from TPLs.
  2. Utilizing and enhancing the go apidiff tool to identify differences between two versions of a TPL. They optimized go apidiff to provide more detailed reports on breaking changes, including module, package, type, condition, and message.
  3. Parsing client program Abstract Syntax Trees (ASTs) to resolve identifiers and determine their types.
  4. Comparing the types of breaking changes in TPLs with the types of identifiers used in client programs to assess impact.

They also constructed a large-scale dataset by crawling GitHub repositories, filtering for quality, and extracting dependency graphs using go.mod files, storing this information in a Neo4j graph database.

Datasets

The paper introduces a novel, large-scale dataset for the Go ecosystem:

  • 124K Third-Party Libraries (TPLs) and their versions.
  • 532K Client Programs that depend on these TPLs.
  • This dataset was constructed by crawling GitHub repositories with more than five stars, cleaning invalid/duplicate entries, and extracting dependency graphs.

Key Insights and Ideas

  • SemVer is a work in progress in Go: While many Go library upgrades claim to follow SemVer, a significant portion of non-major upgrades (which shouldn't break things) still do. This suggests a need for better developer education or tooling.
  • Breaking changes have a disproportionate impact: Even though only a small percentage of identified breaking changes are used by client programs, a much larger fraction of client programs are affected. This highlights the cascading effect of dependencies and the importance of understanding the broader ecosystem impact.
  • Go's module system is relatively new: The study period starts from 2018/09, coinciding with the introduction of Go modules. This means the analysis captures the early adoption and evolution of SemVer practices within this newer module-based system.
  • "Remove" is the most common breaking action: Across all breaking changes, removing code elements (like functions or types) is the most frequent culprit. This is a key area for developers to watch out for.

Key Results

  • 86.3% of library upgrades follow SemVer compliance.
  • 28.6% of non-major upgrades (minor and patch) introduce breaking changes.
  • "Remove in function type" is the most common breaking change condition (36.2%).
  • The tendency to comply with SemVer has significantly improved over time, from 63.7% in 2018/09 to 92.2% in 2023/03.
  • Only 3.1% of breaking changes are directly used by client programs, but 33.3% of downstream client programs may be affected by these changes.
  • The most common breaking changes affecting client programs are "add interface method in interface type" (45.4%) and "value change in basic (const) type" (37.2%).

Limitations and Future Work

  • Internal Validity:
    • The dataset collection starts from 2018/09, missing SemVer's earlier history (since 2013).
  • The filtering of certain directories (like /cmd, /internal) might miss some relevant breaking changes if developers don't strictly adhere to the standard Go project layout.
  • The accuracy of go apidiff was verified on a small sample (91% precision), but a comprehensive benchmark is lacking.
  • External Validity:
    • The study relies on GitHub repositories, which may not perfectly represent the entire Go ecosystem.
  • Future Work:
    • Detect semantic breaking changes (beyond syntactic ones) in the Go ecosystem.
  • Compare breaking change patterns across different package ecosystems (e.g., Npm, Maven) to gain broader insights.

Summary updated

Abstract

Third-party libraries (TPLs) have become an essential component of software, accelerating development and reducing maintenance costs. However, breaking changes often occur during the upgrades of TPLs and prevent client programs from moving forward. Semantic versioning (SemVer) has been applied to standardize the versions of releases according to compatibility, but not all releases follow SemVer compliance. Lots of work focuses on SemVer compliance in ecosystems such as Java and JavaScript beyond Golang (Go for short). Due to the lack of tools to detect breaking changes and dataset for Go, developers of TPLs do not know if breaking changes occur and affect client programs, and developers of client programs may hesitate to upgrade dependencies in terms of breaking changes. To bridge this gap, we conduct the first large-scale empirical study in the Go ecosystem to study SemVer compliance in terms of breaking changes and their impact. In detail, we propose GoSVI (Go Semantic Versioning Insight) to detect breaking changes and analyze their impact by resolving identifiers in client programs and comparing their types with breaking changes. Moreover, we collect the first large-scale Go dataset with a dependency graph from GitHub, including 124K TPLs and 532K client programs. Based on the dataset, our results show that 86.3% of library upgrades follow SemVer compliance and 28.6% of no-major upgrades introduce breaking changes. Furthermore, the tendency to comply with SemVer has improved over time from 63.7% in 2018/09 to 92.2% in 2023/03. Finally, we find 33.3% of downstream client programs may be affected by breaking changes. These findings provide developers and users of TPLs with valuable insights to help make decisions related to SemVer.
BibTeX
@misc{li2023large,
  author = {Wenke Li and Feng Wu and Cai Fu and Fan Zhou},
  title = {{A Large-Scale Empirical Study on Semantic Versioning in Golang Ecosystem}},
  year = {2023},
  eprint = {2309.02894},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.02894},
}

Loading related content...

Figure