CratesBump: A Version-Bump Dataset for the Rust Ecosystem

Muhammad HassnainarXiv 2026 · arXiv

TL;DR

CratesBump is the first large-scale, public dataset designed to study and predict version bumps in the Rust ecosystem. By analyzing over 2 million release pairs from 220,000 crates, the authors provide a rich feature set for machine learning and reveal that nearly 15% of "minor" Rust releases actually contain breaking API changes.

What Problem Does This Paper Address?

In modern software development, we rely heavily on Semantic Versioning (SemVer) to decide if a dependency update is "safe." A major version bump implies breaking changes, while minor and patch bumps should be backward compatible. However, maintainers are human; they make mistakes, or they might not realize a change is breaking.

While researchers have built datasets to study this in Java (Maven) and JavaScript (npm), the Rust ecosystem—with its unique "Cargo" versioning rules—lacked a comprehensive dataset. Specifically, Cargo treats pre-1.0 versions (e.g., 0.4.5) differently than the standard SemVer spec, considering a change in the leftmost non-zero digit (e.g., 0.4.5 to 0.5.0) as a major, breaking bump. This paper fills that gap, providing the data needed to build "semantic version calculators" that can automatically flag potentially mislabeled releases.

Methods

The authors developed an open-source pipeline called SEMVER-EXTRACT to build the dataset. The process involves several sophisticated steps:

  1. Data Collection: It pulls metadata from the official crates.io dump and downloads source code tarballs for every release.
  2. Feature Extraction: For every consecutive pair of releases, the tool computes 41 features. This includes AST diffs (using the syn crate to count changes in functions, structs, and blocks), dependency changes in Cargo.toml, and code complexity metrics.
  3. Social Signal Integration: It clones the crate’s repository and uses the GitHub API to pull commit history, issue activity, and PR data between the two release timestamps.
  4. Labeling: Each pair is given two labels: a "literal" SemVer reading and a "Cargo" label that follows Rust's specific breaking-change rules for 0.y.z versions.

![Fig.

  1. How CratesBump is built (1–4) and used in Section V (5).](/figures/art_c5581cd8458b417599a1ed33bcfd778c)

Datasets

The primary contribution is CratesBump, which contains:

  • 2,079,790 release pairs from 219,955 crates.
  • 41 feature columns categorized into code changes, dependencies, complexity, timing, and GitHub activity.
  • A snapshot of the ecosystem as of September 2026.

The authors also provide the semver-extract tool, allowing researchers to regenerate or update the dataset as the ecosystem grows.

Extracted table from page 3 showing the 48 columns of CratesBump.

Key Insights and Ideas

  • The Pre-1.0 Dominance: A staggering 84% of crates in the dataset have never reached version 1.0. Because Cargo treats 0.y.z bumps as breaking changes, the "literal" versioning and "Cargo" versioning disagree on 82.3% of all pairs.
  • AST Diffs are King: When training models to predict version bumps, features derived from Abstract Syntax Tree (AST) diffs provided the strongest signal. GitHub activity (like the number of open issues) was the weakest predictor.
  • Cargo’s Shielding Effect: By default, Cargo uses "caret" requirements that prevent users from automatically pulling in 0.y.z breaking updates. The authors found that even though many breaking releases exist in the 0.x range, Cargo's rules successfully shield 98.5% of dependents from accidentally adopting them.
  • The "Liar" Releases: An audit using cargo-semver-checks found that many releases are mislabeled. Specifically, 14.9% of minor releases and 8.7% of patch releases actually broke the public API.
Table showing version change disagreements between literal and Cargo labels.

Key Results

The authors benchmarked several machine learning models to see if they could predict the "Cargo" bump label based on the 41 features.

  • Predictability: Tree-based ensembles (like Random Forest and XGBoost) performed best, achieving a Macro-F1 score of ~0.46. While this is significantly better than the random baseline (0.24), it suggests that predicting version bumps is a non-trivial task that is far from "solved."
  • Hardest Class: Patch releases were the most difficult to identify correctly (F1 score of ~0.17), likely because the line between a "bug fix" and a "minor feature" is often blurry in the code itself.
Table III: Predicting the Cargo bump label across various models.

Limitations and Future Work

  • Declared vs. Actual: The dataset labels are based on what maintainers declared, not necessarily what they did. While the authors audited a sample, the full dataset doesn't yet have a "ground truth" label for every single API break.
  • GitHub Dependency: About 16% of pairs lack GitHub features because the crates are hosted elsewhere (like GitLab) or the repositories were deleted/private.
  • Future Directions: The authors suggest that future versions of the dataset could include labels from automated tools like cargo-semver-checks for every buildable pair, as well as signals from test suites and changelogs to capture behavioral changes that AST diffs might miss.

Summary updated

Abstract

We release CratesBump, the first public dataset for predicting version bumps in the Rust ecosystem, together with semver-extract, the tool that builds it. CratesBump is built from every crate on crates$.$io with at least two releases before 14 September 2026 (about 220,000 crates), giving over two million consecutive release pairs. Each pair has 41 features (code changes from AST diffs, dependency changes, complexity, release cadence, GitHub commit and issue activity), its source repository where listed, and two labels: the literal reading of MAJOR$.$MINOR$.$PATCH, and Cargo's rule, under which a 0.y bump is a breaking change. semver-extract regenerates the dataset from public crates$.$io data and the GitHub API. To illustrate its use we (i) benchmark bump predictors on held-out repositories, finding the task learnable but far from solved, with lower scores on later releases; (ii) measure which feature groups carry the signal; (iii) check, against declared dependency ranges, how often Cargo's rule shields dependents from pre-1.0 breaking releases; and (iv) audit a sample of releases with cargo-semver-checks, finding minor and patch releases that break the public API despite their version number. Dataset and tool are released under open licenses.
BibTeX
@misc{hassnain2026cratesbump,
  author = {Muhammad Hassnain},
  title = {{CratesBump: A Version-Bump Dataset for the Rust Ecosystem}},
  year = {2026},
  eprint = {2611.23664},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23664},
}
Figure