Towards Better Comprehension of Breaking Changes in the NPM Ecosystem

Dezhen Kong, Jiakun Liu, Lingfeng Bao, David Lo

Abstract: Breaking changes cause a lot of effort to both downstream and upstream developers: downstream developers need to adapt to breaking changes and upstream developers are responsible for identifying and documenting them. In the NPM ecosystem, characterized by frequent code changes and a high tolerance for making breaking changes, the effort is larger. For better comprehension of breaking changes in the NPM ecosystem and to enhance breaking change detection tools, we conduct a large-scale empirical study to investigate breaking changes in the NPM ecosystem. We construct a dataset of explicitly documented breaking changes from 381 popular NPM projects. We find that 95.4% of the detected breaking changes can be covered by developers' documentation, and about 19% of the breaking changes cannot be detected by regression testing. Then in the process of investigating source code of our collected breaking changes, we yield a taxonomy of JavaScript and TypeScript-specific syntactic breaking changes and a taxonomy of major types of behavioral breaking changes. Additionally, we investigate the reasons why developers make breaking changes in NPM and find three major reasons, i.e., to reduce code redundancy, to improve identifier name, and to improve API design, and each category contains several sub-items. We provide actionable implications for future research, e.g., automatic naming and renaming techniques should be applied in JavaScript projects to improve identifier names, future research can try to detect more types of behavioral breaking changes. By presenting the implications, we also discuss the weakness of automatic renaming and breaking change detection approaches.

Introduction

In the fast-paced world of the NPM (Node Package Manager) ecosystem, code evolves rapidly. While updates bring new features and bug fixes, they often introduce Breaking Changes (BCs)—modifications to an API that cause dependent projects to fail. The challenge is twofold: existing detection tools are often designed for statically typed languages like Java, missing the dynamic nuances of JavaScript and TypeScript, and there is a lack of deep understanding regarding why developers choose to break their APIs.

Methodology

The researchers performed a multi-stage empirical study starting with 35,000 popular NPM projects, randomly sampling 381 for deep analysis. They mined over 5,000 commits tagged with "BREAKING CHANGE" using the Conventional Commits standard. A subset of 1,519 commits was manually inspected to categorize the type of change and the underlying motivation. Furthermore, the team performed regression testing on 16,371 commits to evaluate how many documented breaks were actually caught by existing test suites.

Datasets

The study introduces a high-quality, manually labeled dataset of 1,519 breaking change commits across 131 distinct NPM projects. These projects span six categories: Frontend, Web API, Database Tools, Development Tools, Plugins, and Utilities. The dataset includes commit messages, source code diffs, categorized BC types, and developer motivations, providing a robust benchmark for future research.

Key Insights and Taxonomy

A critical finding is that while developers are diligent about documentation—covering 95.4% of changes detected by tests—nearly 19% of documented breaking changes do not trigger a test failure. This suggests that relying solely on regression testing is insufficient for identifying semantic breaks.

Behavioral changes dominate the NPM landscape, accounting for 68.1% of breaking changes. Unlike syntactic breaks (like removing a method), behavioral breaks allow the code to run but alter its logic, such as changing error handling or default configurations. The researchers developed a taxonomy to classify these changes across various code elements.

Table 1: Taxonomy of breaking change types and the code elements they affect.

Motivations for Breaking Changes

Developers typically break code for three primary reasons:

  1. To reduce redundancy (31%): Cleaning up outdated or duplicated code.
  2. To improve identifier names (7%): Making the code more readable or idiomatic.
  3. To improve API design (62%): Making behaviors more reasonable or convenient for the user.

Interestingly, many renames occur to correct initial poor naming choices, suggesting that technical debt in naming leads directly to future breaking changes.

Conclusion and Future Directions

The study highlights a significant gap in automated detection for behavioral changes and the need for "change-aware" development tools. Future research should focus on applying automatic naming and renaming techniques to JavaScript projects and developing sophisticated semantic analysis tools that can detect logic-based breaking changes that current regression tests miss.

Summary updated

Abstract

Breaking changes cause a lot of effort to both downstream and upstream developers: downstream developers need to adapt to breaking changes and upstream developers are responsible for identifying and documenting them. In the NPM ecosystem, characterized by frequent code changes and a high tolerance for making breaking changes, the effort is larger. For better comprehension of breaking changes in the NPM ecosystem and to enhance breaking change detection tools, we conduct a large-scale empirical study to investigate breaking changes in the NPM ecosystem. We construct a dataset of explicitly documented breaking changes from 381 popular NPM projects. We find that 95.4% of the detected breaking changes can be covered by developers' documentation, and about 19% of the breaking changes cannot be detected by regression testing. Then in the process of investigating source code of our collected breaking changes, we yield a taxonomy of JavaScript and TypeScript-specific syntactic breaking changes and a taxonomy of major types of behavioral breaking changes. Additionally, we investigate the reasons why developers make breaking changes in NPM and find three major reasons, i.e., to reduce code redundancy, to improve identifier name, and to improve API design, and each category contains several sub-items. We provide actionable implications for future research, e.g., automatic naming and renaming techniques should be applied in JavaScript projects to improve identifier names, future research can try to detect more types of behavioral breaking changes. By presenting the implications, we also discuss the weakness of automatic renaming and breaking change detection approaches.
BibTeX
@article{kong2025towards,
  author = {Dezhen Kong and Jiakun Liu and Lingfeng Bao and David Lo},
  title = {{Towards Better Comprehension of Breaking Changes in the NPM Ecosystem}},
  journal = {ACM Trans. Softw. Eng. Methodol.},
  year = {2025},
  doi = {10.1145/3702991},
}
Figure