Understanding myths and realities of test-suite evolution

Saurabh Sinha, Leandro Sales Pinto, Saurabh Sinha 0003, Alessandro OrsoSigsoft fse 2012 · DOI

TL;DR

This paper dives deep into how unit test suites evolve over time, revealing that test repair is only a small piece of the puzzle. The authors introduce a tool and methodology to systematically study this evolution, finding that refactoring, deletion, and addition of tests are far more common than fixing broken tests, and that many "deleted" tests are actually moved or renamed.

What Problem Does This Paper Address?

As software evolves, so do its test suites. While automated test repair is a hot research area, its effectiveness hinges on understanding how tests actually change in practice. Without this knowledge, researchers risk developing repair techniques that are too narrow or don't address the real-world challenges faced by developers. This paper aims to fill this gap by providing a comprehensive empirical study of test-suite evolution, specifically focusing on unit tests. They investigate the types, frequencies, and reasons behind test modifications, deletions, and additions, aiming to guide future research in test maintenance and repair.

Methods

The core of this research lies in a novel technique and a tool called TESTEVOL. This tool analyzes pairs of program versions and their corresponding JUnit test suites. It employs a combination of static and dynamic analysis to:

  • Compile each system version and create separate JAR files for application code and test code.
  • Execute tests against their respective program versions and also cross-execute (old tests on new code, new tests on old code) to detect failures.
  • Classify test outcomes into four categories: Pass, FailCE (Compilation Error), FailRE (Runtime Exception), and FailAE (Assertion Violation).
  • Identify differences between test suites (modified, deleted, added tests) using the WALA analysis infrastructure.
  • Classify these changes into eight distinct evolution categories (e.g., TESTREP for repaired tests, TESTDEL(CE) for obsolete tests, TESTADD(P) for coverage augmentation).
  • Analyze the nature of modifications within repaired tests, categorizing them by changes to method calls (added, deleted, modified parameters) and assertions (added, deleted, modified values).
  • Leverage a coverage analyzer to understand the impact of deleted or added tests on branch coverage.

Datasets

The study was conducted on six real-world, open-source Java programs with JUnit test suites: Commons Lang, Commons Math, Gson, JFreeChart, JodaTime, and PMD. For PMD, two sets of versions were analyzed: official releases and a set of 17 intermediate builds to assess the impact of version granularity. In total, the study analyzed 88 program versions, 14,312 tests, and 17,427 test changes across these programs. No new benchmarks were introduced; instead, existing software projects and their version histories were used.

Key Insights and Ideas

  • Test repair is a small part of the evolution story: While automated test repair is important, it's not the primary driver of test suite changes. Most modifications are for reasons other than fixing broken tests.
  • "Deleted" tests are often moved or renamed: A significant portion of tests categorized as "deleted" don't actually disappear; they are simply refactored into different classes or given new names. This is evidenced by their lack of impact on branch coverage.
  • Failing tests are usually obsolete, not just hard to fix: Tests that fail on new program versions are typically removed because the functionality they test no longer exists or has changed significantly, rather than because the fix is too complex.
  • Tests are added for more than just bug fixes and new features: Developers also add tests to validate changes made to existing code, suggesting a role for test augmentation beyond traditional bug fixing.
  • Assertion-focused repair techniques may be limited: Since many test repairs involve changes to method calls rather than just assertions, techniques solely focused on assertions might not be broadly applicable.

Key Results

  • Out of 17,427 observed test changes, 5,111 (29%) were modifications, 2,541 (15%) were deletions, and 9,775 (56%) were additions.
  • Actual test repairs (modifications to fix broken tests) accounted for only 6.4% of all test changes and 22% of all test modifications.
  • Non-repair test modifications occurred nearly four times as frequently as test repairs.
  • Less than 10% of test repairs involved modifications to existing assertions only; the majority required more complex changes, often involving method calls.
  • Tests failing with compilation errors (FailCE) were the most common reason for deletion, indicating obsolescence.
  • For tests that passed on the new version but were deleted (TESTDEL(P)), over 75% showed no reduction in branch coverage, suggesting they were moved or renamed.
  • For added tests, 69% likely validated newly added code (failed to compile on the old version), while 14% were added to validate bug fixes.

Limitations and Future Work

The authors acknowledge threats to the external validity of their findings, noting that results might differ for other programs or test suites, or if evolution is studied at a finer grain (e.g., individual commits vs. releases). Preliminary analysis of finer-grained data for PMD showed variations, suggesting that release-level analysis might miss multiple repairs between releases.

Future work suggested includes:

  • Extending the empirical study with more programs and analysis techniques, particularly code-clone detection to better distinguish true deletions from moves/renames.
  • Developing intent-preserving repair techniques that ensure repaired tests maintain the original intent behind the test.
  • Investigating automated API migration and adaptation techniques for fixing tests broken by API changes.
  • Exploring the use of constraint-based synthesis for generating method sequences to satisfy failing assertions.
  • Conducting similar empirical studies for GUI test scripts to assess the applicability of current findings.

Summary updated

BibTeX
@inproceedings{sinha2012understanding,
  author = {Saurabh Sinha and Leandro Sales Pinto and Saurabh Sinha and Alessandro Orso},
  title = {{Understanding myths and realities of test-suite evolution}},
  booktitle = {Sigsoft fse},
  year = {2012},
  doi = {10.1145/2393596.2393634},
}
Figure