ConTest: A Unit Test Completion Benchmark featuring Context

Johannes Villmow, Jonas Depoix, Adrian UlgesACL 2021

TL;DR

This paper introduces CONTEST, a large-scale benchmark for automatically completing unit tests by predicting assertion statements. They demonstrate that including surrounding "context" code significantly improves prediction accuracy, and their novel benchmark is crucial for advancing this NLP-for-code research area.

What Problem Does This Paper Address?

Writing comprehensive unit tests is a time-consuming and often tedious task for software developers. While automated test generation tools exist, they often produce tests that are hard to read and, more importantly, lack "meaningful" assertions that truly verify the expected behavior of the code. This paper focuses on the sub-problem of unit test completion, specifically generating the assertion statements given the test's setup code and the method being tested (the "focal method"). Existing approaches often struggle with this part because it requires a deep understanding of the target method's functionality, which is difficult to infer without additional context.

Methods

The authors introduce CONTEST, a new benchmark for unit test completion. Their approach involves:

  • Data Collection: Scraping JUnit projects from GitHub.
  • Focal Method Identification: Using fuzzy string matching and token similarity to link test methods (e.g., testInc()) to their corresponding focal methods (e.g., inc()).
  • Code Splitting: Dividing test code into "setup code" (executed before assertions) and "assertion code."
  • Context Resolution: Crucially, they identify and include "context code." This includes methods called within the test setup and methods called by the focal method itself, recursively. This context is argued to be vital for understanding the target method.
  • Model Architecture: They employ a standard transformer encoder-decoder architecture.
  • Input Representation: They experiment with two ways of encoding the source code:
  • Linearized Abstract Syntax Trees (ASTs): Representing the code's structure as a tree, then linearizing it for the transformer.
  • Tokenized Source Code: A more standard approach using Byte Pair Encoding (BPE) on raw tokens.

Datasets

The paper introduces CONTEST, a new benchmark for unit test completion.

  • Size: It contains 365,450 data points.
  • Content: Each data point includes the test method's setup code, the focal method, and crucially, context methods called within either.
  • Relation to Existing Datasets: CONTEST is significantly larger than previous datasets like ATLAS (which has ~158k tests and lacks context). CONTEST also provides a project-based split, which is important for evaluating generalization.

Key Insights and Ideas

  • Context is King (for Assertions): The most significant finding is that including context code (methods called by the setup or focal method) dramatically improves the accuracy of predicting assertions. This highlights that understanding the ripple effects of method calls is essential for generating meaningful checks.
  • ASTs Provide Syntactic Advantage: Using linearized Abstract Syntax Trees (ASTs) as input to the transformer yields a notable improvement in BLEU scores compared to simply using tokenized source code. This suggests that explicitly encoding the code's structure is beneficial.
  • Generalization is Hard: Models struggle to generalize knowledge learned from one project to another, especially when using a project-based split. This indicates that current models might be overfitting to project-specific coding styles or idioms.
  • Unparsable Code is a Minor Issue: Despite the complexity of generating code, their models produce unparsable code in only 1.92% of cases, suggesting that generating syntactically valid code is largely achievable.

Key Results

  • Their core model, using linearized ASTs and context, achieved a BLEU score of 38.2 (adjusted for unparsable code).
  • Including context code improved BLEU scores by 10.1 points compared to a baseline without context.
  • Using ASTs improved BLEU scores by 4.1 points compared to a token-only baseline.
  • Models trained on a project-based split showed a significant drop in BLEU score to 15.4, highlighting the generalization challenge.

Limitations and Future Work

  • Generalization Across Projects: The paper acknowledges that their models struggle to generalize to unseen projects. This is identified as a major challenge and a key area for future research. They suggest that fine-tuning large pre-trained language models for code might be a promising direction.
  • Project-Based Splits as a Standard: The authors advocate for project-based splits as the "golden standard" for evaluating code models to prevent test leakage and ensure true generalization.
  • Unfiltered Dataset: While they release a filtered dataset of 365,450 samples, an unfiltered version with more data points is available upon request, which could be used for further research.

Summary updated

BibTeX
@inproceedings{villmow2021contest,
  author = {Johannes Villmow and Jonas Depoix and Adrian Ulges},
  title = {{ConTest: A Unit Test Completion Benchmark featuring Context}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure