Can Coding Agents Write and Transpile Unit Tests

Gil Pasternak, Aj Langley, Tom Yotam, Ellie KitanidisarXiv 2026 · arXiv

TL;DR

While modern coding agents like Claude Code and Cursor can flood a codebase with tests to achieve impressive coverage numbers, their performance is often a case of quantity over quality. This paper reveals that agent-generated tests struggle to scale with project size and are surprisingly unreliable when translating (transpiling) existing test suites between programming languages.

What Problem Does This Paper Address?

As Large Language Models (LLMs) move from writing snippets to managing entire codebases, we increasingly rely on them to "modernize" legacy code—for example, porting an old C++ library to Rust. We usually trust these transformations because the code passes its unit tests. However, in many real-world scenarios, those tests are also written by the AI.

The authors set out to answer a critical question: Can we actually trust these agents to write and translate reliable test suites? While prior research showed LLMs are decent at writing tests for isolated functions, this paper investigates their performance at the scale of full, complex repositories across multiple languages (Python, C++, and Rust).

Methods

The researchers benchmarked three heavy hitters: Claude Code (using Claude-Opus-5), Codex (using GPT-5.6-sol), and Cursor (using Grok-4.6). They evaluated these agents on two main tasks:

  1. Generation: Writing a test suite from scratch for an existing project.
  2. Transpilation: Translating an existing C++ test suite into Rust.

To go beyond simple "did it pass?" metrics, they employed a rigorous evaluation framework:

  • Mutation Scoring: They intentionally injected small bugs ("mutants") into the code to see if the AI's tests were strong enough to catch them.
  • Vacuous Test Rate: They measured how often tests passed even after the function body was completely deleted—a sign of a "shallow" test.
  • LLM-as-Judge: They used a separate model (Kimi K3) to identify critical behaviors and edge cases for each project, then checked if the agent's tests actually covered them.
  • Trace Analysis: They manually inspected 675 "reasoning traces" to understand the internal logic (or lack thereof) behind the agents' failures.

Datasets

The study utilized a newly assembled corpus of 240 open-source GitHub projects across Python, C++, and Rust.

  • The projects were stratified by size: Small (0-1k lines), Medium (1k-5k), and Large (5k-10k).
  • All projects were anonymized (renaming variables, removing docs) to ensure the models weren't just reciting memorized tests from their training data.
  • A subset of 50 C++ projects was specifically prepared for the transpilation experiment.

Key Insights and Ideas

  • The Brute Force Illusion: Agents often beat human-written suites on aggregate coverage metrics, but they do so through sheer volume. For instance, Claude Code writes roughly seven times as many tests as a human but covers significantly fewer lines of code per individual test.
  • The Scale Wall: Agent performance isn't linear. As projects get larger and more complex, the quality of agent-generated tests decays much faster than human-written ones. On large projects, the AI's "lead" over humans almost entirely vanishes.
  • Transpilation is Brittle: Translating tests (e.g., C++ to Rust) is much harder for agents than writing them from scratch. Agents often produce "unfaithful" translations that pass on the new code but fail to capture the semantic edge cases of the original suite.
  • Feedback Blindness: In over 55% of the analyzed reasoning traces, the agents never even attempted to measure their own code coverage. They essentially "flew blind," repairing tests until they passed without checking if they were actually testing the important parts of the code.

Key Results

The agents generally outperformed the human baseline in Mutation Scores (the ability to catch injected bugs), as seen in Figure 2a. Claude led the pack with a score of 0.82, followed by Cursor (0.78) and Codex (0.72), all surpassing the human average of 0.63.

Mutation score by author; all agents exceed human baseline.

However, Figure 3a highlights the efficiency gap: humans are far more "economical" authors. A single human test covers an average of 8.5 production lines, while a Claude test covers only 2.2.

Production lines covered per test written: Codex is the most economical; Claude uses volume to reach coverage.

In the transpilation task, the results were sobering. Agents underperformed the original source suite in edge-case coverage 58% of the time. In the worst cases, the "translated" suite saw a 70% drop in its ability to catch bugs compared to the original C++ version.

Limitations and Future Work

The authors acknowledge several limitations:

  • Scope: The study focused on open-source projects under 10k lines; enterprise-scale codebases (which are often much larger and messier) might pose even greater challenges.
  • Harness vs. Model: The study tested agents in their "native" environments, making it hard to tell if a failure was due to the underlying LLM or the specific tools (the "harness") provided by the agent framework.
  • Trace Narrowness: The deep dive into reasoning traces was only performed on Codex.

Future work should focus on building "tighter" loops where agents are forced to use coverage and behavioral verifiers to validate their own work, rather than relying on simple pass/fail feedback.

Summary updated

Abstract

Coding assistants built on Large Language Models (LLMs) have become a staple of modern software development, increasingly trusted to execute workflows and modernize code end-to-end. Much of this trust rests on passing tests, which are often written by the coding agents themselves. In this work, we examine the ability of these agents to both write reliable tests and translate them between programming languages. We benchmark three popular coding agents, each evaluated with a native frontier model and high reasoning effort. Verifying across a broad suite of metrics, we find that coding agents far exceed human written suites on traditional measures of coverage. This advantage, however, is uneven: agent test quality degrades sharply with project size, and much of the advantage lies in a greater number of written tests. We also show that agentic test translation---while seemingly strong in aggregate---remains inconsistent and unreliable.
BibTeX
@misc{pasternak2026can,
  author = {Gil Pasternak and AJ Langley and Tom Yotam and Ellie Kitanidis},
  title = {{Can Coding Agents Write and Transpile Unit Tests}},
  year = {2026},
  eprint = {2611.21797},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21797},
}
Figure