A Low Similarity Score Does Not Show Independence after Source-Conditioned Repository Rewriting

Micheline Bénédicte Moumoula, Serge Lionel Nikiema, Tailia Malloy, Abdoul Kader Kaboré, Jacques Klein, Tegawendé F. BissyandearXiv 2026 · arXiv

TL;DR

Can an AI "cleanse" copyleft code by rewriting it from scratch? This paper demonstrates that LLM coding agents can produce functionally identical versions of software with extremely low similarity scores, but these scores are highly unstable and fail to prove true independence. A single low score is often just a "lucky draw" from a wide distribution of possible outputs, making it a weak defense for license compliance.

What Problem Does This Paper Address?

In the world of open-source software, "copyleft" licenses (like the GPL) require that any derivative work also be open-source. To move a project to a more permissive license (like MIT), maintainers sometimes claim they have performed a "ground-up rewrite" using an AI agent, citing low similarity scores from tools like JPlag as proof that the new code is independent of the original.

The authors investigate whether these token-similarity scores are actually a reliable "witness" for independent authorship. Specifically, they look at source-conditioned rewriting: a scenario where an AI agent has full access to the original code in its workspace while it generates the "new" version. If an agent with full access can easily produce a low-similarity score, then a low score cannot be used to prove the agent didn't rely on the original.

Methods

The researchers developed a rigorous, human-in-the-loop (but "hands-off") protocol:

  1. Rewriting: An operator gives an LLM agent (via GitHub Copilot mode) the original repository and a prompt to rewrite it. They tested three prompt strategies: S1 (preserving layout), S2 (shifting programming paradigms), and S3 (ground-up rearchitecture).
  2. Iterative Repair: The agent is given a "behavioral harness" (a suite of tests). If the rewritten code fails, the agent must fix it autonomously until it passes. No human edits are allowed.
  3. Measurement: The team measured JPlag similarity scores, session time, and "provenance signals"—specific imports and "fingerprint facts" (like unique strings or numeric literals) that might leak from the original to the rewrite.
  4. Control Group: To see what "true" independence looks like, they commissioned Same-Task Independent Controls (STIC)—implementations written by a human from a specification without ever seeing the original code.

Datasets

The study focused on six popular copyleft Python projects, all under 15,000 lines of code (SLOC).

Table 2. Subject systems. SLOC counts Python lines excluding blanks and comment-only lines.

The researchers used five different LLM agents (Claude Haiku 4.5, DeepSeek V4 Pro, Gemini 2.5 Pro, GPT-5.4 mini, and Qwen 3.7 Plus) to produce a total of 69 rewrites.

Table 1. Study design. A run is one execution of the protocol for a project, an agent and a prompt.

Key Insights and Ideas

  • The "Minimum of Several Draws" Fallacy: Similarity scores are incredibly volatile. The same agent, given the same project and the same prompt, can produce a 15% similarity score in one run and a 75% score in the next. A maintainer could simply run the agent three times and only publish the lowest score.
  • Variance is the King: 61% of the variance in similarity scores comes from the "repetition" effect (the inherent randomness of the LLM), while the specific agent used accounted for less than 1%.
  • Task-Imposed Facts: Many signals used to detect "copying" (like specific library imports) are actually just requirements of the task. If you're building a face-swapping tool, you must import OpenCV. The study found that independent human implementations often shared as many "fingerprints" with the original as the AI rewrites did.
  • Type-4 Clones: Ground-up rewrites are essentially "Type-4 clones"—code that does the same thing but looks completely different. Traditional detectors like JPlag were never designed to catch these, making their "low" scores essentially meaningless in this context.

Key Results

  • Rewriting is Cheap: All 15 initial test runs passed the behavioral harness after agent-only repair. The median time for an initial rewrite was just 18 minutes.
  • Low Scores are Common: Under the most disruptive "ground-up" prompt, 42 out of 54 rewrites fell below a 25% similarity threshold. Six even scored below 5%.
  • Unreliable Evidence: In more than a third of the test configurations, three identical runs produced scores that straddled the 25% threshold (some high, some low).
  • Signal Failure: Neither import retention nor fingerprint facts could reliably distinguish an AI rewrite (which saw the source) from an independent human implementation (which didn't). The Area Under the Curve (AUC) for these signals was around 0.40–0.45—effectively no better than a coin flip.

Limitations and Future Work

The authors acknowledge several limitations:

  • Scale: The study used relatively small Python repositories (<15k SLOC). Results might differ for massive codebases or different languages.
  • Control Limitations: There was only one human-written independent control per project. A larger pool of human implementations would provide a more robust baseline.
  • Training Data Leakage: It is possible the LLMs saw these public projects during their initial training, though the "source-conditioned" setup (giving them the code directly) makes this less of a factor for the study's specific conclusions.

Future Work: The researchers suggest expanding the benchmark to include cross-language rewriting (e.g., C++ to Python) and testing whether agents can achieve similar results when given only a specification rather than the source code.

Summary updated

Abstract

Maintainers sometimes cite a low token-similarity score as evidence that an LLM-assisted rewrite of a copyleft project is independent of the original. We test what such a score can support under \emph{source-conditioned} rewriting: a coding agent keeps the original in its working copy, rewrites the repository, and iterates until a fixed behavioural harness passes, with no human edit to the code. On 69 rewrites of six copyleft Python projects we find that: \ding{182}~Agent-only rewriting is inexpensive and often yields low scores. Every controlled rewrite passes the harness after repair, usually within half an hour for the initial rewrite. Under a ground-up rearchitecture prompt, most rewrites fall below 25\% JPlag similarity and still pass the harness. \ding{183}~A single low score is an unreliable witness. Identical project--agent--prompt configurations produce wide spreads in similarity; run-to-run variation dominates project and agent effects, and in more than a third of configurations the three repeats straddle the threshold. Rewrites retain more of the original's imports and fingerprint facts than unrelated projects do, yet hand-written independent implementations of the same task often retain as much, and neither signal separates the two groups better than chance. Thus a low score on one rewrite is weak evidence of independence: another run of the same agent may score high. Claims based on similarity should report repeated generations, and candidate provenance signals should be checked against same task independent implementations, not only against unrelated projects.
BibTeX
@misc{moumoula2026low,
  author = {Micheline Bénédicte Moumoula and Serge Lionel Nikiema and Tailia Malloy and Abdoul Kader Kaboré and Jacques Klein and Tegawendé F. Bissyande},
  title = {{A Low Similarity Score Does Not Show Independence after Source-Conditioned Repository Rewriting}},
  year = {2026},
  eprint = {2611.07887},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.07887},
}
Figure