TL;DR
Can an AI "cleanse" copyleft code by rewriting it from scratch? This paper demonstrates that LLM coding agents can produce functionally identical versions of software with extremely low similarity scores, but these scores are highly unstable and fail to prove true independence. A single low score is often just a "lucky draw" from a wide distribution of possible outputs, making it a weak defense for license compliance.
What Problem Does This Paper Address?
In the world of open-source software, "copyleft" licenses (like the GPL) require that any derivative work also be open-source. To move a project to a more permissive license (like MIT), maintainers sometimes claim they have performed a "ground-up rewrite" using an AI agent, citing low similarity scores from tools like JPlag as proof that the new code is independent of the original.
The authors investigate whether these token-similarity scores are actually a reliable "witness" for independent authorship. Specifically, they look at source-conditioned rewriting: a scenario where an AI agent has full access to the original code in its workspace while it generates the "new" version. If an agent with full access can easily produce a low-similarity score, then a low score cannot be used to prove the agent didn't rely on the original.
Methods
The researchers developed a rigorous, human-in-the-loop (but "hands-off") protocol:
- Rewriting: An operator gives an LLM agent (via GitHub Copilot mode) the original repository and a prompt to rewrite it. They tested three prompt strategies: S1 (preserving layout), S2 (shifting programming paradigms), and S3 (ground-up rearchitecture).
- Iterative Repair: The agent is given a "behavioral harness" (a suite of tests). If the rewritten code fails, the agent must fix it autonomously until it passes. No human edits are allowed.
- Measurement: The team measured JPlag similarity scores, session time, and "provenance signals"—specific imports and "fingerprint facts" (like unique strings or numeric literals) that might leak from the original to the rewrite.
- Control Group: To see what "true" independence looks like, they commissioned Same-Task Independent Controls (STIC)—implementations written by a human from a specification without ever seeing the original code.
Datasets
The study focused on six popular copyleft Python projects, all under 15,000 lines of code (SLOC).
The researchers used five different LLM agents (Claude Haiku 4.5, DeepSeek V4 Pro, Gemini 2.5 Pro, GPT-5.4 mini, and Qwen 3.7 Plus) to produce a total of 69 rewrites.
Key Insights and Ideas
- The "Minimum of Several Draws" Fallacy: Similarity scores are incredibly volatile. The same agent, given the same project and the same prompt, can produce a 15% similarity score in one run and a 75% score in the next. A maintainer could simply run the agent three times and only publish the lowest score.
- Variance is the King: 61% of the variance in similarity scores comes from the "repetition" effect (the inherent randomness of the LLM), while the specific agent used accounted for less than 1%.
- Task-Imposed Facts: Many signals used to detect "copying" (like specific library imports) are actually just requirements of the task. If you're building a face-swapping tool, you must import OpenCV. The study found that independent human implementations often shared as many "fingerprints" with the original as the AI rewrites did.
- Type-4 Clones: Ground-up rewrites are essentially "Type-4 clones"—code that does the same thing but looks completely different. Traditional detectors like JPlag were never designed to catch these, making their "low" scores essentially meaningless in this context.
Key Results
- Rewriting is Cheap: All 15 initial test runs passed the behavioral harness after agent-only repair. The median time for an initial rewrite was just 18 minutes.
- Low Scores are Common: Under the most disruptive "ground-up" prompt, 42 out of 54 rewrites fell below a 25% similarity threshold. Six even scored below 5%.
- Unreliable Evidence: In more than a third of the test configurations, three identical runs produced scores that straddled the 25% threshold (some high, some low).
- Signal Failure: Neither import retention nor fingerprint facts could reliably distinguish an AI rewrite (which saw the source) from an independent human implementation (which didn't). The Area Under the Curve (AUC) for these signals was around 0.40–0.45—effectively no better than a coin flip.
Limitations and Future Work
The authors acknowledge several limitations:
- Scale: The study used relatively small Python repositories (<15k SLOC). Results might differ for massive codebases or different languages.
- Control Limitations: There was only one human-written independent control per project. A larger pool of human implementations would provide a more robust baseline.
- Training Data Leakage: It is possible the LLMs saw these public projects during their initial training, though the "source-conditioned" setup (giving them the code directly) makes this less of a factor for the study's specific conclusions.
Future Work: The researchers suggest expanding the benchmark to include cross-language rewriting (e.g., C++ to Python) and testing whether agents can achieve similar results when given only a specification rather than the source code.
Summary updated