A Differential Fuzzing-Based Evaluation of Functional Equivalence in LLM-Generated Code Refactorings

Simantika Dristi, Matthew B. Dwyer

TL;DR

Large Language Models (LLMs) are increasingly used to refactor code, but they frequently break program logic in ways that standard test suites fail to detect. By using differential fuzzing—generating thousands of random inputs to compare original and refactored code—researchers found that 19–35% of LLM-generated refactorings are functionally incorrect, and over 21% of these errors slip right past existing benchmarks.

What Problem Does This Paper Address?

Code refactoring is the process of restructuring existing computer code—changing the how without changing the what (its functional behavior). While LLMs like GPT-4o are great at rewriting code for better performance or simplicity, ensuring they haven't accidentally introduced a bug is difficult.

Traditionally, researchers evaluate LLMs using "pass@k" metrics on predefined test suites (like HumanEval). However, these test suites are often "thin"—they only cover a few common execution paths. If a refactored program passes the three tests provided in a dataset but fails on a specific edge case, it isn't truly equivalent to the original. This paper addresses the overestimation of LLM reliability caused by relying on inadequate, static test suites.

Methods

The authors introduce Eq@DFuzz, an evaluation framework based on differential fuzzing.

  1. Input Generation: Instead of relying on a handful of fixed tests, they use a fuzzer (specifically Atheris) to generate a massive, diverse set of test inputs (1,000 for programs, 2,000 for functions).
  2. Constraint Inference: To ensure the fuzzer generates "sensible" data (e.g., not passing a string where an integer is expected), they use GPT-4 to infer input constraints and types, which are then manually validated.
  3. Differential Execution: Both the original code and the LLM-refactored version are executed with the same fuzzed inputs.
  4. Equivalence Check: If the outputs differ for even a single input, the refactoring is flagged as "non-equivalent."

This approach is "black-box" and doesn't require predefined test cases, allowing it to explore a much larger portion of the program's input space than standard benchmarks.

Datasets

The study evaluated refactorings across three popular Python coding benchmarks:

  • HumanEval: 164 function-level problems.
  • MBPP (Mostly Basic Python Problems): 100 sampled function-level problems.
  • APPS: 100 sampled program-level problems (more complex than the others).

They tested six LLMs: CodeLlama-13B, Codestral-22B, StarChat2-15B, Qwen-2.5-Coder-32B, Olmo-3-32B, and GPT-4o.

Key Insights and Ideas

  • Refactoring is Risky Business: Even when explicitly told not to change functionality, LLMs failed to preserve semantics in roughly one-quarter to one-third of all attempts.
  • Complexity Breeds Error: Failure rates were significantly higher on the APPS dataset (32.09% non-equivalent) compared to HumanEval (22.10%). This suggests that as code becomes more "real-world" and less "snippet-sized," LLMs become much less reliable.
  • Simplification isn't "Simple": You might expect "code simplification" to be safer than "performance optimization," but the study found they produced nearly the same fraction of errors (~26%). Even removing "redundant" code often leads to logic errors.
  • The "Test Suite Gap": The most alarming finding is that ~21.65% of broken refactorings passed all existing tests in the datasets. This proves that standard benchmarks are often too shallow to serve as a "source of truth" for code correctness.

Key Results

  • Failure Rates: Across all models and datasets, 19–35% of refactorings were functionally non-equivalent.
  • Model Performance:
    • Codestral had the highest rate of semantic breakage (~35%).
  • GPT-4o was the most "cautious" but still produced non-equivalent code 18.58% of the time.
  • Detection Failure: In the APPS dataset, 22.41% of the refactorings that were proven broken by fuzzing actually passed every single one of the 21.2 (average) pre-defined test cases.

Limitations and Future Work

The authors acknowledge a few limitations:

  • Language Scope: The study was limited to Python. Other languages with different type systems or paradigms might yield different results.
  • Prompting Strategy: The study used zero-shot prompting. Future work could explore if "Chain-of-Thought" or "Few-Shot" prompting helps models maintain functional equivalence.
  • Computational Cost: Differential fuzzing is much more computationally expensive than running a standard test suite, which might limit its use in real-time CI/CD pipelines.

The authors suggest that future LLM training should move beyond just "passing tests" and instead optimize for functional equivalence using metrics like Eq@DFuzz.

Summary updated

Abstract

With the rapid adoption of large language models (LLMs) in automated code refactoring, assessing and ensuring functional equivalence between LLM-generated refactoring and the original implementation becomes critical. While prior work typically relies on predefined test cases to evaluate correctness, in this work, we leverage differential fuzzing to check functional equivalence in LLM-generated code refactorings. Unlike test-based evaluation, a differential fuzzing-based equivalence checker needs no predefined test cases and can explore a much larger input space by executing and comparing thousands of automatically generated test inputs. In a large-scale evaluation of six LLMs (CodeLlama, Codestral, StarChat2, Qwen-2.5, Olmo-3, and GPT-4o) across three datasets and two refactoring types, we find that LLMs show a non-trivial tendency to alter program semantics, producing 19-35% functionally non-equivalent refactorings. Our experiments further demonstrate that about 21% of these non-equivalent refactorings remain undetected by the existing test suites of the three evaluated datasets. Collectively, the findings of this study imply that reliance on existing tests might overestimate functional equivalence in LLM-generated code refactorings, which remain prone to semantic divergence.
BibTeX
@inproceedings{dristi2026differential,
  author = {Simantika Dristi and Matthew B. Dwyer},
  title = {{A Differential Fuzzing-Based Evaluation of Functional Equivalence in LLM-Generated Code Refactorings}},
  booktitle = {SIGSOFT FSE Companion},
  year = {2026},
  doi = {10.1145/3803437.3806717},
}
Figure