Improving Code LLM Robustness to Prompt Perturbations via Layer-Aware Model Editing

Shuhan Liu, Xing Hu, Kerui Huang, Xiaohu Yang, David Lo, Xin XiaarXiv 2025 · DOI · arXiv

Introduction

Large Language Models (LLMs) specialized for code generation have demonstrated remarkable capabilities in translating natural language requirements into functional code. However, these models exhibit a significant lack of robustness. Minor, semantically irrelevant variations in a prompt—such as a simple typo, a change in word order, or a shift from active to passive voice—can cause the model to produce functionally incorrect code.

A minor typo in the docstring ('mmodulo', 'numerixs') causes the model to generate incorrect logic, illustrating the robustness gap in current Code LLMs.

Existing approaches to mitigate this brittleness typically involve either expensive full-model fine-tuning or inference-time "denoising" prompts that increase latency. This paper introduces CREME (CodeLLM Robustness Enhancement via Model Editing), a lightweight and targeted method to patch the internal sensitivity of LLMs to these perturbations without degrading their baseline performance.

The CREME Framework

CREME adapts the principles of knowledge editing to the domain of behavioral robustness. Instead of updating a factual association (e.g., "The CEO of X is Y"), CREME identifies and modifies the specific neural pathways responsible for a model's sensitivity to prompt noise. The process consists of two primary stages:

1. Key Layer Localization

Using a technique known as causal tracing, CREME identifies which layers in the model are most critical for robustness. Given a pair of prompts—one clean and one perturbed—the model systematically replaces the hidden states of the perturbed prompt with those of the clean prompt, layer by layer. The layer that, when replaced, most effectively restores the model's ability to pass unit tests is designated as the "key layer."

2. Layer Editing

Once the culprit layer is localized, CREME performs a targeted update on the MLP output projection matrix of that specific layer. This update is driven by a dual-objective loss function:

  • Alignment Loss: Minimizes the distance between the hidden states of the perturbed prompt and the clean prompt.
  • Preservation Loss: Ensures that the model's response to the original clean prompt remains unchanged, preventing "catastrophic forgetting" of its baseline capabilities.
The CREME framework consists of two stages: Key Layer Localization to identify robustness-sensitive layers and Layer Editing to patch the model's behavior.

Experimental Evaluation

The authors evaluated CREME using the HumanEval and MBPP benchmarks, augmented with 20 categories of perturbations generated by NLPerturbator. These perturbations included character deletions, keyboard typos, synonym substitutions, and complex grammatical rephrasing.

Key Findings

  • Significant Robustness Gains: CREME achieved a 63% relative improvement in Pass@1 accuracy on perturbed prompts compared to unedited base models.
  • Localization Insights: Sensitivity to perturbations is not distributed uniformly. It is concentrated in specific middle-to-deep layers (e.g., layers 28–30 for CodeLlama-7b). Interestingly, while simple typos activate similar layers, complex paraphrasing shifts the sensitive region deeper into the network.
  • Generalization: A single edit based on one perturbed example allows the model to generalize to similar perturbations across entirely different coding tasks, achieving an average Generalized Relative Improvement Ratio (G-RIR) of 30%.
  • Efficiency and Stability: The editing process is highly efficient, requiring only ~5.3 seconds of computation on an A800 GPU. Furthermore, the edits are safe; performance on clean prompts remained stable within a $\pm 1\%$ margin.

Conclusion and Future Work

CREME demonstrates that the robustness of Code LLMs can be significantly enhanced by targeting a tiny fraction of the model's parameters. By treating robustness as a patchable internal behavior rather than a data-augmentation problem, CREME offers a scalable path toward more reliable AI programming assistants. Future research will explore extending this layer-aware editing to broader NLP tasks and defending against more sophisticated adversarial attacks.

Summary updated

BibTeX
@misc{liu2025improving,
  author = {Shuhan Liu and Xing Hu and Kerui Huang and Xiaohu Yang and David Lo and Xin Xia},
  title = {{Improving Code LLM Robustness to Prompt Perturbations via Layer-Aware Model Editing}},
  year = {2025},
  eprint = {2507.16407},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.16407},
}
Figure