When a code model is given a familiar benchmark problem with one piece of logic flipped (sort descending instead of ascending, intersection instead of union), it often writes the memorized original solution anyway. A common response is to run a refinement loop: execute the code, show the model the failing tests, and ask it to fix the program. No one has checked whether this loop actually pulls the model away from the memorized answer, or whether each failed attempt drags it back toward the canonical solution it has seen thousands of times. We propose to measure this directly inside the model. We locate a low-dimensional direction in the model's hidden states that encodes which version of the task the model believes it is solving, the original or the flipped one. We track that direction across refinement rounds, test whether it predicts which loops will succeed, and check whether pushing along it rescues failing loops.
The technical tool is supervised dimensionality reduction, meaning compression of high-dimensional features that keeps only the directions predictive of a chosen label. Our variant also explicitly removes directions tied to a nuisance variable. This matters because of a confound that has not been addressed. The "evolved" problems used to expose memorization are written by an LLM, so they differ from the originals in writing style as well as in logic. Code models are also known to swing widely in accuracy under purely stylistic rewording. A naive probe for "flipped vs. original" could therefore learn "LLM-written vs. human-written." We build the probe to be blind to prompt style by construction, and that design choice is what makes the measurement trustworthy.
Core Claim
Our central claim is a hypothesis, not yet a result. In models whose memorization comes from supervised fine-tuning, execution feedback on logic-flipped problems produces memorization pull: across rounds, the hidden-state representation of task identity drifts toward the original-problem side even as the textual feedback says the code is wrong. We further claim that a direction recovered by fair partial least squares predicts refinement failure earlier and more reliably than behavioral signals, and that steering along it increases repair success. Fair partial least squares finds directions that covary with task identity while being penalized for covarying with prompt style. "Earlier" means before code is generated. The behavioral signals we compare against are similarity of the output to the memorized solution and instability across paraphrases.
Grounding in Prior Work
The memorization side comes from Memorize or Generalize? Evaluating LLM Code Generation with Evolved Questions. That paper builds the Code Rewriting suite by flipping one logic step in MBPP+ and BigCodeBench problems and reverse-engineering a new prompt. It defines a Memorization Risk Index (MRI) as similarity to the original solution times the accuracy drop. It reports that MRI on MBPP+ falls from 0.0722 at Qwen-2.5-Instruct 0.5B to near zero at 32B, that supervised fine-tuning nearly doubles MRI for a 7B model on BigCodeBench, and that PPO does not. Its evaluation is single-turn. Whether refinement breaks the trap is explicitly left open, and that is the gap we take up. Its finding that SFT and RL checkpoints differ gives us a built-in contrast.
Two results motivate the worry that loops can make things worse. Do Users Write More Insecure Code with AI Assistants? found that developers who fed a model its own flawed code without independent verification compounded vulnerabilities. CodeTransOcean reports self-debugging gains that plateau within one to three rounds, with GPT-3.5-turbo going from 48.57% to 52.57% Debugging Success Rate on LLMTrans. Neither study looks at memorized content specifically. Generate-Feedback-Refine supplies our loop design. It separates generator, feedback, and refiner roles and shows that natural-language feedback beats raw test errors, so we run both feedback types.
The style confound is grounded in Prompt Stability in Code LLMs. That paper reports accuracy swings up to 40% across 14,760 emotion- and persona-styled HumanEval variants, and finds that raw skill and stability are decoupled. The MRI paper reports robustness to paraphrase, which might seem to make the confound moot. However, that finding concerns outputs. A hidden-state probe can still latch onto style features that are correlated with the flip label, so the confound must be controlled at the representation level.
The method comes from PLS-based approach for fair representation learning. That paper maximizes covariance with a target while penalizing covariance with a sensitive attribute. It shows that unsupervised compression discards low-variance predictive directions: on Adult Income, Fair PLS keeps over 83% accuracy versus about 76.5% for Fair PCA. The authors name LLM embeddings as future work, and we apply their objective to transformer hidden states.
The closest representation-level competitor is CREME. It localizes prompt sensitivity in CodeLlama-7b to layers 28–30 by causal tracing and edits an MLP projection so that perturbed and clean hidden states align, reporting a 63% relative Pass@1 improvement on perturbed prompts. Our aim differs from CREME's in a basic way. CREME makes the model ignore differences between prompts. Our task requires the model to preserve the one difference that matters, the logic flip, while ignoring style. Applied naively to an original/flipped pair, CREME's alignment loss would erase exactly the information the model needs. CREME also studies only one-shot generation. Finally, On the Relation between Sensitivity and Accuracy in In-Context Learning shows that output instability across prompt variants predicts errors (its SENSEL method, +4.1 AUC on GPT-J-6B). It therefore serves as the behavioral uncertainty baseline our probe must beat.
Research Program
We first build a factorial prompt set that separates logic from style. For each Code Rewriting problem on MBPP+ and BigCodeBench, we create four prompt versions. One is the original text with the original logic. A second is a paraphrase of the original logic written by the same LLM that wrote the evolved prompts, so it carries "rewritten" style but the original logic. The third is the evolved prompt, which has flipped logic. The fourth is the evolved prompt restyled with PromptSE-type emotional or persona variations. Under this design, the task-identity label $Y$ (flipped or not) and the style labels $S$ are no longer confounded, and we can measure how much any probe relies on style.
We then extract residual-stream hidden states at the last prompt token, and at the last token of each feedback message during refinement. Call the matrix of these states $H$. Following Fair PLS, we find directions $w$ that solve
$$\max_{w}\; \mathrm{Cov}^2(Hw, Y) \;-\; \eta \sum_{j} \mathrm{Cov}^2(Hw, S_j),$$
where $Y$ is the flip label, each $S_j$ is a style indicator (LLM-paraphrased, emotion variant, persona variant), and $\eta \ge 0$ controls how strongly style is suppressed. We keep a few orthogonal components and pick the layer by held-out accuracy, with the CREME-identified middle-to-deep layers as a prior. Probes are fit on one half of the problems and evaluated on the other half, so the probe never sees the test problems.
The quantity we track is memorization pull. Let $h_t$ be the hidden state at refinement round $t$. Let $\mu_{\text{orig}}$ and $\mu_{\text{flip}}$ be the mean projections of original-logic and flipped-logic prompts onto $w$. We define
$$m_t = \frac{(h_t^\top w - \mu_{\text{flip}})}{(\mu_{\text{orig}} - \mu_{\text{flip}})},$$
so that $m_t = 0$ means the model represents the flipped task and $m_t = 1$ means it represents the original. Memorization pull is a positive slope of $m_t$ over failed rounds on flipped problems.
We run refinement loops on Qwen-2.5 base, SFT-instruct, and RL-trained 7B checkpoints. The Qwen-2.5 family is used because MRI was measured on it. We also run CodeLlama-7b to allow a direct CREME comparison. Each loop has up to four rounds, which covers the documented plateau, with both raw-test and natural-language feedback. For the causal test, at refinement turns on flipped problems we add $-\alpha w$ to the residual stream. This nudges $m_t$ toward zero. We then measure pass rate, and separately confirm that the same intervention leaves original-logic problems unharmed.
Baselines
The naive reference point is single-shot generation with no refinement on the evolved prompts, scored by pass@1 and MRI. This tells us whether loops help at all on memorization traps.
The strongest refinement baseline is execution-feedback refinement in the Generate-Feedback-Refine style, run with natural-language feedback. It is given the same number of rounds and the same token budget as every steered variant. As the strongest trained refinement policy, we use TGPR if its checkpoint is available. Its reported results are 77.6% on MBPP and 41.2% on LiveCodeBench at 7B, against GPT-4o-mini's 74.8% and 36.7%. If the checkpoint is unavailable, we use a GRPO-trained refiner on matched data as a stand-in. We found no reported result on Lacuna for any refinement loop evaluated on the Code Rewriting suite, so we will establish these baseline numbers ourselves. Beating the baseline means higher final pass@1 on flipped problems at matched rounds and tokens, with no more than a one-point drop on original problems.
For prediction, three competitors face the probe. The first is MRI's similarity term computed on each round's generated code. The second is SENSEL-style disagreement across five prompt paraphrases, given the extra forward passes it requires. The third is the model's token-level entropy, following the confidence account in ProSA. Each is scored by area under the ROC curve (AUROC) for predicting final loop failure from round 0. The probe beats them if its AUROC is at least 0.05 higher and the difference holds across both benchmarks.
Three ablations isolate the key ingredient. Setting $\eta = 0$ gives plain PLS. Logistic regression gives an ordinary linear probe. PCA gives unsupervised compression. All are fit on identical data and layers. The decisive ablation compares their accuracy on the "LLM-paraphrased original" cell. A style-contaminated probe will misclassify these prompts as flipped, and fair PLS should not. For steering, the controls are a random direction of equal norm, the $\eta = 0$ direction, and a CREME edit followed by the same refinement loop.
Why not just…?
Why not just track output similarity to the memorized solution each round? This is the most serious threat. It is cheap, it is the MRI's own ingredient, and it may track pull well once code exists. The probe's possible edge is timing: it is readable before any code is generated and at the feedback token. That would allow early abstention or a change of strategy. If output similarity at round 0 matches the probe's AUROC, we will report that the representation adds no predictive value. The contribution then reduces to the behavioral finding about pull plus the steering test, which output similarity cannot provide.
Why not just train an ordinary probe on hidden states? The flipped prompts are LLM-authored, so an ordinary probe can succeed by detecting authorship style. The factorial design makes this measurable. If a plain probe classifies LLM-paraphrased originals correctly as often as fair PLS does, the style penalty is unnecessary, and we will say so.
Why not just tell the model the problem differs from the textbook version, or sample more? Explicit hints and larger pass@k budgets are legitimate fixes. We include both as arms of the loop at matched tokens. If a one-sentence hint closes the gap, steering is moot for practice, but $m_t$ still tells us whether the hint works by moving the representation.
Why not just use RL-trained models, which the MRI paper says do not memorize? This may be partly right. Our design predicts pull mainly in SFT checkpoints, and an absence of pull in RL checkpoints is an expected outcome rather than a failure. The SFT-versus-RL comparison inside the same family settles how far the problem extends.
Why not just apply CREME? CREME aligns perturbed prompts to clean ones. On flipped problems this risks collapsing exactly the distinction the model must keep. Running CREME followed by refinement as a baseline tests this directly: we predict its flipped-problem pass rate falls or its MRI rises.
Decisive Checks
The project hinges on four observations. The first is whether $m_t$ rises over failed rounds in SFT checkpoints but not in RL checkpoints on BigCodeBench rewrites, where MRI is reported to remain non-trivial. The second is whether fair PLS stays accurate on the style-only cell where plain probes fail. The third is whether round-0 probe readings beat output similarity and paraphrase instability at predicting final failure. The fourth is whether steering along $-w$ raises flipped-problem pass rates beyond random-direction and plain-probe controls without hurting originals. As a contamination control, we repeat the prediction analysis on LiveCodeBench problems released after the model's cutoff. There, $Y$ cannot be defined, so we instead check that $m_t$ stays near its uninformative midpoint, which confirms that the direction encodes memorized identity rather than generic difficulty. If pull appears and steering helps, refinement loops for contaminated models should monitor task identity rather than only test outcomes. If pull does not appear, we will have the first direct evidence that execution feedback is a sufficient remedy for memorization traps.
References
- Memorize or Generalize? Evaluating LLM Code Generation with Code Rewriting
- Do Users Write More Insecure Code with AI Assistants?
- CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
- Generate-Feedback-Refine: How Much Does Model Quality in Each Role Matter?
- Prompt Stability in Code LLMs: Measuring Sensitivity across Emotion- and Personality-Driven Variations
- PLS-based approach for Fair Representation Learning
- Improving Code LLM Robustness to Prompt Perturbations via Layer-Aware Model Editing
- On the Relation between Sensitivity and Accuracy in In-Context Learning
- From Search to Policy: Thompson Sampling Trees for Robust LLM Code Refinement
- ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs