Where Should Code Refinement Be Smooth?

Coding agents now routinely edit their own earlier code as requirements change, and the code gets worse as they do. On a benchmark built for exactly this setting, agent code drifts toward bloated, tangled designs far faster than human code does, and asking the agent to "write clean code" does not slow the decline. Machine learning has a standard fix for models that drift when they are re-fit at each step of a sequence: a temporal smoothness penalty, which charges the model for moving its state too far between consecutive steps. This note asks what that idea means for code. The answer is not obvious, because the most literal translation, "prefer the smallest edit," may be exactly what causes the decay. Agents that bolt small patches onto existing functions end up with more and more logic piled into the functions that were already the most complicated.

We propose a selection rule for multi-checkpoint coding agents that keeps the program's architecture smooth while leaving its text free to change. The agent samples several candidates at each checkpoint. Among the candidates that pass the tests, it picks the one whose structural fingerprint stays closest to the previous checkpoint's. That fingerprint records which functions exist, how they call each other, and how complexity is spread across them. The distance penalty is robust, so it tolerates the occasional large jump a new specification really requires. The central test is whether this rule improves future checkpoint pass rates relative to two alternatives: minimal-text-diff selection, and selecting directly on the benchmark's own erosion metric. That is the kind of downstream benefit smoothness priors deliver in sequential representation learning.

Core Claim

Smoothness regularization pays off only when it is placed on the quantity whose drift actually hurts. In sequential embedding models, smoothing the embedding coordinates removes arbitrary rotations between time slices and lets each slice borrow statistical strength from its neighbors. Dynamic Word Embeddings for Evolving Semantic Discovery jointly factorizes yearly co-occurrence matrices with a penalty on squared change between consecutive years, and stays aligned even when the data are subsampled to 0.1%. Tensor Decompositions for Temporal Knowledge Base Completion adds a squared-difference prior on adjacent timestamp embeddings and reports up to 0.07 Mean Reciprocal Rank gain on ICEWS. These two papers supply the mechanism. The same knowledge-base paper, together with the general observation that quadratic penalties oversmooth genuine step changes, constrains the design: specification checkpoints are real discontinuities, so the penalty must not punish them quadratically.

Our claim has three parts, each falsifiable. First, smoothness in text space (minimum diff) will increase structural erosion relative to unregularized selection, because it rewards in-place patching. Second, smoothness in architecture space, under a robust penalty, will reduce erosion and verbosity. Third, and most important, architecture-space smoothness will raise pass rates at later checkpoints by more than direct optimization of the erosion metric does. The reason is that it preserves a design the agent can keep extending, rather than gaming a complexity score.

The Selection Rule

Let $C_{k-1}$ be the accepted code at checkpoint $k-1$, and let $S_k$ be the set of $N$ sampled candidates at checkpoint $k$ that pass the visible tests. Every candidate in $S_k$ is equally acceptable to the ICER-style relaxed policy. Let $\phi(c)$ be a structural fingerprint of program $c$. It has three components. The first is the call graph over named functions and classes. The second is the per-function cyclomatic complexity profile; cyclomatic complexity counts independent paths through a function's branches. The third is the set of near-duplicate code clones. Let $d$ be a distance on fingerprints: graph edit distance on the call graph, plus a transport distance between complexity profiles, plus the change in clone mass. The rule picks

$$ C_k = \arg\min_{c \in S_k} \; \rho_\delta\big(d(\phi(c), \phi(C_{k-1}))\big), $$

where $\rho_\delta$ is a Huber function. It is quadratic below a threshold $\delta$ and linear above it, so one large spec-driven restructuring costs far less than it would under a squared penalty. If $S_k$ is empty, we fall back to the candidate passing the most tests.

We also test a second-order variant borrowed from Temporal Straightening for Latent Planning. That paper penalizes the angle between consecutive latent velocity vectors and reports 20–60% open-loop planning success gains. In our variant, the anchor is the extrapolated fingerprint $\phi(C_{k-1}) + (\phi(C_{k-1}) - \phi(C_{k-2}))$ instead of the previous fingerprint. This charges the agent for reversing a design direction, for example re-inlining helpers it extracted one step earlier, rather than for moving at all.

The method needs no training, which is deliberate. It isolates the question of which space to smooth from confounds introduced by fine-tuning.

Research Program

The work proceeds in three stages. The first stage builds the fingerprint extractor for Python, the language of the benchmark, and validates that $d$ tracks something real. On the 473-repository human history corpus that accompanies SlopCodeBench: Measuring Code Erosion Under Iterative Specification Refinement, we measure per-commit fingerprint distances for human commits and for agent checkpoints. The prediction is that agent trajectories show larger, more erratic architectural motion even when their text diffs are comparable. If this does not hold, the premise fails and we stop.

The second stage runs the selection-rule comparison described below on SlopCodeBench. The third stage is a diagnostic. On trajectories where architecture-smooth and erosion-metric selection pick different candidates, we inspect what each chose. We look specifically for metric gaming in the erosion-selected code, such as trivially split functions or helpers that wrap a single line, and for over-anchoring in ours, where a candidate was rejected even though the new spec genuinely demanded a redesign.

Baselines

The naive reference point is the unmodified agent producing one sample per checkpoint. This is the regime where SlopCodeBench reports that the best agents pass about 14.8% of checkpoints and erosion rises in 77% of trajectories. We rerun it ourselves with the same backbone model so that every comparison shares one base rate.

The relaxed-acceptance baseline samples $N$ candidates and picks uniformly among those that pass the tests. This is the policy Improving Code Efficiency with Iterative Refinement using LLMs argues for against strict monotone filtering. It matters because it shows how much gain comes purely from best-of-$N$ sampling with no smoothness at all.

The text-smoothness ablation is the same rule with $d$ replaced by line-level diff size. It isolates the key ingredient: if architecture and text smoothing perform alike, the structural fingerprint adds nothing.

The rollback baseline follows MEIC: Re-thinking RTL Debug Automation using LLMs. MEIC pairs a debug agent with a scorer and reverts to the last stable state on regression. Its reported numbers are 93% syntax and 78% functional fix rates on RTL bugs, a different domain from ours. We port it by reverting whenever a checkpoint increases the erosion metric, with the same sampling budget for retries.

The strongest competitor is direct erosion-metric selection: pick the test-passing candidate with the lowest SlopCodeBench erosion-plus-verbosity score. It optimizes the benchmark's own quality measure, so beating it on that measure is not expected. Beating it means two things. On the primary outcome, it means higher pass rates on checkpoints $k+1$ through $K$. On secondary outcomes, it means comparable or better scores on quality measures not used for selection, namely maintainability index, maximum function length, and the human-relative erosion ratio. No reported result found on Lacuna for best-of-$N$ or metric-guided selection on SlopCodeBench, so all selection baselines are run by us.

Fairness is enforced in the same way for every condition. All conditions share the backbone model and the per-checkpoint sample count $N$ (we sweep 4, 8, and 16). The penalty settings ($\delta$ and the weights inside $d$) and the rollback threshold get equal tuning budgets, selected by problem-level cross-validation over the 36 problems. Quality-aware prompting from SlopCodeBench is included as a reference run.

Why not just…?

Why not just prompt the agent to keep code clean or keep edits minimal? SlopCodeBench already tested quality-aware prompting. Prompting improved the first checkpoint but did not change the rate of degradation over later checkpoints, raised token cost, and slightly lowered correctness. We include it as a reference row, and we expect a "minimal edits" prompt to behave like our text-smoothness ablation.

Why not just select on the erosion metric directly? This is the real threat, and we treat it as the main baseline rather than a straw man. If metric selection matches our rule on future-checkpoint pass rate and on the held-out quality measures, then smoothness adds nothing, and we will report that outcome. Our reason for expecting a difference is mechanistic. A per-checkpoint score has no memory of the design the agent committed to, so it can prefer a candidate that lowers complexity by reorganizing everything, leaving the next checkpoint harder to extend. The Stage 3 diagnostic checks this directly against selected code.

Why not just run a periodic refactoring pass or use a bigger model? A refactor pass is an extra step that itself causes large architectural jumps. We test it as an extra condition, with a refactor every two checkpoints at matched token budget. On model scale, SlopCodeBench reports that even frontier agents produce code 2.0x more eroded than human code. MEIC reports that structured GPT-3.5 (64% syntax fix rate) beat standalone GPT-4 (62%). Both results suggest that process structure, not scale alone, is what matters. We nonetheless run two backbone sizes to check whether the effect shrinks as models get stronger.

Proposed Experiments

The primary experiment runs all conditions on all 196 SlopCodeBench checkpoints, with three seeds each. The primary metric is pass rate on checkpoints $2..K$, reported conditional on the condition's own earlier choices so that downstream effects are captured. Secondary metrics are SlopCodeBench erosion and verbosity, the fraction of trajectories in which erosion rises (77% at baseline), and the held-out quality measures. Because there are only 36 problems, we report paired per-problem differences with bootstrap intervals and pre-register the future-pass-rate comparison against metric selection as the single primary test.

Three checks are decisive. First, if text-diff selection does not raise erosion relative to relaxed selection, the premise that minimal patching drives decay is wrong. Second, if architecture-space smoothing does not beat metric selection on future pass rate, the transfer of smoothness priors to code is not worth pursuing beyond the metric itself. Third, swapping the Huber penalty for a squared one should hurt specifically at checkpoints whose spec changes are large. Large spec changes are identified by the size of the diff between successive specification texts. This mirrors the oversmoothing failure known from sequential embeddings, and it would confirm that the robust penalty matters. A smaller follow-up applies the straightening variant within the debugging rounds of a single checkpoint on CodeTransOcean's LLMTrans set, where self-debugging plateaus from 48.57% to 52.57% after three rounds CodeTransOcean. The question there is whether penalizing design reversals reduces back-and-forth thrashing between edits.