When people ask a code-generating language model for code that is correct and also readable, modular, fast, and memory-light, the model eventually starts failing. Two benchmarks describe this failure. RACE finds that instruction following "approaches zero when five or more simultaneous requirements are imposed" (Beyond Correctness). SWE-IF finds that five non-functional instructions cause 5–10% functional regression and push task success below 50% (SWE-IF). Both vary the number of constraints. Neither varies how tight each constraint is. "Keep functions short" can mean under 40 lines or under 6, and "be efficient" can mean within 10× of a reference solution's runtime or within 1.2×. So the "complexity wall" might come from constraint count, from one or two constraints being set near an infeasible severity, or from specific pairs of constraints interacting badly. These three explanations call for different fixes, and the current evidence cannot tell them apart.
We propose to treat code-quality constraints the way structural engineers treat loads in a parametric study. Each constraint becomes a continuous knob, and each knob is normalized against the value achieved by a human reference solution for the same problem. Engineers normalize a pile's roof-thickness threshold by pile diameter in the same way (karst cave piles). We sample the joint space of knob settings with a Latin hypercube, which is a space-filling random design. We fit a Gaussian-process emulator, a smooth statistical model that predicts success at untried settings together with its uncertainty, following the hemodynamics workflow of Mining data from hemodynamic simulations via Bayesian emulation. From the emulator we compute variance-based sensitivity indices and locate collapse thresholds. The concrete claim is testable: once severity is controlled, constraint count explains little of the failure. We expect most of it to come from a small number of severity thresholds plus a few pairwise interactions, and we expect those thresholds, in normalized units, to transfer across problems.
Core Claim
Our hypothesis is that joint success under constraints behaves like a failure surface with sharp edges, and that the edges lie at roughly fixed normalized severities. By joint success we mean passing the tests and satisfying every constraint. An example of such an edge would be a cyclomatic-complexity ceiling below about 0.7× the reference solution's complexity, or a runtime budget below about 1.5× the reference runtime. If the hypothesis is right, the count-based picture in RACE and SWE-IF is a confound. Five loose constraints should be nearly free, and one tight constraint should be expensive.
The same lens explains an apparent contradiction in the literature. Beyond Pass@k fine-tunes models toward a human-centric quality score and reports a "correctness tax" of only about 1% for a 16-point quality gain. SWE-IF reports 5–10% regression when constraints are added in the prompt. Our prediction is that the two papers sampled different regions of the constraint space. The fine-tuning study stayed far from the edges, and the instruction benchmark hit them.
Research Program
We define six continuous knobs. All are checked by deterministic tools, so no LLM judge is needed. This matters because NoFunEval shows models discriminate code quality poorly, with GPT-4 at 42% on NoFunClassify.
The first four knobs are structural:
- the per-function cyclomatic-complexity ceiling;
- the maximum function length in lines;
- the maximum nesting depth;
- the minimum number of helper functions, which measures modular decomposition, where RACE reports most models below 45%.
The last two knobs are about efficiency. They are a wall-clock budget and a time-integrated memory budget, each expressed as a multiple of the reference solution's measured value, using EffiBench's NET and NTMU instrumentation (EffiBench).
Two categorical style instructions from SWE-IF's verifiable taxonomy are also included, checked with Ruff. They let us relate our continuous knobs to SWE-IF's count-based regime.
Every structural and efficiency knob is expressed as a ratio to the reference solution's own value. A severity of 1.0 therefore means "as tight as the human solution." Settings below 1.0 approach infeasibility in a way that can be compared across problems.
The problem pool combines three sources. EffiBench's 1,000 LeetCode problems provide canonical solutions and efficiency measurement. HumanEval+ provides strong functional tests. A subset of Big-SWE-IF lets us reproduce the count-based finding on its home turf.
For each model, we draw about 256 Latin hypercube configurations over the knob space. For each configuration, we sample problems and five generations each. We record three outcomes per generation:
- functional pass;
- satisfaction of each individual constraint;
- joint success.
We fit a Gaussian-process classifier per model on these outcomes. We then add samples near the estimated collapse boundary using an acquisition rule that targets high predictive uncertainty at the 50%-success contour. This is the same idea as the Probability-of-Improvement search in the hemodynamics work, repointed at boundaries instead of optima. The motivation comes from the karst cave study, which warns that coarse sweeps miss abrupt regime changes such as its shift to thin-plate failure.
From the fitted emulator we compute two kinds of Sobol indices:
- first-order indices, the share of outcome variance explained by each knob alone;
- total-effect indices, which add every interaction that knob participates in.
We also fit a logistic "failure curve" along each knob to estimate a threshold location and a steepness. Toward the Simplified Design of Single-Angle Beam-Columns supplies the method template: its one-factor sweeps showed that eccentricity sensitivity depends on slenderness, with a 35% capacity loss for stocky members versus about 6% for slender ones. The analogue here is whether efficiency-budget sensitivity depends on problem type. EffiBench already shows dynamic programming is where memory blowups concentrate.
The models are chosen to span size and alignment style: StarCoder2-15B, DeepSeekCoder-33B-Instruct, Llama-3-8B-Instruct, and one closed API model. To test whether preference training moves the failure edges, we train a 7B model with an efficiency-preference recipe modeled on CodeDPO. That paper builds preference pairs in which the faster of two verified-correct solutions wins, and it reports 1.25–1.45× speedups. We compare the trained model's failure surface with its base model's.
Baselines
Naive reference: no constraints. The first reference point is each model's unconstrained pass rate and quality metrics on the same problems, with the same sampling temperature and five samples. Every tax is measured against this.
Count-only model. The strongest existing account is the count model implied by RACE and SWE-IF: joint success is a decreasing function of how many constraints are active. We implement it as a logistic regression on constraint count, with per-problem random effects, fit on exactly the same generations the emulator sees. The published reference points are:
- SWE-IF's 5–10% functional regression and below-50% task success under five instructions (SWE-IF);
- RACE's near-zero instruction following at five or more requirements, where the best overall RACE score is o1-mini at 63.5 (Beyond Correctness).
We first reproduce the count effect on Big-SWE-IF to confirm our pipeline matches. We then say the emulator beats this baseline if its held-out log-loss and Brier score on unseen configurations improve by a margin larger than the spread across bootstrap resamples of problems. We found no reported result on Lacuna for held-out prediction of constraint-conditioned success, so both numbers will be produced here.
One-factor-at-a-time additive model. This baseline isolates the key ingredient, which is joint sampling that can detect interactions. We spend the same generation budget on one-factor sweeps with all other knobs held loose, and combine the per-knob curves additively. If this model predicts held-out joint configurations as well as the emulator, the interactions are negligible and the Latin hypercube design adds nothing. We would report that result as such.
Multi-turn repair. The strongest practical competitor is multi-turn repair, in which the model is shown checker feedback and asked to fix its code. SWE-IF reports a 3–8% advantage for multi-turn editing on Live-SWE-IF. We run up to two repair rounds with identical checker output for every model. We then ask whether repair shifts the failure thresholds or only lifts the curve uniformly.
Why not just…?
Why not just count constraints, as RACE and SWE-IF already do? Counting is cheap and already predictive. It stays a live competitor until the decisive test below is run. The problem is that in existing designs count is correlated with total severity, so counting cannot separate "many constraints" from "at least one tight constraint." Our count-matched and severity-matched arms break that correlation directly. If count still dominates after severity is controlled, our core claim is wrong.
Why not just sweep one knob at a time and skip the emulator? Unlike the 14.5-CPU-hour CFD runs in the hemodynamics study, LLM generations are cheap. A skeptic can fairly argue that simple sweeps suffice. We agree this is a real threat, which is why the one-factor model is a baseline with a matched budget. The emulator earns its place only if total-effect Sobol indices exceed first-order indices by a meaningful amount for at least some knob pairs. We expect this for structural and efficiency pairs, for example a tight complexity ceiling combined with a tight memory budget, because rolling-state dynamic-programming tricks raise complexity. If the gap is absent, we will recommend one-factor sweeps as the benchmark protocol.
Why not just prompt better, or lint afterward? Linters can fix naming and formatting but cannot change algorithm choice or decomposition. EffiBench reports GPT-4 code running 3.12× slower on average than canonical solutions, with worst cases at 13.89× slower and 43.92× more memory. Prompting strategies are the very thing whose failure edges we measure. We include chain-of-thought and NoFunEval-style concept-legend prompting (NoFunEval) as prompt conditions and compare their surfaces with the plain prompt. If better prompting moves the thresholds substantially, that is itself a finding about which knobs are prompt-addressable.
Decisive Experiments
Count versus severity. The central test is a two-by-two contrast at fixed total budget:
- five constraints, all at loose severity (at least 3× the reference value);
- five constraints with one knob at tight severity (below 1.0×);
- a single tight constraint;
- a single loose constraint.
Our claim predicts that five loose constraints nearly match the unconstrained reference in joint success, and that a single tight constraint does worse than five loose ones. The count model predicts the opposite ordering.
Threshold transfer. We estimate per-knob thresholds on half the EffiBench problem categories. We then test whether the same normalized threshold predicts collapse on the held-out categories and on HumanEval+.
Correctness-tax reconciliation. We place the operating points used by Beyond Pass@k and by SWE-IF onto our surface, as closely as their described instructions allow. We then check whether the fitted emulator reproduces both a ~1% tax and a 5–10% tax from their respective regions.
Preference-training effect. We compare the CodeDPO-style model with its base. Our prediction is that preference training moves the efficiency-budget threshold toward tighter budgets while leaving the structural thresholds unchanged. Such a pattern would show that alignment targets specific failure edges rather than improving quality uniformly.
References
- Beyond Correctness: Benchmarking Multi-dimensional Code Generation for Large Language Models
- SWE-IF: Aligning Code Evaluation with Human Preference
- Study on the vertical bearing performances of piles on karst cave
- Mining data from hemodynamic simulations via Bayesian emulation
- Beyond Pass@k: Developing Human-Centric Evaluation Metrics for AI Pair Programmers
- NoFunEval: Funny How Code LMs Falter on Requirements Beyond Functional Correctness
- EffiBench: Benchmarking the Efficiency of Automatically Generated Code
- Toward the Simplified Design of Single-Angle Beam-Columns
- CodeDPO: Aligning Code Models with Self Generated and Verified Source Code