Code-writing language models regularly produce programs that run correctly but are insecure: SQL built by string concatenation, weak cryptography, unsafe file handling. The most reliable fix today is an external loop. A security scanner reads the generated code, and its report is pasted back to a large model, which is asked to patch the flaws. That works for GPT-4-class models but is expensive. It is also unclear whether it works at all for the 0.5B–7B models that people actually want to run locally, inside editors or CI pipelines.
This proposal asks a single question. Can a small model be trained, with reinforcement learning, to use scanner and test feedback well across several repair turns, so that it closes the security gap without paying for it in functional correctness or in code bloat? And at what parameter scale does that learned skill stop beating the same small model wrapped in a hand-built external loop?
The core claim is specific. Suppose the reward is split across turns, so each repair step is credited for its net change in (tests passing, vulnerabilities flagged), and suppose that reward also carries a penalty on edit size. Then multi-turn reinforcement learning should teach 1.5B–3B models real security repair rather than the shortcuts small models fall into, such as deleting functionality, wrapping everything in exceptions, or silencing the scanner. We expect this to reach or exceed the best reported secure-pass@1 on CodeGuard+ while leaving general coding scores intact. We treat the "alignment tax", the loss of general capability that comes with safety training, as something to measure honestly rather than assume. The parameter-scale reasoning gap sets the regime we study. Domain specialization, meaning narrow training on security-relevant code with a security toolchain, is the method.
What Exists and What Is Missing
Three lines of work each cover part of this problem. None covers the combination.
External feedback loops. Leveraging Static Analysis for Feedback-Driven Security Patching in LLM-Generated Code (FDSP) feeds Bandit and CodeQL reports back to GPT-4 and CodeLlama through prompting. It leaves 7.4% (Bandit) residual vulnerabilities on PythonSecurityEval while preserving functionality in 96.2% of patched examples. Nothing is trained, and small models are not studied.
Security training that is single-turn. Instruction Tuning for Secure Code Generation (SafeCoder) improves security by about 30% with supervised losses on commit diffs, with no functional drop, but it only covers single-turn generation. Constrained Decoding for Secure Code Generation shows that prefix tuning trades correctness for security. It reaches 51.25% secure-pass@1 on CodeGuard+ with CodeGen-2.7B, above GPT-4's 47.45%, but it needs hand-written patterns for each vulnerability class.
RL-trained self-correction without security. ReflexiCoder trains an 8B model with reflection-aware GRPO (group relative policy optimization, an RL method that scores each sample against others drawn for the same prompt) to self-correct. It reaches 95.73% on HumanEval, but its reward is purely functional and it never goes below 8B. Domain-Adaptable Reinforcement Learning for Code Generation with Dense Rewards shows that a 1.5B model trained with linter-derived dense rewards gains 19.3 points on MBPP (46.0% to 65.3%), but it optimizes single-pass generation only.
Our proposal differs from all of these. It is the first to train compact models to repair security flaws across turns from analyzer feedback, and to measure where that skill breaks down as models shrink.
Two negative findings shape the design. SlopCodeBench shows that models iterating on their own code accumulate structural erosion in 77% of trajectories and verbosity in 75.5%. Separately, Do Users Write More Insecure Code with AI Assistants? finds that iterative AI-assisted loops raised SQL-injection rates to 36% versus 7% for controls. Iteration therefore does not help by default, so a naive multi-turn reward is likely to teach bloat. This is why the edit-size term is central to our design and not decoration.
On scale, Effective Learning for Small Reasoning Models finds that at 0.5B, distillation alone collapses GSM8K accuracy from 45.5% to 18.7%, while direct GRPO improves results. That pushes us toward RL from the small model's own samples rather than imitation of a frontier teacher.
Research Program
Training tasks. We build a security-seeded task pool in the style of SelfCodeAlign, which generates instructions from seed concepts and validates them with self-generated tests in a sandbox. Our seeds are weakness classes (CWE types, i.e., standard categories of software vulnerability) together with SafeCoder-style commit pairs. Every task carries unit tests. CodeGuard+, PythonSecurityEval, and CodeLMSec are fully held out.
The repair episode. The policy writes code. The environment runs the tests and a static analyzer and returns both reports. The model then revises, for up to $T=4$ turns.
The reward. Let $f_t$ be the fraction of tests passed and $v_t$ the number of analyzer findings at turn $t$. Each turn receives
$$r_t = \alpha\,(f_t - f_{t-1}) + \beta\,(v_{t-1} - v_t) - \lambda\,\frac{|\Delta_t|}{|c_{t-1}|},$$
where $|\Delta_t|$ is the size of the line-level diff, $|c_{t-1}|$ is the length of the previous program, and $\alpha,\beta,\lambda$ are weights. A terminal bonus is paid if the final program is both correct and flagged clean.
Anti-gaming rules are part of the design. We strip suppression comments such as # nosec. We reward only reductions confirmed after re-analysis. We train against Bandit but evaluate with CodeQL as well, so that "fooling the scanner" shows up as a gap between the two.
Models. We train Qwen2.5-Coder at 0.5B, 1.5B, 3B and 7B with identical recipes. Holding the recipe fixed turns the scale gap into a measured curve rather than an assumption.
Measuring the tax. We measure the alignment tax off-distribution, on HumanEval+, MBPP+ and LiveCodeBench. An optional arm projects the security gradient away from a capability matrix built from general coding samples, following Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimization (NSPO). NSPO held degradation within 1% for conversational safety on Llama3-8B but never tested code security. We include it only as an ablation, for reasons given below.
Baselines
Do-nothing reference. The first reference point is each base model generating code once, with no repair. This anchors both the security gain and any functional loss. We also report the same base model given a plain "write secure code" instruction, which costs nothing and is what practitioners try first.
Strongest external competitor. The strongest competitor on our actual question is an FDSP-style external loop run on the same small model. It gets the same analyzer, the same four turns, and the same prompt-tuning effort we spend on our own system prompt. This isolates whether training adds anything beyond giving the model the feedback.
No reported result was found on Lacuna for FDSP applied to sub-7B models. The closest number is GPT-4 with FDSP at 7.4% residual Bandit findings and 96.2% functional preservation on PythonSecurityEval. We treat that as the large-model ceiling to approach.
Strongest trained competitor. On CodeGuard+, the strongest trained or decoding-time competitor is constrained decoding at 51.25% secure-pass@1 on CodeGen-2.7B, from Constrained Decoding for Secure Code Generation. We rerun it with our 3B base model where its patterns apply. Our 3B model beating that figure, and beating the rerun, on the unseen weakness classes is what counts as a win. On the classes its patterns cover, parity is enough.
SafeCoder-style SFT. We train SafeCoder-style supervised tuning on the same commit pairs and the same task pool, with matched GPU-hours. This is the supervised alternative.
Single-turn RL. The last baseline is single-turn GRPO, with the reward being the final correct-and-clean indicator. It resembles the dense-reward setup but has no repair turns, and it receives the same total number of sampled programs as our multi-turn policy.
Ablations of the key ingredients. We remove the per-turn credit, keeping only the terminal reward. We remove the edit penalty ($\lambda=0$). We add NSPO-style projection. We also remove analyzer feedback at test time, which tests whether the skill has moved into the weights as ReflexiCoder claims for functional bugs.
Why not just…?
Why not just run the external scanner loop on the small model? This is the most serious alternative. Prior work suggests small models handle complex meta-instructions poorly. Mitigating Boundary Ambiguity and Inherent Bias for Text Classification in the Era of Large Language Models reports that chain-of-thought prompting hurts models below 10B, so parsing a CWE report and editing surgically may be exactly what an untrained 1.5B model cannot do. That is a hypothesis, not a result. The matched external-loop baseline settles it directly at every scale. If the untrained loop matches our trained model at 3B and 7B, we will report that the training benefit is confined to the smallest scales.
Why not just distill repair trajectories from GPT-4 running FDSP? Effective Learning for Small Reasoning Models shows distillation damaging 0.5B models, and SelfCodeAlign reports that self-generated data beat distillation-heavy baselines at 7B. Those results come from math and functional code, though, not security repair. We therefore include a distillation arm, with SFT on frontier FDSP trajectories and matched compute, rather than dismiss it.
Why not just train single-turn with a security reward, since the tests already guard correctness? This is a real threat too. A joint correct-and-clean reward may already prevent the on-distribution tax, and iteration may add little. The single-turn GRPO baseline, at an equal sample budget, answers this. If it matches the multi-turn policy, the contribution shrinks to the scale study, and we will say so.
Why not just mix general coding data into training instead of null-space projection? NSPO reports 5–10% drops with ordinary methods versus under 1% with projection, but in conversational safety. In our setting the functional tests inside the reward may make data mixing sufficient. That is why projection is an ablation and not the core. The HumanEval+ and LiveCodeBench deltas under data mixing versus projection decide whether it earns its cost.
Proposed Experiments and Decisive Checks
Security. The headline evaluation is secure-pass@1 on CodeGuard+, plus residual Bandit and CodeQL findings on PythonSecurityEval and SecurityEval. We also count vulnerable completions on the 280 CodeLMSec prompts from CodeLMSec.
Functional cost. We report pass@1 on HumanEval+, MBPP+ and LiveCodeBench, with the dense-RL MBPP figure of 65.3% at 1.5B as a reference for what a specialized 1.5B model can hold.
Erosion. We compute SlopCodeBench's erosion and verbosity measures on the repair trajectories themselves, and on SlopCodeBench checkpoints, to test whether the edit penalty keeps growth close to that of the human reference.
Decisive check: unseen vulnerability classes. We train on half of the weakness classes and test on the rest. This is where hand-written constraints cannot follow, and it is the clearest separation from constrained decoding.
Decisive check: scanner gaming. We compare Bandit-clean rates against CodeQL-clean rates. If the Bandit-trained model shows a much larger drop in Bandit findings than in CodeQL findings, it learned to fool its trainer, not to fix code.
Decisive check: scale curve. We plot, for each model size, the margin of trained repair over the matched external loop. The proposal's main scientific output is where that margin appears and where it vanishes. A flat, zero margin across sizes would be a clean negative result, showing that security feedback helps small models only as a prompt.
References
- Leveraging Static Analysis for Feedback-Driven Security Patching in LLM-Generated Code
- Instruction Tuning for Secure Code Generation
- Constrained Decoding for Secure Code Generation
- ReflexiCoder: Teaching Large Language Models to Self-Reflect on Generated Code and Self-Correct It via Reinforcement Learning
- Domain-Adaptable Reinforcement Learning for Code Generation with Dense Rewards
- SlopCodeBench: Measuring Code Erosion Under Iterative Specification Refinement
- Do Users Write More Insecure Code with AI Assistants?
- Effective Learning for Small Reasoning Models: An Empirical Study on 0.5B Reasoning LLMs
- SelfCodeAlign: Self-Alignment for Code Generation
- Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimization
- Mitigating Boundary Ambiguity and Inherent Bias for Text Classification in the Era of Large Language Models
- CodeLMSec Benchmark: Systematically Evaluating and Finding Security Vulnerabilities in Black-Box Code Language Models