Does Domain Pre-Training Turn Fine-Tuning into Amplification?

When a small language model is specialized for a narrow technical job such as writing GPU kernels, practitioners face a costly choice. They can first continue pre-training the model on a large pile of domain text, or they can go straight to fine-tuning on task examples, either by imitating reference solutions (supervised fine-tuning, SFT) or by reinforcement learning (RL) against a compiler and unit tests. The behavioral evidence favors pre-training. Domain exposure during pre-training predicts code-generation quality better than the choice of fine-tuning method, according to A Comparative Study of PEFT Methods for Python Code Generation. Nobody knows why this holds inside the model, though, and nobody has a cheap way to tell, before spending the compute, whether a given base model is "ready" to be specialized.

This note proposes a simple mechanistic account and a test. Interpretability work on fine-tuning finds that post-training mostly amplifies machinery the model already has rather than building new machinery. We hypothesize that this is only true when the base model has already seen the domain. On a model that has seen enough domain data, RL or SFT gains should be recoverable by a tiny intervention on the base model's internal activations, and fine-tuning should overwrite little. On a model that has not, fine-tuning must build new pathways, and that construction is where the forgetting caused by SFT comes from. We measure this "amplification ratio" along a controlled dose curve of domain pre-training. We then test whether it predicts downstream gain and forgetting better than the off-the-shelf signals people use today.

Core Claim

Define the amplification ratio of a post-trained model as the fraction of its task gain over its own base that can be recovered by adding a single learned vector at one layer of the base model's residual stream. The residual stream is the running hidden state that every layer reads from and writes to. We claim three things. First, this ratio rises with the amount of domain continual pre-training. Second, the gap between RL's and SFT's preservation of general-purpose circuits shrinks as the ratio rises. Third, the ratio measured cheaply before full post-training predicts how much general capability SFT will destroy. It should predict this better than output divergence or representation-similarity scores do.

The claim has three possible outcomes, and each is informative. If the ratio is flat across pre-training doses, then "fine-tuning amplifies existing circuits" is a general property rather than something domain pre-training provides. If the ratio rises but does not predict forgetting, the mechanism story is right but useless as a diagnostic.

How This Relates to Prior Work

Three papers supply the finding we generalize. Fine-Tuning Enhances Existing Mechanisms shows that a math-tuned Llama-7B reuses and sharpens the same 72-head entity-tracking circuit as its base. Small Vectors, Big Effects shows that a single-layer steering vector closes over half the gap between a base model and its RL-trained reasoning counterpart. A Mechanistic Understanding of Alignment Algorithms shows that DPO routes around toxic neurons rather than deleting them. All three study tasks for which the general base model already had latent competence (entity tracking, grade-school math, toxicity). None varies how much the base model knows about the target domain, which is exactly the variable we manipulate. We reuse the steering-vector method of Sinii et al. as our measuring instrument rather than as a proposed product.

Two papers motivate the forgetting side. Mechanistic origins of catastrophic forgetting finds that SFT on SciKnowEval compresses Qwen2.5-3B-Instruct into a few specialist heads and retains 59.0% of the base circuit, while Dr.GRPO retains 72.5%. It also finds that output KL divergence misleads about circuit preservation. The study uses a single general checkpoint, so it cannot say whether domain pre-training changes this gap; our dose curve does. Its KL paradox is also why we treat output divergence as a baseline to beat rather than as ground truth. OpenRFT finds that an SFT-then-RL pipeline works best on SciKnowEval, which seems to contradict "SFT destroys circuits." We handle this by including the SFT→RL pipeline as an arm. Our hypothesis offers a resolution: SFT damage should be small when the base is already domain-ready, so the order matters less.

Two papers constrain the design. How Do LLMs Acquire New Knowledge? tracks knowledge circuits during continual pre-training, but only on synthetic subject–relation–object facts at 1.5B scale or smaller. Its finding that knowledge related to existing representations integrates faster motivates our prediction that domain-ready bases adapt by amplification. We move the question to unstructured code corpora and to what happens after pre-training. Layer Importance for Mathematical Reasoning is Forged in Pre-Training shows that the critical layers for math do not move under instruction tuning, distillation, or RL with verifiable rewards. It never tests continual pre-training, so we check whether pre-training dose moves them.

Finally, Resurrecting the Salmon supplies a tool. A sparse autoencoder decomposes activations into a large dictionary of interpretable features, and one trained on domain text captures 15–20% more variance than general-purpose ones and finds fine-grained technical features those miss. We use domain-trained sparse autoencoders to check whether the features that post-training upweights already exist in the base model.

Research Program

The domain is GPU kernel programming in Triton. It is genuinely low-resource for general models: CUDA is 0.073% of The Stack according to KernelBench. It also has execution-verified benchmarks and a published RL recipe in AutoTriton. SciKnowEval (SciKnowEval) serves as a replication domain, because it lets us connect directly to Rojas Nunez et al.

Starting from a general 3B base (Qwen2.5-3B, matching Rojas Nunez et al.), we build a dose curve of continual pre-training checkpoints. The checkpoints use zero, small, medium, and large token budgets of kernel code (Triton, CUDA, and the PyTorch code they implement). A fixed fraction of general replay text is mixed in so that dose is not confounded with forgetting from pure domain drift. Each checkpoint then receives three identical post-training treatments on the same task data: SFT alone, GRPO with compile and execution rewards following AutoTriton, and SFT followed by GRPO.

For every base–post-trained pair we measure five things. The first is task gain. The second is the amplification ratio, obtained by training single-layer steering vectors on the base, sweeping layers, and reporting the best. The third is circuit retention on general tasks, using Rojas Nunez et al.'s differential masking on attention heads, extended to MLP layers with edge attribution patching. The fourth is general retention on MMLU, HellaSwag, TruthfulQA, IFEval, and HumanEval. The fifth is whether the top features upweighted by post-training are already active in the base, using a sparse autoencoder trained on kernel code at a middle layer. We also compute a cheap "early" amplification ratio from a short post-training run, which is the version that could serve as a practical diagnostic.

Baselines

The naive reference point is the zero-dose checkpoint: a general base model post-trained directly with each treatment. Every dose effect is measured against it, and it is the regime in which prior mechanistic work was done.

The strongest existing behavioral system is AutoTriton's SFT-plus-GRPO recipe on a code-pre-trained 8B backbone. It reports 39.16% execution accuracy on TRITONBENCH-T (versus 30.12% for DeepSeek-R1-0528 and 15.06% for GPT-4o) and 97.0% compilation accuracy on KERNELBENCH Level 2 (AutoTriton). We do not aim to beat this number at 3B. We rerun the AutoTriton recipe at our scale on every checkpoint, with identical data, reward, rollout budget, and hyperparameter search, so it is the post-training method under study rather than a competitor. For the replication domain we compare against OpenRFT's reported 11% average gain on SciKnowEval, with the caveat that their base model differs. For the VerilogEval sanity check (codegen-multi versus codegen-verilog), there is no reported aggregate pass rate on Lacuna for this comparison, so we report our own.

For the diagnostic claim, the strongest competitors are the signals a practitioner would use to predict forgetting: output KL between post-trained and base models, linear centered kernel alignment between their hidden states, and the norm of the weight update. Rojas Nunez et al.'s circuit retention is the expensive gold-standard signal. The amplification ratio beats these signals if its rank correlation with measured general-benchmark forgetting across all dose-by-treatment cells is clearly higher, with bootstrap intervals that do not overlap. To isolate the key ingredient, we train steering vectors on the post-trained model's own activations, at random layers, and with shuffled task labels. If these recover as much gain, the ratio measures steerability in general rather than pre-existing domain structure.

Why not just…?

Why not just measure whether domain pre-training makes RL more effective, without any mechanistic analysis? We do measure this, and it is the headline behavioral curve. It cannot, however, distinguish "the base already had the pathways and fine-tuning turned them up" from "the base learned faster how to build new pathways." The two explanations make different predictions about forgetting and about which interventions suffice. The comparative PEFT study already establishes that pre-training helps behaviorally; it does not say how.

Why not just use output KL or representation similarity to predict forgetting? This is the real threat, because these signals are cheap. Rojas Nunez et al.'s finding that RL preserves circuits despite higher KL suggests output divergence is unreliable. Representation similarity has not been tested this way. We include both as baselines. If either predicts forgetting as well as the amplification ratio, we will report that the simpler signal suffices and scale back the diagnostic claim to the mechanistic finding alone.

Why not just steer the base model and skip post-training? On domain-ready bases our hypothesis predicts steering will recover much of the gain, so this is partly what we test. On low-dose bases, steering should fail if the domain structure does not exist, and the zero-dose ratio directly measures how badly. Sinii et al.'s own gap closure was "over 50%," not all of it, even on math where the base was competent.

Why not just mix in more general replay data to prevent forgetting? Replay addresses the symptom and is orthogonal to the question. We hold the replay fraction fixed across doses. If time allows, we add one high-replay arm to check that the dose effect on circuit retention survives it.

Decisive Checks

The central prediction fails if the amplification ratio does not increase with pre-training dose on TritonBench and KernelBench gains. It also fails if the SFT-versus-RL retention gap is the same at zero and large dose. The diagnostic claim fails if the early amplification ratio loses to KL or representation similarity in predicting forgetting. The sparse-autoencoder check should show that post-training upweights features already present in high-dose bases, while features newly appearing after post-training should concentrate in low-dose bases. The secondary test from Layer Importance is to ablate each layer at each dose and check whether critical layers for kernel correctness move with dose. If they stay fixed while circuit retention changes, then pre-training reshapes computation within layers rather than relocating it. That would be a sharper statement than either prior paper could make.