Rerouting Pair Memorization Out of Kernel-Writing Adapters

Language models write GPU kernels poorly in languages like Triton and CUDA because almost no training code exists for them; CUDA is about 0.073% of The Stack (KernelBench). The hardest kernels to write are fused ones, where several operators (say a convolution, a bias add and an activation) are merged into one kernel so intermediate results never leave fast memory. With so little data, a cheaply fine-tuned model can do one of two things. It can learn fusion as a reusable skill that combines operator building blocks. Or it can memorize the specific operator pairs it happened to see. Vision-language models make the second mistake with words: they memorize frequent phrases like "white cat" instead of binding attributes to objects. We expect kernel adapters trained on scarce data to make the analogous mistake, and nobody has measured it.

This proposal does two things. First, it builds compositional train/test splits of kernel-fusion tasks, so that every operator is seen in training but certain operator pairs are never fused together until test time. This measures how much of a small adapter's fusion ability survives on unseen combinations. Second, it tests one specific fix borrowed from image personalization. During training, a shared low-rank adapter is paired with tiny per-combination adapters whose job is to absorb pair-specific idiosyncrasies. The per-combination adapters are thrown away at test time. A low-rank adapter (LoRA) here means a small trainable add-on to a frozen model's weight matrices, written as a product of two thin matrices. The claim is that discarding the absorbers leaves a shared adapter that fuses unseen operator pairs better than a plain adapter trained on the same data with the same parameter count.

Core Claim

Under scarce domain-specific language data, parameter-efficient fine-tuning learns operator fusion partly as memorized compounds, which produces a measurable seen-versus-unseen gap on held-out operator pairs. That gap can be narrowed without new data by "shortcut rerouting." The idea comes from Preventing Shortcuts in Adapter Training via Providing the Shortcuts, which adds auxiliary modules during diffusion personalization to explain away pose and background, then discards them so the main adapter keeps only the target identity. That paper never touches code or compositional splits. We transplant the mechanism so that the confounder being explained away is compound identity itself.

Why this framing, and what prior work constrains it

The motivation comes from the vision-language compositionality literature. When and Why Vision-Language Models Behave like Bags-Of-Words shows that contrastive models ignore word order, with order sensitivity on COCO-Order at 46% for CLIP. CREPE separates systematicity (unseen compounds of seen atoms, with relative Recall@1 drops of up to 12%) from productivity (longer compositions, near chance at 9+ atoms). Deciphering the Role of Representation Disentanglement ties out-of-distribution compositional success to how independently attributes and objects are represented. We reuse CREPE's two axes directly: held-out operator pairs test systematicity, and longer operator chains test productivity.

ColorSwap adds a second point. Fine-tuning CLIP on only 1,400 targeted pairs lifted its Group Score from 11.67% to 63%, so the failure is a training bias rather than a hard ceiling. That is why a training-time intervention is plausible.

On the code side, the research file describes the KernelBench authors as finding that models know what to fuse but implement it badly. AutoTriton shows that supervised fine-tuning on about 14,000 distilled Triton examples followed by reinforcement learning works well. Neither paper asks whether the learned fusion skill transfers to operator combinations absent from training.

Two adapter papers constrain the design. Beyond Adapter Retrieval composes LoRA weights by a sparse linear combination of task prototypes. That raised zero-shot segmentation DICE from 19.2% to 64.5%, but only in continuous output domains. Linearly blending weight matrices has no reason to respect the discrete grammar of a kernel, so we treat weight composition as a baseline expected to fail on syntax, not as our method. Asymmetry in Low-Rank Adapters shows that training only the up-projection $B$ while freezing a random orthogonal down-projection $A$ gives tighter generalization bounds and better out-of-domain robustness. That is a cheaper competing explanation for any gain we see, so it enters as both a baseline and a combinable variant.

Research Program

Benchmark construction. We take the single-operator vocabulary of KernelBench Level 1 and the 3–6 operator sequences of Level 2. From these we generate PyTorch reference programs over a grid of operator chains. We assign them to splits in the style of CLEVR-CoGenT, as used by Re-Thinking Inverse Graphics With Large Language Models. Every operator appears in training both alone and fused with some partners, but a held-out set of adjacent operator pairs never co-occurs in any training chain. A second productivity split trains on chains of length 2–3 and tests on lengths 4–6.

Training kernels are produced the way AutoTriton built its data, through reasoning-model distillation and torch.compile extraction. Every kernel is kept only if it compiles and matches the PyTorch reference numerically on randomized inputs. Total training size is capped at a few thousand verified examples to stay in the scarce regime.

Method. The frozen base model's weights $W$ are augmented with a shared adapter $S$ and, for each training compound cluster $c$ (all chains containing a given fused operator pair), a private adapter $P_c$ of much lower rank. The effective weight during training on an example from cluster $c$ is

$$W + S + m \cdot P_c,$$

where $m \in \{0,1\}$ is a random mask that switches $P_c$ off with probability $p$. The mask forces $S$ alone to keep producing valid code, which prevents the degenerate solution where $P_c$ absorbs everything. At test time only $W + S$ is used. The design rationale follows the shortcut-rerouting paper: whatever is cheaply predictable from compound identity is easiest to store in $P_c$, so gradient pressure on $S$ concentrates on what is shared across compounds. The parameter count of $S$ is matched exactly to every single-adapter baseline.

Analysis. Beyond accuracy, we probe whether the hidden states produced under $S$ separate operator identity from position in the chain. We use the disentanglement measures from the CLIP disentanglement paper, applied to pooled per-operator token states. The goal is to see whether any accuracy gain comes with the representational change the theory predicts.

Baselines

The naive reference point is the frozen base model, Seed-Coder-8B (the base AutoTriton used), prompted zero-shot. The base model with no adaptation shows how much of the seen/unseen pattern comes from pre-training.

The main naive adapter is a single standard LoRA trained on the identical verified data. Its rank equals the rank of $S$, and it gets the same learning-rate and rank sweep and the same number of GPU hours. If this baseline shows no seen-versus-unseen gap, the core premise is false and we report that as the finding.

The strongest existing method is an AutoTriton-style pipeline: supervised fine-tuning followed by reinforcement learning with syntax and execution rewards, rerun on our capped training split so it does not see held-out compounds. For reference, AutoTriton reports 39.16% execution accuracy on TRITONBENCH-T and 97.0% compilation accuracy on KernelBench Level 2 for AutoTriton-8B. There is no reported result found on Lacuna for any method on a held-out-compound split of kernel fusion, so the comparison has to be run fresh. We run it fully only if budget allows. Otherwise we run a reduced reinforcement-learning budget and state the cap explicitly.

A Beyond-Adapter-Retrieval-style baseline trains one LoRA per primitive operator. For an unseen pair it builds an adapter by $\ell_1$-sparse combination of the primitive adapters, weighted by prototype similarity.

The Asymmetry baseline trains only $B$ with a frozen random orthogonal $A$, at matched parameter count.

Two ablations isolate the key ingredient. The first assigns private adapters to random example groups instead of compound clusters. This keeps the extra training-time capacity and the masking but removes the link to compound identity. If it matches our method, the gain is regularization rather than rerouting. The second removes the mask ($p=0$).

"Beating" a baseline means a smaller seen-minus-unseen execution-accuracy gap and higher absolute unseen-compound execution accuracy, with seen-compound accuracy not lower by more than seed noise. Significance is assessed across at least three seeds and two different random choices of held-out pairs.

Why not just…?

Why not just generate training data for the unseen combinations? This is the most serious threat. Reference kernels for any PyTorch chain can be synthesized and execution-verified, and AutoTriton does exactly this kind of extraction. If augmentation is cheap, a compositional gap might simply be a coverage problem. We do not argue around this. We measure, for the naive LoRA, how many additional verified compounds per held-out pair are needed to match the rerouted adapter's unseen accuracy. If a handful suffice, the method's value is limited to settings where verified references are expensive. That would point to languages without a reference compiler, such as Verilog designs judged against hand-written testbenches as in VeriGen, and we would say so.

Why not just use compiler and profiler feedback at inference time? KernelBench reports that ten refinement turns raise DeepSeek-R1's Level 2 success from 36% to 72%, so iterative repair is powerful. It is orthogonal to how the adapter was trained, though. We report every method both single-shot and under an identical ten-turn feedback budget. The question is whether rerouting still helps once refinement is applied. If the gap disappears under refinement, the method only buys single-shot efficiency, and we report it as that.

Why not just compose per-operator adapters, or use B-only training? Both appear as baselines above. Weight composition is expected to fail on compilation, because nothing in a linear blend respects syntax; the Beyond Adapter Retrieval paper itself only tests continuous outputs. B-only training is a genuine competitor, because its out-of-domain advantage could account for any unseen-compound gain we see. We therefore also run rerouting with B-only $S$ to test whether the effects add.

Proposed Experiments and Decisive Checks

All methods are evaluated on the held-out-pair split and on the productivity split. Metrics are compilation rate, execution accuracy (numerical match to PyTorch on randomized inputs), and KernelBench's $\text{fast}_1$ relative to PyTorch Eager, so that correct but slow fusions are not counted as successes. Kernels that call PyTorch operators instead of Triton are flagged and counted as failures, following the reward-hacking concerns raised in AutoTriton and Kevin. We also report results on the standard TRITONBENCH-T split, to check that rerouting does not hurt in-distribution performance relative to the published 39.16%.

The work hinges on three outcomes. The first is that a plain LoRA shows a seen/unseen execution gap clearly larger than seed variance. The second is that the rerouted shared adapter narrows this gap and beats the random-grouping ablation, which rules out a generic regularization explanation. The third is that the improvement survives ten turns of compiler feedback and costs fewer additional training examples than the augmentation curve requires. If the first fails, the paper becomes a negative measurement result. If the second fails but the B-only baseline succeeds, the conclusion is that asymmetric adapters, not rerouting, are the right tool for scarce-data fusion.