Small language models specialized for a technical domain can match giant generalist models. A 7-billion-parameter Verilog model trained with fine-grained simulation feedback reaches 81.4% pass@1 on VerilogEval 1.0 Machine, ahead of GPT-4o's 67.7% and DeepSeek-v3's 77.6%. Yet these specialists still keep the uniform architecture of the general model they came from. Every layer keeps the same number of attention heads and the same feed-forward width, even though the domain may need only part of that machinery.
Architecture search tools can already cut a large model into a faster, irregular one. The most scalable of them, Puzzle, trains cheaper replacement variants for each layer and assembles the best mix under a latency budget. It decides which variants to keep by how closely each one imitates the original layer's outputs. We propose to keep that machinery but change what it scores. Each candidate layer variant is judged by whether the model still writes Verilog that passes simulation testbenches, not by how well it imitates the parent. The claim is that for a verifiable domain, imitation error is the wrong signal for choosing what to cut, and functional correctness is the right one. It should let the search remove more of the model at the same pass rate.
Core Claim
Start from a domain specialist with a checker: a compiler plus simulation against reference signals. The capacity that matters for passing tests is a small, irregularly placed subset of the network. Choosing that subset by measured functional correctness, then running a single shared distillation pass, should give a faster model than choosing by distillation loss on domain text. The comparison is made at equal latency and memory on the same GPU. We also expect it to beat a smaller off-the-shelf model put through the same specialization pipeline. The proposal is not "run NAS on domain data." That version would be Puzzle with a different calibration set, and we treat it as a baseline. What is new is using execution feedback as the selection criterion inside the layer-assembly step of architecture search. As the research file notes, no existing large-model search does this: InstaTune, MODNAS, Puzzle and Jet-Nemotron all score candidates by cross-entropy, divergence from the teacher, or hardware measurements.
Where This Sits
Two groups of prior work motivate the idea. The first shows that model capacity is used unevenly across layers. Puzzle solves a mixed-integer program over per-layer variants and finds that the best large models keep their middle layers intact and prune the outer ones hard. Neural Architecture Search for Parameter-Efficient Fine-tuning finds that trainable adapter parameters concentrate in middle-layer feed-forward modules. Its non-uniform bias selection scores 84.6 GLUE against 74.3 for uniform selection at the same 25k-parameter budget. Both papers work on general data and study either inference cost or training parameters, never domain function. Jet-Nemotron adds a sharper hint: math reasoning needs more retained full-attention layers than general knowledge does. Which layers matter depends on the task, and so may depend on the domain.
The second group supplies the domain and the checker. VeriGen fine-tunes CodeGen models uniformly on curated Verilog. VerilogEval shows that supervised fine-tuning raises pass@1 while lowering pass@5 and pass@10, so any compression must be checked for loss of output diversity. QiMeng-SALV turns simulation into signal-level partial-credit rewards through abstract-syntax-tree tracing, and releases code, model and data (QiMeng-SALV repository). We reuse its signal-level scorer as our measurement instrument. It changes only weights, never the architecture, and we change the architecture.
Some prior work constrains the design. Puzzle reports that layer-wise local distillation fails to preserve long-range dependencies beyond 64K tokens, so we limit claims to module-scale contexts. DARTS for Inverse Problems warns that search outcomes are unstable across runs, so every search is repeated with different seeds and problem subsets. The cross-domain search methods AdaptNAS and DAMPC-NAS argue that architecture itself carries domain transfer, but only for vision classifiers. We do not borrow their adversarial alignment, because our target is a single domain rather than invariance across domains.
Research Program
The parent model is the released SALV Qwen2.5-Coder-7B specialist. Following Puzzle, we build a library of replacement variants for each of its layers: reduced attention-head counts, reduced feed-forward widths, attention removed, and the whole layer skipped. Each variant is trained by local distillation to mimic its parent layer, using Verilog text from the CodeV-derived SALV training data.
The departure comes at the scoring step. For each single-variant substitution, we generate solutions to a held-in pool of training problems drawn from the SALV dataset. These problems have reference modules, and none overlap with VerilogEval or RTLLM. We score the outputs with the SALV signal-level checker, which gives a smooth fraction of correct output signals rather than a binary pass. The score for a variant is the drop in this signal-correctness relative to the parent. We then solve the same knapsack-style mixed-integer program Puzzle uses, with measured latency and memory on the target GPU as constraints and the summed functional drop as the objective.
Single-swap effects may not add up. To correct for this, we run two or three rounds of re-scoring: each round re-measures the variants in the currently chosen assembly, following the logic of iterative refinement. The assembled model then gets the same global distillation budget, and optionally the same SALV preference-tuning pass, as every baseline.
The program also has an analysis arm. We compare the layer maps chosen by functional scores with those chosen by imitation loss, using both domain text and general text. The question is whether the differences are systematic, for example sequential-logic problems depending on specific attention layers. We then test whether the functional map predicts which problem categories degrade under the imitation-loss map.
Baselines
The naive reference points are the uncompressed SALV-7B parent and uniform compression of it. The parent sets the quality ceiling and the latency to beat. Uniform compression means shrinking every layer's heads and feed-forward width by the same ratio to meet the latency target, then running the same distillation. Puzzle's own finding of strong non-uniformity predicts that uniform compression loses clearly. Beating it only shows that non-uniformity helps, which is already known, so it establishes nothing new.
The strongest existing method is Puzzle-style assembly scored by local distillation loss, run on Verilog text. It is the published large-scale search, and domain calibration is the obvious adaptation of it. Puzzle reports 98.4% retention of parent benchmark capability with 2.17× throughput on a single H100 for Nemotron-51B. Those figures come from general benchmarks. We found no reported result on Lacuna for any architecture-searched model on VerilogEval or RTLLM.
To keep the comparison fair, both searches use the identical variant library, latency constraint, global distillation tokens and preference-tuning pass. They differ only in the scoring signal, so this baseline doubles as the key ablation. We add a third arm scored by imitation loss on general code text, to separate "domain data" from "functional signal." A fourth arm scores by binary pass/fail instead of signal-level credit, to test whether dense partial credit is what makes the scoring practical.
The strongest simpler competitor is a smaller dense model from the same family, Qwen2.5-Coder at 1.5B or 3B. We pick whichever matches the carved model's measured latency, and put it through SALV's full pipeline on the same data with the same token budget.
We count the proposal as winning under a single pre-registered condition. At matched measured latency and memory, the functionally scored model must exceed both the imitation-loss search and the dense small model on VerilogEval 1.0 Machine pass@1 and RTLLM v1.1 pass@1, beyond the seed-to-seed spread of three independent searches. It must also not lose on pass@10. The external anchors are SALV-7B's reported 81.4% on VerilogEval 1.0 Machine and 62.6% on RTLLM v1.1. We will report how much of each the carved model retains.
Why not just…?
Why not just fine-tune a smaller off-the-shelf model with the SALV pipeline? This is the real threat, and the file offers no evidence against it. If a 3B dense model trained the SALV way matches the carved model at the same latency, the architecture search added nothing. The matched-latency dense baseline settles the question directly. We will report the result either way. A tie would itself be informative: it would mean domain capacity lives in weights rather than in layer placement.
Why not just run Puzzle on Verilog text? It is the cheaper, obvious move, and it may capture most of the gain. Imitation loss treats all output differences as equally bad, including stylistic differences in comment text or naming that never affect simulation. It can also tolerate small shifts that flip a signal's logic. The scoring-signal ablation, run with everything else held fixed, is exactly the test of whether this distinction matters. If the two layer maps agree and the scores tie, the claim fails.
Why not just quantize the SALV model? Quantization reduces memory and latency without any search. It is orthogonal to changing the architecture, so we apply the same quantization to the parent, to the carved models and to the dense small model. We then report whether the carved model's advantage survives it.
Why not just use Lawton-style non-uniform adapters? That method decides which parameters to train, while every layer still runs at inference. It does not speed the model up, so it answers a different question from ours.
Proposed Experiments
Evaluation uses VerilogEval 1.0 and 2.0 and RTLLM v1.1 and v2.0 at pass@1, pass@5 and pass@10. Diversity loss is flagged by the VerilogEval pass@k gap. We report measured tokens per second, time to first token and peak memory on a single GPU at fixed batch sizes, along with total search cost in GPU hours. We will also report VerilogEval results broken down by problem category, to see whether sequential and state-machine problems suffer most under the imitation-loss map.
The decisive check is the head-to-head above at two or three latency targets, repeated over three search seeds. As a secondary check, we swap only the most divergent layers between the two maps. If performance on the affected problem categories moves accordingly, the layer placement is doing causal work, not mere correlation. The whole program is sized to a small parent and short global distillation, far below the roughly 18,000 H100 hours Jet-Nemotron reports. That makes the result a direct test of whether checker-scored selection earns its place.
References
- Puzzle: Distillation-Based NAS for Inference-Optimized LLMs
- InstaTune: Instantaneous Neural Architecture Search During Fine-Tuning
- Large Language Model Compression with Neural Architecture Search
- Jet-Nemotron: Efficient Language Model with Post Neural Architecture Search
- Neural Architecture Search for Parameter-Efficient Fine-tuning of Large Pre-trained Language Models
- VeriGen: A Large Language Model for Verilog Code Generation
- Invited Paper: VerilogEval: Evaluating Large Language Models for Verilog Code Generation
- QiMeng-SALV: Signal-Aware Learning for Verilog Code Generation
- DARTS for Inverse Problems: a Study on Stability
- Adapting Neural Architectures Between Domains
- Domain Adaptation by Maximizing Population Correlation with Neural Architecture Search