Where Domain Pre-Training Belongs in a Team of Language-Model Agents

Complex technical tasks such as writing hardware circuits are increasingly handled by teams of language-model agents rather than a single model. In these teams one agent plans, another writes code, another writes tests, and another judges and debugs. Separately, a long line of work shows that a model pre-trained on a narrow technical corpus, for example Verilog hardware code, understands that domain far better than a generalist of similar size. The two ideas have never been combined in a controlled way. Every agent team in the literature is built from prompted generalist models, and every domain-pretrained model is evaluated alone. This note proposes to measure, role by role, where domain pre-training pays off inside an agent team. It also asks whether pre-training and task decomposition add up or partly cancel each other.

The core idea is simple. Build one agent pipeline for Verilog generation, and give each role a backbone that is either a generalist model or the same model after continued pre-training on Verilog. Then run every assignment of backbones to roles. Our hypothesis is that the benefit of domain pre-training is concentrated in the roles that touch domain semantics: the code writer, the testbench writer, and the debugger. The planner, by contrast, gains little. If this holds, a mixed team with a generalist planner and domain-pretrained workers should match an all-specialist team while costing less. It should also clearly beat an all-generalist team of the same size.

Core Claim

Two questions have been studied only in isolation, and we frame them jointly. The first is whether domain pre-training and decomposition are complements or substitutes. MAGE reports that its single-agent ablation on the same base model reaches 83.9% Pass@1 on VerilogEval-Human 2, against 93.6% for its multi-agent architecture. That gap is what decomposition buys on a strong generalist. The same paper reports 72.4% for standalone Claude 3.5 Sonnet, so a single agent with MAGE's tooling already recovers much of the gap to the full system. VerilogEval reports that fine-tuning models without Verilog pre-training yields limited gains.

If domain pre-training mainly fixes the syntax and concurrency errors that decomposition exists to catch, then a domain-pretrained single agent should close much of the decomposition gap. We call this the substitution hypothesis. If instead decomposition mainly buys separation between writing code and judging it against a simulator, the two effects should add. We call this the complement hypothesis.

The second question is which roles need the domain backbone. Plan-and-Act splits web agents into a planner and an executor. It finds that executor quality plateaus quickly while planner quality keeps scaling with data. That finding motivates our prediction that roles differ in what they need. Our prediction runs the other way from theirs for technical domains, however. Here the executor-side roles are the ones starved of domain knowledge, so domain pre-training should help them most. What would be new is a factorial answer to both questions under controlled training budgets. The practical result would be a recipe for small open agent teams in technical domains.

Research Program

The primary domain is Verilog, because correctness is checked by simulating against golden testbenches. This avoids the "feedback trap" described in MapCoder: when agents write their own tests without ground truth, they can "fix" correct code into broken code. The simulator gives every judgment an external check, and it also lets us measure how often the judge makes this mistake.

We start from one open generalist code model of about 7B parameters, called G. We create D by continued pre-training of G on a deduplicated corpus of public Verilog. The corpus is filtered against HDLBits-derived problems to prevent leakage into VerilogEval. We also create a control backbone, C, trained on the same number of tokens of general non-Verilog code. C separates the effect of "more training" from the effect of "domain training."

The pipeline follows MAGE's four roles: a testbench writer, a code writer, a judge, and a debugger. We add a lightweight planner that writes a natural-language design sketch, following the planning and execution split of Lumos. Each role receives a role-specific low-rank adapter, a small set of extra weights trained on top of a frozen model. The adapters are trained with identical data and hyperparameters on each backbone.

Role training data come from pipeline trajectories whose outcomes were confirmed by simulation, collected on problems disjoint from the evaluation set. Only the backbone differs across arms. We then evaluate all 2^5 = 32 assignments of G or D to the five roles. Because the adapters are shared, this costs only inference once the ten adapters are trained. We also evaluate a single-agent configuration trained on the same trajectories flattened into one prompt, on each of G, D, and C.

The analysis fits a simple additive model of pass rate over the role assignments. This yields a per-role "domain dividend," the gain from switching that role's backbone from G to D. It also yields the interaction between backbone and decomposition. We track four outcomes. The first is Pass@1. The second is pass@5 and pass@10, because VerilogEval documents diversity collapse under domain fine-tuning. The third is tokens consumed per solved problem, because Single-agent or Multi-agent Systems? Why Not Both? reports 4x to 220x more prefill tokens for multi-agent systems. The fourth is the rate at which the judge or debugger turns simulator-passing code into failing code.

A secondary domain tests whether the pattern transfers to formal verification. We use VERINA (sunblaze-ucb/verina), a Lean 4 benchmark that splits tasks into writing code, writing specifications, and writing proofs. The same design applies, with continued pre-training on Lean 4 code and proofs and the Lean compiler as ground truth. VERINA reports that o3 reaches 72.6% on code, 52.3% on specifications, and 4.9% on proofs. The proof step is the bottleneck, so it is the natural test of whether domain pre-training helps a role that "superficial decomposition" leaves stuck.

Baselines

The naive reference point is the generalist backbone G used as a single agent, trained on the flattened trajectories. It anchors every gain.

The strongest existing method is MAGE, which reports 95.7% Pass@1 on VerilogEval-Human 2 with Claude 3.5 Sonnet. VerilogCoder reaches 94.2% in the same paper. These frontier numbers are near the benchmark ceiling. We report them as the reference, but we do not claim a 7B team will beat them. We found no reported result on Lacuna for open models of about 7B inside a MAGE-style pipeline on VerilogEval-Human 2. For codegen-16B-verilog-sft, VerilogEval says only that it is "comparable to GPT-3.5," with no exact figure. We will therefore rerun MAGE's prompts on G itself, with the same sampling budget, as the fair open-model comparison.

Beating this baseline means one of two things. An all-D or mixed team could exceed prompted-MAGE-on-G by a margin larger than the seed-to-seed spread. Alternatively, a 7B mixed team could come within a stated margin of the 95.7% frontier result at a small fraction of its per-problem tokens.

The ablation that isolates the key ingredient is the C backbone: the same continued pre-training compute spent on non-Verilog code. If C matches D in any role, then that role's dividend comes from training volume, not domain exposure.

A second ablation replaces continued pre-training with extra role-specific adapter training on G, spending matched compute. This follows the step-specific adapters of Type-Compliant Adaptation Cascades, which raised Qwen 3 8B on FinQA from 12.0% to 24.7% without changing the base model.

The cost baseline is the agent cascade of Gao et al., in which the team runs only when a single agent fails. Gao et al. report accuracy gains of 1.1% to 12% for cascading and cost savings of up to 88.1% for the related routing strategy. All arms share the same problems, sampling temperature, number of candidates, simulator-call cap, and hyperparameter search budget.

Why not just…?

Why not just prompt a frontier model inside MAGE? On VerilogEval it already reaches 95.7%, and we say so plainly. Our question is not whether frontier models can do this. It is whether small open models can, and what makes them able to. Where frontier scaffolds clearly fail, as in VERINA proof writing where 64 rounds of compiler feedback reach only about 20%, the per-role question is open and consequential.

Why not just put a domain-pretrained model in a single agent? This is the substitution hypothesis. It is a real threat, because domain training might remove most of the errors decomposition catches. The monolithic-D arm tests it directly. If monolithic D reaches the Pass@1 of the decomposed-D team at lower token cost, we will report that decomposition is unnecessary once the backbone knows the domain. That would itself answer the question raised by Gao et al.

Why not just train role adapters on a general model, as in the typed-cascade work? A Comparative Study of PEFT Methods for Python Code Generation found pre-training exposure a stronger predictor of code quality than the choice of fine-tuning method. That study concerned single-model generation, however, not roles within a pipeline. The compute-matched adapter arm settles the question for each role.

Why not just cascade or route to cut cost? Cascading is orthogonal to our question and can be applied on top of any backbone assignment. We report every configuration with and without cascading, so a reader can see whether the savings from a mixed team survive once cascading is used.

Proposed Experiments

The decisive analysis is the 32-configuration sweep on VerilogEval-Human 2 with three seeds, followed by the per-role dividend analysis.

Four outcomes would confirm the thesis. First, the dividends for the code writer and debugger should be significantly positive and larger than their C-backbone counterparts. Second, the planner's dividend should be indistinguishable from zero. Third, the mixed team (planner G, rest D) should match all-D within seed noise. Fourth, decomposition should still add a clear gain on D over monolithic D, which is the complement hypothesis.

Two outcomes would refute the thesis. If the dividends are uniform across roles, role-asymmetric teams offer nothing, and the practical recipe collapses to "use D everywhere." If monolithic D matches decomposed D, decomposition is redundant once the backbone knows the domain.

Two secondary checks follow. We check whether D reduces the judge's harmful-fix rate, which would show that domain pre-training improves judgment and not only generation. We also check whether D preserves pass@10 relative to G, since VerilogEval documents diversity loss under domain fine-tuning.

We then repeat the reduced design on VERINA with three roles. There we ask whether a domain-pretrained proof writer moves the 4.9% proof-writing bottleneck at all. Gains in the code and specification roles alone would support the "superficial decomposition" reading in VERINA: the pipeline structure is fine, but the base model's reasoning at the proof step is missing.