Chip designers spend most of a project, estimated at 60 to 75 percent of development effort, checking that the circuit does what the specification says (Learning Semantic Representations to Verify Hardware Designs). One of the hardest parts is writing assertions: short temporal-logic statements such as "two cycles after a request, grant must be high." A formal tool either proves an assertion or returns a counterexample trace showing it can be violated. Language models are weak at this. On AssertionBench, no evaluated model exceeded 44% functionally valid assertions, and up to 63% were disproven by the formal checker (AssertionBench). On the Comprehensive Verilog Design Problems benchmark, Claude 3.7 Sonnet scores 48% on writing hardware code but only 6% on writing testbench checkers (CVDP). Recent work on the neighbouring task of generating test inputs found that separately trained checkpoints of one model solve different subsets of problems, and that compressing them into a single dense model recovers only part of that combined advantage (CHORUS).
This note proposes to keep the specialists apart at inference time instead of merging them. We would build a mixture-of-experts model (a network whose layers contain several parallel sub-networks, or "experts," plus a small gating network, the "router," that chooses which experts process each input). Its experts are trained with formal-checker rewards on distinct families of temporal properties. The scoring counts an assertion only if it is proven on the correct design and also catches deliberately injected bugs, so the model gains nothing from trivially true statements. The central bet is that assertion families interfere with each other inside a single dense model. Off-by-one cycle errors and confusion between "same cycle" and "next cycle" implications are the typical symptoms. If that is right, routing should beat distillation, weight averaging, and sample-then-check ensembles at the same compute and the same formal-checker budget.
Core Claim
Assertion generation breaks into property families with conflicting surface conventions. Single-cycle combinational invariants, multi-cycle implications (the overlapping |-> versus non-overlapping |=> operators), reset and clock behaviour, finite-state-machine transitions, and datapath arithmetic relations all look alike on the page but need different timing arithmetic. We claim that a sparse mixture-of-experts model, upcycled from a dense Verilog-capable coder, beats three dense alternatives on mutant-killing assertion accuracy. The alternatives are the same experts distilled into one dense model, the same experts merged back into dense weights, and a dense model with matched total parameters. The model has one always-active shared expert for Verilog syntax and signal naming, plus routed experts each trained by reinforcement learning against a formal checker on one property family. The claim is falsifiable. If a dense model trained with CHORUS-style gated multi-teacher distillation matches the routed model within seed noise at matched training compute, then interference is not the bottleneck, and we report that.
This differs from CHORUS in three ways. CHORUS uses "experts" only as training-time teachers for a dense 4B student. It targets stimulus coverage rather than assertions. Its experts come from an ad-hoc curriculum rather than from a principled split of the task. It also differs from χFT, which upcycles a dense coder into a mixture-of-experts model with a shared expert and then merges the experts back into a dense model. χFT was tested only on Python. The merge-back step is exactly what our hypothesis predicts will hurt when the experts encode conflicting timing conventions.
Research Program
The base model is Qwen2.5-Coder-7B, the base used by QiMeng-CodeV-R1, whose distill-then-reinforce recipe we reuse for warm-up. We upcycle its feed-forward layers following χFT: copy them into one shared expert and six routed experts, and activate the top two routed experts per token. Supervised warm-up uses the 20,000 synthetic design-assertion pairs of VERT. VERT already labels structure (if-else, case, combined) and clocking (synchronous versus asynchronous). We extend these labels to the five property families above using the assertion's operators and cycle delays, which can be parsed deterministically.
Reinforcement learning then trains each routed expert on its own family, with the shared expert kept frozen or at a low learning rate. The reward is the mutant-jury idea from CorrectBench, transplanted from testbenches to assertions. A generated assertion earns reward only if the formal tool proves it on the correct design and it produces a counterexample on at least one mutated design (a flipped comparison, a shifted register delay, a dropped reset branch). This closes the loophole that AssertionBench's pass metric leaves open, where a trivially true assertion counts as a success.
Formal checks are expensive, so we borrow two ideas from LLM4Cov. The first is its memoryless single-step state, meaning the prompt contains the current design plus the latest checker output rather than the full conversation history. The second is its worst-state-prioritized sampling, which concentrates checker calls on failing cases. Formal checking uses Cadence JasperGold, as in AssertionBench, with a Yosys-based open flow (CVDP ships Yosys in its containers) as a fallback so the pipeline can be reproduced.
The second contribution is the router. We compare three variants. The first is a standard learned token-level router. The second is an instance-level router that chooses experts from structural features of the design and specification: counts of state registers, detected state machines, arithmetic operators, and reset style, computed from a control-data-flow graph in the spirit of Design2Vec. The third is a hybrid in which structural features bias the learned router. Decoding Knowledge Attribution in Mixture-of-Experts motivates this comparison. It found that routing in general-purpose mixture-of-experts models tracks attention semantics (r = 0.68). It also found that shared experts absorb much of the damage when routed experts are blocked in deep models: a 43% drop in mean reciprocal rank for Qwen1.5-MoE, against 76% for the shallower OLMoE. This is why we keep a shared expert and run expert-blocking probes. We then check whether our routed experts actually specialize by family, or whether the shared expert does all the work.
Baselines
The naive reference point is the base Qwen2.5-Coder-7B prompted few-shot, alongside the AssertionBench-reported ceiling. No evaluated model exceeded 44% accuracy on AssertionBench, and GPT-4o gained only 1.2× valid assertions going from 1-shot to 5-shot (AssertionBench). Beating the base model is necessary but uninformative.
The strongest supervised baseline is dense LoRA fine-tuning on VERT, as done in VERT. That paper reports a 96.88% improvement in valid assertions over base models and a gain of up to 24.14% in functional correctness over GPT-4o on its OpenTitan, CVA6, OpenPiton and Pulpissimo evaluations. We found no reported result on Lacuna for VERT-fine-tuned models on AssertionBench, so we run this ourselves. We then add the same formal-checker reinforcement learning to the dense model, with the same reward, the same checker-call budget and the same token budget.
The strongest method from the closest work is CHORUS-style adaptive multi-teacher on-policy distillation with a reward gate. Our trained family experts act as teachers for a dense 7B student, and the student learns on a task only from a teacher that beats its own mutant-killing reward. CHORUS reports 88.0% Pass@1 on CVDP-ECov against 74.5% for DeepSeek-R1 (CHORUS); we found no reported number for it on assertion tasks. This is the most direct test of whether routing matters or merely keeping specialists as teachers suffices. It receives the same experts, the same checker budget for distillation, and a tuning sweep of equal size.
Two further baselines complete the set. The first is χFT merge-back of our mixture into a dense 7B model. χFT's reported results are 67.1 HumanEval pass@1 at 1.3B and 81.7 at 6.7B, which is not a hardware number. The second is a training-free weight average ("model soup") of the family experts, which CHORUS found insufficient for stimulus generation.
To separate specialization from sheer parameter count, we train a dense model widened to match the mixture's total parameters, and a mixture whose experts receive random rather than family-based data partitions. The routed model beats the field only if it exceeds gated distillation and the random-partition mixture by more than three seeds' spread on held-out mutant-killing accuracy. It must do so while using no more formal-checker calls during training and exactly one generation per query at test time.
Why not just…?
Why not just use a bigger frontier model? The material consistently shows that scale does not fix hardware verification. DeepSeek-R1 at 671B trails a 4B specialist by 13.5 points on CVDP-ECov (CHORUS). Claude 3.7 Sonnet reaches only 6% on checker generation (CVDP). LLaMa3-70B got worse with more in-context examples on AssertionBench. We still include one frontier model as a reference row, but it is not the competitor that decides the question.
Why not just distill the experts into one dense model, as CHORUS does? This is the real threat. CHORUS recovers more than half of its oracle headroom this way, and a dense model is cheaper to serve. Our only argument is empirical. If property families truly conflict, the distilled student should improve on some families while regressing on others, and the routed model should not. The decisive experiment reports per-family accuracy for both models and a cross-family interference score: the accuracy drop on family A after adding family B's training data. If distillation shows no such interference and matches the mixture overall, we drop the routing claim and report a dense assertion model instead.
Why not just sample from every expert and let the formal checker pick? At test time the correct design is unknown, so "proven" alone rewards vacuous assertions. Mutant-based selection also needs K generations and K formal runs per query, and formal runs dominate cost. We therefore compare against this ensemble at matched checker calls per query, plotting accuracy as a function of the checker budget. If the ensemble dominates across that entire curve, routing loses. We expect it to win only at budgets well above one check per query.
Why not just fine-tune a dense model on more data? VerilogEval reports that synthetic supervised fine-tuning raised pass@1 but lowered pass@5 and pass@10, meaning the model produced less diverse outputs. CHORUS reports that single-model reinforcement learning saturates early. The widened dense baseline, trained on the same pooled data, tests this alternative directly.
Proposed Experiments
The primary evaluation is AssertionBench's 100 OpenCores designs. We report its native pass, counterexample and syntax-error split, plus our mutant-killing accuracy, computed with ten mutants per design drawn from a fixed operator set. The secondary evaluation uses the CVDP assertion and checker categories, including the checker category where the reported frontier result is 6%. VERT's held-out processor designs are the in-distribution check. All training uses only VERT and the CodeV-R1 RTL corpus, and we deduplicate against AssertionBench designs.
Four checks decide the outcome. The first is the per-family interference measurement described above; the whole proposal depends on it. The second is expert blocking: if disabling the multi-cycle expert collapses multi-cycle accuracy while leaving invariant accuracy intact, specialization is real. If nothing changes, the shared expert is carrying everything and the mixture is decoration. The third is the router comparison: structural routing from design features against learned token routing. This tells us whether circuit structure predicts which expert helps, which is an open question given that general-purpose routers track semantics. The fourth is a transfer probe on CVDP-ECov stimulus generation. We ask whether the shared-expert-plus-family design matches the reported 88.0% of CHORUS when the same recipe is applied to coverage rewards. That would indicate the result reflects verification work generally, not just the assertion task.
References
- Learning Semantic Representations to Verify Hardware Designs
- AssertionBench: A Benchmark to Evaluate Large-Language Models for Assertion Generation
- Comprehensive Verilog Design Problems: A Next-Generation Benchmark Dataset for Evaluating Large Language Models and Agents on RTL Design and Verification
- CHORUS: Complementary Experts for High-Coverage Testbench Stimulus Generation
- \mathcal XFT: Unlocking the Power of Code Instruction Tuning by Simply Merging Upcycled Mixture-of-Experts
- QiMeng-CodeV-R1: Reasoning-Enhanced Verilog Generation
- Enhancing Large Language Models for Hardware Verification: A Novel SystemVerilog Assertion Dataset
- CorrectBench: Automatic Testbench Generation with Functional Self-Correction using LLMs for HDL Design
- LLM4Cov: Execution-Aware Agentic Learning for High-coverage Testbench Generation
- Decoding Knowledge Attribution in Mixture-of-Experts: A Framework of Basic-Refinement Collaboration and Efficiency Analysis
- Invited Paper: VerilogEval: Evaluating Large Language Models for Verilog Code Generation