Small language models trained on one technical domain can now beat much larger general models. One example is a 4-billion-parameter model that writes hardware test stimuli better than a 671-billion-parameter one. These models are trained only on tasks that have a solution, so they learn to always produce one. Hardware verification is a domain where that habit is costly. Many coverage targets (the signal states or code branches a test is supposed to exercise) can never be reached, because of dead code, signals tied to constants, or contradictory specifications. A model that keeps generating stimulus for an impossible target wastes its budget and returns false confidence.
We propose to post-train a compact verification model so that, for each target, it does three things in order. First, it decides whether the target is reachable. If it is not, it names the specific condition in the design that blocks it. Then it proposes the smallest change to the specification that would make the target reachable. Every stage receives its own verifiable reward, computed by tools rather than by human judges. The bet is that this graded "detect, localize, repair" signal fixes the model's always-solvable habit. We also bet it does so without the opposite failure, where the model gives up on targets that could be reached, and that it does better here than either a prompting checklist or a plain yes/no abstention label.
Core Claim
Our central claim concerns the right unit of supervision for teaching a domain model to recognize impossibility. The literature shows that this bias is pervasive, and that scale and specialization do not remove it. Existing fixes are either inference-time checklists or binary feasibility labels. We claim that a three-level taxonomy, used as a training signal with tool-checked rewards at each level, is what makes calibrated abstention stable and transferable.
The reason is credit assignment. A binary "unsolvable" reward can be maximized by abstaining whenever things look hard, which is the pessimistic failure. A reward for naming the blocking condition can only be earned by actually reasoning about the design. The localization level therefore acts as an anchor against both optimism and pessimism.
The claim is falsifiable in a specific way. If a model trained with the binary label matches the three-level model on both of the following, then the hierarchy adds nothing and the claim fails:
- Missed impossibilities: how often the model attempts targets that cannot be reached.
- False abstentions: how often the model gives up on targets that can be reached.
Where This Sits
Three lines of work motivate the gap.
Diagnostic benchmarks. ToolBeHonest builds the exact three-level taxonomy we borrow: solvability detection, bottleneck sub-goal identification, and missing-tool description. Frontier models score above 90 on solvable tool tasks but collapse on unsolvable ones (GPT-4o 37.0 overall, Gemini-1.5-Pro 45.3). ToolBeHonest is purely diagnostic and limited to API calls; we turn its taxonomy into a training reward in a domain where every level can be checked by a tool.
Feasibility and state checking. FineReason shows that GPT-4o is near chance (about 52%) at judging whether a partial Sudoku state is still solvable, and that reinforcement learning on state checking helps a 7B model. Its signal is a flat solvable/unsolvable check plus backtracking on closed puzzles. Towards Budget-Aware Agents shows that supervised fine-tuning lifts a 7B model's feasibility prediction from 25.5% to roughly 90% on Sokoban. It also finds that reinforcement learning without a supervised warm start collapses, and that agents typically notice impossibility only in the final 20% of a trajectory. We reuse its warm-start recipe as a design constraint, and its open question about cross-domain transfer is one we test directly.
Specialization without negatives. CHORUS, AutoTriton and VeriGen specialize compact models for hardware and kernel code using execution-checked rewards. None of them include impossible instances. CHORUS assumes every coverage target is reachable. It is our method donor and our strongest positive-task baseline.
Two papers constrain the design.
Execution access can backfire. Can LLMs Detect Benchmark Defects? finds that giving GPT-5.4 access to the execution harness drops defect recall from 78.3% to 53.9%. The model assumes the environment is correct and blames its own code, which the authors call self-attribution anchoring. Because our training is execution-driven, it could amplify exactly this. Our design therefore includes episodes where the environment, not the model, is at fault.
Structured prompting is a strong cheap fix. Can Tool-augmented LLMs be Aware of Incomplete Conditions? shows that a five-step verification checklist alone raises Gemma-3-4B's abstention F1 from 20.90% to 68.57%. This is the cheap alternative we must beat.
The specialization tax documented by Specializing Smaller Language Models towards Multi-Step Reasoning is a further risk. Narrow fine-tuning there erased general reasoning outside the target domain, so we measure out-of-domain abstention explicitly.
Research Program
Training data. We build an augmented version of the coverage-stimulus task from the training repositories in CodeV-R1-11kRTL, which CHORUS uses.
- Making targets unreachable. For each design we apply mutations with known effects: tying an enable signal to a constant, adding a contradictory guard to a branch, or removing a state transition. Each mutation makes a chosen coverage target unreachable, and we know the blocking condition because we created it.
- Confirming the label. We confirm unreachability by simulation exhaustion on small designs and by a formal reachability check where one is available.
- Keeping both directions honest. Mutations that leave the target reachable serve as distractors, so the model cannot learn that "any edit means impossible."
- Environment faults. A further slice contains broken testbench harnesses, such as a wrong reference or a stale include. Here the correct answer is "the target is reachable, but the environment is defective." This slice directly targets self-attribution anchoring.
Model outputs. For every target the model emits one of three things: a stimulus, a verdict that the target is unreachable, or a verdict that the environment is defective. An unreachability verdict comes with a named blocking signal or condition and a proposed specification edit.
Reward. We use gated rewards:
$$ R = \mathbb{1}[v=v^*]\,\big(r_1 + \mathbb{1}[\text{L1 correct}]\,(r_2\,\mathbb{1}[b \in B^*] + r_3\,\mathbb{1}[\text{edit restores reachability}])\big) + \mathbb{1}[v^*=\text{reachable}]\,c - \lambda\,\mathbb{1}[v\neq v^*,\,v^*=\text{reachable}] $$
The symbols are defined as follows:
- $v$ and $v^*$ are the predicted and true verdicts.
- $b$ is the blocking condition the model names, and $B^*$ is the set of conditions that actually block the target, known from the mutation.
- The "edit restores reachability" term is checked by re-running coverage on the edited design.
- $c$ is the coverage reward CHORUS already uses for reachable targets.
- $\lambda$ penalizes false abstention on reachable targets.
Training stages. Following the budget-aware finding, we warm-start with supervised fine-tuning on teacher-written traces for all three levels. We then run the reinforcement-learning stage of CHORUS (DAPO with worst-state refinement) under this reward.
Starting point. We start from the same 4B base as CHORUS, which isolates the effect of the new data and reward.
Baselines
Naive reference. The naive reference is the CHORUS-4B recipe trained only on reachable targets and evaluated on the augmented test set. It reports 88.0% Pass@1 on CVDP-ECov, against 74.5% for DeepSeek-R1, both from CHORUS. We expect it to attempt nearly every unreachable target.
- Our first bar is to keep its Pass@1 on reachable targets within about one point of 88.0.
- Our second bar is to cut missed impossibilities by at least half.
Prompting baseline. The strongest cheap competitor is Structured Verification from the incomplete-conditions paper, with its five steps adapted to coverage targets. We apply it to both the naive CHORUS-style model and the untouched base model.
- Its reported number is 68.57% abstention F1 for Gemma-3-4B on ToolBench API Replacement. That is a different task, so it serves as an order-of-magnitude reference only.
- For unreachable coverage targets specifically, no reported result was found on Lacuna. We will measure it ourselves under the same decoding budget.
- Beating it means a higher abstention F1 at equal or lower false-abstention rate, with a confidence interval from a bootstrap over held-out repositories.
Frontier references. DeepSeek-R1 and GPT-4o, each with Structured Verification, serve as references for whether scale alone removes the bias. ToolBeHonest and FineReason suggest it will not.
Key ablation: binary label. This ablation isolates the hierarchy. It uses the same data, the same supervised warm start, the same reinforcement-learning compute and the same tuning budget, but keeps only the verdict term and the false-abstention penalty.
Second ablation: environment faults. We drop the environment-fault slice to measure its specific contribution to resisting self-attribution anchoring.
Fair comparison. All trained variants receive identical numbers of gradient steps and rollouts, and hyperparameters are tuned on a shared validation split of repositories.
Why not just…?
Why not just prompt the model with a verification checklist? The checklist is cheap and has produced large gains on tool calls, so it is a real threat. Two pieces of evidence suggest it has limits:
- Even with the checklist, Gemma-3-4B stays well below the roughly 81% human baseline on ICT.
- The benchmark-defects study found that the choice of pipeline mattered far less than the choice of base model.
Our experiments settle the question by running the checklist on our trained model as well. If the checklist on the naive model matches the trained model, then training is unnecessary in this domain, and we will report that.
Why not just add a binary "unreachable" label to the training data? This is the most serious alternative. The budget-aware study shows that supervised fine-tuning alone makes binary feasibility prediction quite accurate. Our hypothesis is that binary reward invites blanket abstention under reinforcement learning, and that localization supervision prevents it. The binary ablation tests this directly. The deciding quantity is the false-abstention rate on hard but reachable targets, the ones CHORUS solves only after worst-state refinement. If the binary variant matches on that rate, we drop the hierarchy claim.
Why not just run a formal reachability checker on every target? Where a formal check is cheap and conclusive, it is the right tool, and we include it as an oracle upper bound. On large designs, however, such checks can be expensive or inconclusive. Even when they succeed, they say only that a target is unreachable, not why or how to change the specification. We report the cost at which the trained model's triage matches the oracle's decisions, and we use the oracle to route only uncertain cases. If routing every target to the checker is cheap and decisive across CVDP-ECov designs, our contribution narrows to the localization and repair levels.
Why not just use a frontier model? ToolBeHonest, FineReason and the budget-aware study all show frontier models failing at exactly this kind of impossibility detection. On top of that, CHORUS-4B already beats DeepSeek-R1 on the positive task.
Proposed Experiments
In-domain evaluation. We test on held-out repositories from CVDP-ECov and AutoEval-ECov, augmented with mutated unreachable targets and defective harnesses whose true labels are fixed by construction. We report:
- Pass@1 on reachable targets.
- Abstention precision, recall and F1 on unreachable targets.
- False-abstention rate.
- Accuracy at naming the blocking condition, and validity of the proposed repairs.
- Recall on environment defects, broken out by whether the model had execution access, mirroring the comparison in the benchmark-defects paper.
- Tokens spent before a correct abstention, compared against the 28–64% savings on doomed runs reported in Towards Budget-Aware Agents.
Decisive check 1: the hierarchy. The hierarchy matters only if the three-level model beats the binary ablation on false abstention for hard reachable targets while matching it on detection.
Decisive check 2: execution access. Training with environment faults should remove the drop in recall when execution access is granted. The reference drop is from 78.3% to 53.9% for GPT-5.4 in the TVB study. If our model still shows a comparable drop, execution-driven specialization reinforces self-attribution anchoring, and that is a negative result worth publishing.
Decisive check 3: transfer. We evaluate the hardware-trained models zero-shot on ToolBeHonest and on the ICT split (dataset). The comparison is against the base model and the binary ablation, with GSM8K as a check for the specialization tax. The budget-aware study left this kind of cross-domain transfer open. Seeing gains on ToolBeHonest's level-two and level-three scores, but not on level one, would be evidence that the model learned a portable habit of localizing what blocks a task, rather than a domain-specific abstention reflex.
References
- ToolBeHonest: A Multi-level Hallucination Diagnostic Benchmark for Tool-Augmented Large Language Models
- FineReason: Evaluating and Improving LLMs' Deliberate Reasoning through Reflective Puzzle Solving
- Towards Budget-Aware Agents: Can LLM Agents Know What They Will Spend?
- CHORUS: Complementary Experts for High-Coverage Testbench Stimulus Generation
- AUTOTRITON: Automatic Triton Programming with Reinforcement Learning in LLMs
- VeriGen: A Large Language Model for Verilog Code Generation
- Can LLMs Detect Benchmark Defects? A Meta-Benchmark from Benchmark Updates
- Can Tool-augmented Large Language Models be Aware of Incomplete Conditions?
- Specializing Smaller Language Models towards Multi-Step Reasoning