Causal Signal Credit Assignment for Verilog Code Generation via Counterfactual Hardware Verification

Yali Du, Ming LiarXiv 2026 · arXiv

TL;DR

Researchers have developed Causal Signal Credit Assignment (CSCA), a framework that moves beyond simple pass/fail simulation feedback for Verilog generation. By using counterfactual interventions—deliberately breaking specific lines of code to see how output signals change—the system identifies exactly which statements are functionally correct, allowing models to learn from partially correct code without reinforcing hidden bugs.

What Problem Does This Paper Address?

Generating hardware description languages (HDLs) like Verilog is significantly harder than writing Python or C++. In hardware, you need cycle-level precision and strict interface compliance. Currently, we train Large Language Models (LLMs) using simulation feedback: if a module passes the testbench, it's "good"; if it fails, it's "bad."

This approach has two major flaws:

  1. Granularity Mismatch: A module might be 90% correct but fail a testbench due to one tiny bug. Traditional training throws the whole thing out, losing valuable "partially correct" evidence.
  2. The Attribution Gap: Just because a line of code is syntactically related to an output doesn't mean it's responsible for that output's correct behavior. Models can accidentally learn "spurious correlations"—reinforcing bad code that just happened to be present in a passing module.

Methods

CSCA introduces a three-step pipeline to bridge these gaps:

  1. Continuous Behavioral Verification: Instead of a binary pass/fail, CSCA compares the generated waveforms against a reference signal-by-signal. It calculates a continuous correctness score based on how many clock cycles each individual output signal matches the reference.
  2. Counterfactual Causal Attribution: This is the "secret sauce." For a statement that drives an output, CSCA creates five "mutant" versions of the code: Zero (force to 0), One (force to 1), Hold (self-assignment), Invert (logical NOT), and Delete.
  3. If breaking a specific line causes the output signal to degrade, that line is proven to have a "causal influence" on the correct behavior and receives high "credit."
  4. Statement-Level Weighted DPO: These credits are mapped directly to token weights in a Direct Preference Optimization (DPO) objective. Instead of rewarding the whole response, the model only gets a strong "thumbs up" for the specific tokens that were causally verified.
Overview of the CSCA pipeline, showing how candidate sampling leads to behavioral verification and causal attribution before updating the model.

Datasets

  • Training: A high-quality subset of CodeV, consisting of 135K valid Verilog modules scraped from GitHub.
  • Evaluation: The models were tested on two industry-standard benchmarks:
  • VerilogEval 2.0: Focuses on code completion and specification-to-RTL tasks.
  • RTLLM v2.0: Covers a diverse range of designs including arithmetic, control logic, and memory.

Key Insights and Ideas

  • Partially Correct is Useful: By treating correctness as a vector of signal scores rather than a single bit, the model can learn from "almost right" code.
  • Causality over Correlation: In complex RTL, signals often have overlapping dependencies. The counterfactual approach (shown in Figure 2) ensures the model doesn't get credit for "dead code" or redundant logic that doesn't actually drive the output.
  • Localized Reinforcement: Standard DPO can be "noisy" because it reinforces every token in a preferred response. CSCA’s weighted approach acts like a precision laser, only reinforcing the functionally validated logic.
The five types of counterfactual interventions used to measure the functional impact of a statement.

Key Results

Using a Qwen2.5-Coder-7B backbone, CSCA achieved remarkable gains:

  • Massive Baseline Boost: Improved greedy-decoding Pass@1 by 72.5% on VerilogEval Code Completion and 114.3% on Specification-to-RTL compared to the base model.
  • Beating Specialized Models: It outperformed QiMeng-SALV, the previous state-of-the-art Verilog-specific model, by up to 8.7% across different tasks.
  • Punching Above Its Weight: The 7B-parameter CSCA model remained competitive with the significantly larger GPT-5.4-mini on major benchmarks.
  • Synergy Matters: Ablation studies showed that combining behavioral scores with counterfactual evidence (the hybrid $\alpha=0.7$ setting in Figure 4) performed better than using either signal alone.
Sensitivity analysis showing that the best performance is reached by balancing behavioral and counterfactual evidence.

Limitations and Future Work

  • Finite Traces: Counterfactuals are only as good as the testbench. If a testbench doesn't exercise a certain logic path, the causal attribution might be incomplete.
  • Scope of Interventions: Currently, the method focuses on combinational output-driving assignments. Future work could extend this to complex sequential states and inter-module dependencies.
  • Computational Overhead: While the DPO stage is efficient (adding only ~8.8% to the GPU budget of SFT), the initial simulation and intervention phase requires running many parallel Icarus Verilog instances.

Summary updated

Abstract

Large language models have improved Verilog generation, yet functional correctness remains difficult to optimize because simulation commonly provides a coarse module-level pass/fail outcome. Such feedback discards useful evidence from partially correct RTL modules and cannot determine whether a source statement is functionally responsible for an observed output behavior. We present a new method named Causal Signal Credit Assignment (CSCA), which is a counterfactual preference-optimization framework for Verilog generation. CSCA first performs continuous signal-level behavioral verification on sampled candidate modules. It then locates output-driving assignment spans and applies five interventions while holding the remaining module and testbench fixed. The resulting changes in signal-level errors are combined with behavioral agreement to form a continuous statement credit, which is mapped to response-token weights for strict weighted direct preference optimization. This enables CSCA to learn from locally supported statements without uniformly reinforcing an otherwise incorrect response. Built on Icarus Verilog and a CodeV-derived corpus, CSCA is evaluated on VerilogEval~2.0 and RTLLM~v2.0, including both code completion and specification-to-RTL tasks. With a Qwen2.5-Coder-7B-Instruct backbone, CSCA leads the evaluated 7B-scale models and remains comparable to the much larger GPT-5.4-mini on the reported Verilog-generation benchmarks.
BibTeX
@misc{du2026causal,
  author = {Yali Du and Ming Li},
  title = {{Causal Signal Credit Assignment for Verilog Code Generation via Counterfactual Hardware Verification}},
  year = {2026},
  eprint = {2611.11848},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11848},
}
Figure