Learning Weighted Test Generators for Chips That Differ from Their Simulation

Every manufactured chip is slightly different from its design. At nanometer scale, random atomic-level fluctuations change how fast each gate switches, so the slowest path through a circuit is not the same on every die. A path that looked safe in the nominal timing report can become the one that fails on a particular chip. Delay tests are the input patterns applied at speed to catch slow chips before they ship. Today these tests are generated against the nominal design and then checked against statistical simulation. That leaves two gaps: patterns chase the wrong paths, and the simulator's variation model is itself only an approximation of real silicon.

This note proposes to learn a probability distribution over test patterns rather than a fixed list of them. The tool is an estimation of distribution algorithm (EDA), an optimizer that keeps a probability model over candidate solutions, samples from it, and shifts probability toward the samples that score best. Here the model is scored across a population of simulated dies with different process variation. The learned distribution is itself the deliverable. A per-bit probability vector, or a small dependency model, is exactly what a weighted pseudo-random on-chip test generator needs. The generator is a random bit source with biased bits, which costs a few bytes of on-chip storage, whereas a stored pattern set costs kilobytes. In two sentences: we fit a biased random test generator whose patterns catch the slow paths that move from die to die, by optimizing its probabilities against many variation-sampled dies at once. We then test whether it transfers across the simulation-to-silicon gap by evaluating it under a harsher variation model and on physical FPGA boards.

Core Claim

The claim has two parts. The first is that averaging fitness over sampled dies is the right objective for delay test under variation. The second is that a distribution-based optimizer is the natural way to optimize it, because the output must be a compact generator rather than a single best pattern.

Let $x$ be a two-vector launch/capture pattern over a circuit's inputs, and let $\theta$ parameterize the generator distribution $q_\theta(x)$. Let $d$ index dies drawn from a process-variation model, and let $T$ be the target clock period. A die $d$ is bad if some functionally sensitizable path on it has delay above $T$. A pattern catches it if the pattern sensitizes such a path. The optimization target is the expected test escape rate after drawing $N$ patterns from the generator:

$$ E(\theta) = \Pr_{d}\big[\, d \text{ is bad and no pattern among } x_1,\dots,x_N \sim q_\theta \text{ catches } d \,\big]. $$

This objective has an unusual shape, and it is the reason EDA theory matters here. Whether a path is sensitized is all-or-nothing. A pattern that launches a transition along 29 of 30 gates of a critical path gets no credit, which produces fitness cliffs. Die-to-die variation, by contrast, acts like high-frequency noise on which pattern is best. The theory pulls in opposite directions on these two features. Friedrich et al. proved that the compact genetic algorithm (cGA) averages away additive noise that traps point-based local search: it reaches $n(1-\epsilon)$ ones on noisy OneMax where randomized local search reaches only $n(1/2+o(1))$. Neumann, Sudholt and Witt proved the cGA falls into a reinforcement trap on cliffs, needing more than $10^8$ evaluations at $n=18$, which is worse than random search. Delay-test generation under variation has both features at once. It is therefore a concrete, practically meaningful setting in which to find out which effect dominates, and whether a cliff-softening credit signal restores progress. The second contribution is modest but real: the first empirical study of EDAs on a landscape where noise and cliffs coexist, with a physical ground truth.

Research Program

The circuits are the ISCAS85 and ISCAS89 benchmarks already used for variation-aware delay test by Li-C. Wang et al.. That work maps sensitized-path features to failure probability with a support vector machine (SVM), but only ranks patterns that commercial test tools (TR10, TR15) had already generated. It also showed that nominal longest paths correlate poorly with post-silicon failures. That result is the motivation here. Their SVM ranks existing patterns; we generate new ones.

We build a statistical timing simulator that draws per-gate delays for thousands of dies. Its first version assumes independent gates, which is the usual simplification. For each die, we estimate ground-truth badness by combining exhaustive path enumeration on small circuits with a very large random-pattern pool on larger ones. Pattern evaluation uses logic simulation for sensitization plus summed sampled delays. This is cheap enough to support tens of thousands of evaluations, so oracle cost does not dominate the way ESMFold calls did for Key-cutting machine. That paper used EDA islands for peptide design, entirely against an idealized simulator. We compare three generator families of increasing expressiveness. The first is a univariate cGA/UMDA probability vector per input bit, which maps directly onto weighted random-pattern hardware. The second is a small Bayesian network over input bits in the style of hBOA (Pelikan), which preserves correlated input groups needed to launch a path. The third is a Tensor Train distribution following PROTES for the widest circuits.

Each generation samples patterns, evaluates them against a fresh minibatch of dies, and updates toward patterns that catch dies not yet caught by the current set. This marginal-coverage credit keeps the distribution from collapsing onto one path. To soften cliffs, we add an optional partial-sensitization credit: the fraction of a near-critical path's gates along which a transition propagates. We then test whether this credit is needed or harmful.

The sim-to-silicon gap enters in two stages. In the first stage, generators trained under the independent-gate model are evaluated under a mismatched model. That model adds spatially correlated variation and inter-gate loading, which Mukhopadhyay et al. showed raises path leakage standard deviation by over 40%, so independence is known to understate tails. In the second stage, a subset of circuits is mapped onto a batch of nominally identical FPGA boards. Each pattern's failing clock period is measured on each board by sweeping the capture clock. Commodity devices show large die-to-die spread, up to 22% under power caps on identical GPUs according to Sinha et al., so this gives real variation with real ground truth. Finally, in the spirit of the hardware-in-the-loop calibration of Air Learning, which cut a 37.73% flight-time mismatch to 0.5% by injecting measured latency distributions into simulation, we recalibrate the die sampler from a few measured boards and retrain. We then ask how much of the transfer loss this recovers on held-out boards.

Baselines

The naive reference point is an unweighted pseudo-random generator: uniform bits, the default built-in self-test. Its escape-rate curve against pattern count is the floor every learned generator must beat at equal $N$.

The strongest existing method is deterministic automatic test pattern generation (ATPG) targeting nominal longest paths, using the TR10/TR15 transition-fault sets studied by Li-C. Wang et al.. We add that paper's SVM-based pattern ranking, which selects a variation-aware subset. Both get the same die population and the same pattern budget $N$. We found no reported escape-rate or detection result for these ATPG sets on Lacuna; that paper reports SVM prediction accuracy "often exceeding 95%" and large speedups, not escape rates, so we will measure the baseline ourselves. For weighted-hardware comparisons, we also fit weights to the ATPG set, meaning per-bit frequencies, to give ATPG a fair generator form.

The strongest competitor on optimization power is a point-based genetic algorithm evolving an explicit pattern set, using fitness averaged over the same die minibatches. This mirrors the NSGA-II approach Balaskas et al. used for variability-aware synthesis, where it cut critical path delay by 16.4% on average and removed violations across 1,000 Monte Carlo points. Its final set will be distilled into generator weights. It receives the same evaluation budget and the same hyperparameter-tuning trials as the EDAs.

Three ablations isolate the key ingredients. The first is an EDA trained on the nominal die only, which isolates the value of variation averaging. The second is the EDA with and without partial-sensitization credit, which isolates the cliff effect. The third is univariate versus Bayesian-network models, which isolates the value of dependencies. Beating the baselines means lower held-out escape rate at equal pattern count, especially under the mismatched model and on FPGA boards, and at equal or lower generator storage.

Why not just…?

Why not just run nominal-path ATPG with more patterns (N-detect)? More patterns on nominally critical paths do not reach paths that become critical only on some dies. Wang et al. found nominal longest paths predict post-silicon failures poorly, and Balaskas et al. report 79× to 1936× higher worst-case error for variation-unaware tools. Still, ATPG with large $N$ may close much of the gap on small circuits. The escape-versus-$N$ curves will show where, if anywhere, the learned generator's advantage survives.

Why not just use a genetic algorithm with die-averaged fitness and fit weights afterward? This is the real threat. Averaging over dies may supply most of the benefit regardless of optimizer. We run it with a matched budget. If it matches the EDA, the conclusion is that the die-population objective, not the EDA, is the contribution, and the note will say so. Our expectation is based on the cGA's proven noise averaging and on the mismatch between optimizing a vector list and then compressing it: fitting weights discards the correlations a directly optimized distribution keeps. But this is an expectation, not evidence.

Why not just use the SVM surrogate to select from a huge random pool? Selection can only find patterns that the pool happens to contain. On deep paths requiring coordinated input bits, the probability of a uniform sample launching the path is exponentially small. The comparison against pool selection at matched simulator calls, broken down by path depth, settles whether generation beats selection.

Proposed Experiments

The decisive checks are three. First, on ISCAS85/89 circuits under the independent-gate model, does the variation-averaged EDA generator reach lower held-out escape rate than uniform BIST, ATPG plus SVM selection, and the die-averaged GA at equal $N$ and equal simulator calls? Second, when the model is swapped to correlated and loading-aware variation, and then to measured FPGA boards, which method degrades least, and how much does calibrating the die sampler from a few boards recover? Third, on the cliff question, does the plain cGA stall on circuits whose critical paths need long coordinated input assignments, as the reinforcement-trap theory predicts? And does partial-sensitization credit or a dependency model remove the stall, as the noise-averaging result would suggest? We report escape rate, catching fraction of bad dies, generator storage in bits, and evaluations to convergence, averaged over independent seeds with confidence intervals.