Testing Steel Design Code Against Physics, Not Branches

Software that checks whether a steel beam is safe follows the building specification closely. It computes a few slenderness ratios, compares each to a threshold, and picks a capacity formula. A slenderness ratio is how thin a plate is relative to its width, or how long a span is relative to its bracing. Because the code mirrors the specification's if-else structure, a test suite can reach 100% branch coverage with a handful of beams, one per branch. Nonlinear finite-element studies show something those branches cannot see. In asymmetric angles and tees, and in web-tapered girders, a thin plate wrinkling locally and the whole member twisting sideways feed on each other. Where that interaction happens depends jointly on yield strength, plate slenderness and unbraced length. A specification that checks these one at a time draws its thresholds in the wrong place. A suite that covers every branch can therefore certify software that is wrong exactly where beams fail.

This note proposes a different measure of test adequacy, called regime coverage. It asks whether a test suite has probed every physically distinct failure regime, as labeled by nonlinear simulation, and in particular the regime where local and global buckling are coupled. The note also proposes a test generator that spends a fixed simulation budget looking for inputs where the design code and the physics disagree. The core claim is that, on equal-leg single angles, tee stems and web-tapered I-girders, branch-adequate and threshold-boundary test suites will systematically miss the unconservative regions that the regime-guided search finds. "Unconservative" means the code overpredicts strength. The evidence that such regions exist is already in the literature. We aim to show that conventional coverage cannot find them and that a cheap, targeted search can.

Core Claim

The research file sets up two parallel observations that nobody has connected. On the software side, execution coverage is a weak proxy for catching faults. Code2Bench required 100% branch coverage on reference solutions and still found that nearly 7% of model submissions passed 98% of tests while failing edge cases. EvalPlus expanded HumanEval's tests about 80-fold. GPT-4's pass@1 fell from 88.4% to 76.2%, and 11% of the "correct" reference solutions turned out to be defective. Qian et al. explain the mechanism: inputs that follow the same execution path can differ sharply in whether they expose a fault.

On the mechanics side, Earls and coworkers showed the following through parametric nonlinear finite-element (FE) studies:

Combining the two gives a specific statement. A design-code implementation partitions input space along axis-aligned thresholds, such as $b/t \lessgtr \lambda_p$ and $L_b \lessgtr L_p$. The physical failure-mode boundaries are curved surfaces that cut across those partitions. Every branch can be exercised while the coupled region is never entered. Test adequacy should be measured in the space of physical regimes, and the oracle for that space is nonlinear simulation, not a reference implementation.

This also addresses an open problem the file identifies for EvalPlus, Code2Bench, VeriScale and HARDTESTGEN. All four depend on a trusted reference implementation or a formal specification. Here the reference implementation is the specification itself, and the specification is the thing under suspicion. Only physics can be the oracle.

Research Program

Code under test. We implement three provisions as reference code:

  • the equal-leg single-angle flexural proposals of Earls (2003), including $\lambda_p = 0.68\sqrt{E/F_y}$, the 2.6 stress-gradient factor and the shape-factor caps;
  • the tee-stem limit $h/t_w \approx 0.42\sqrt{E/F_y}$ from Yielding Limit State of Tee Stems in Flexural Compression;
  • the Appendix F web-tapered provisions evaluated by Miller and Earls.

Alongside each reference we add two families of faulty versions. The first is seeded mutants: wrong exponent, swapped $E/F_y$, omitted orientation check, wrong comparison operator at a threshold. The second is independent implementations written by code-generating language models from the specification text. These are the realistic "plausible but wrong" programs the software literature warns about.

Physics oracle. The oracle is a nonlinear shell FE model with initial geometric imperfections and elastic-plastic steel with strain hardening. We validate it before trusting it. It must reproduce the Prawel et al. (1974) tapered-girder load-deflection curves used by Miller and Earls, and it must recover their 0.949–1.29 ratio range and the $b_f/2t_f \le 3.0$ result for 6-inch flanges. Each FE run returns two things: ultimate moment, and a failure-regime label. The regimes are full plastic moment reached with rotation capacity $R \ge 3$; local-buckling-governed; global-buckling-governed; and coupled. A run counts as coupled when the onset of local buckling and the onset of lateral-torsional deformation fall within a small rotation window of each other.

We deliberately do not use the learned surrogates in the file as the oracle. The physics-augmented networks of Schommartz et al. are hyperelastic and limited to circular sections. The reduced-order models of Pichi and Rozza follow only the first elastic bifurcation branch. Both exclude the inelastic, asymmetric coupling that is the whole point here, so they constrain the design rather than supply it.

Regime coverage. We work in normalized coordinates $x = (\lambda/\lambda_p,\ L_b/L_p,\ F_y/E,\ \text{taper ratio},\ \text{orientation})$. We grid this space into cells and let the FE labels determine which regime each cell belongs to. A suite $T$ has regime coverage

$$C_R(T) = \frac{1}{|\mathcal{R}|}\sum_{r \in \mathcal{R}} \mathbb{1}\left[\exists\, t \in T : t \text{ lies within distance } \delta \text{ of the boundary of regime } r\right],$$

where $\mathcal{R}$ is the set of FE-identified regime regions and $\delta$ is a tolerance in normalized units. The criterion rewards tests near regime boundaries, because that is where small input changes flip the answer. This plays the role that infection and propagation play in software fault models: an input must land where the error actually changes the output.

Generator. The generator uses a Gaussian-process surrogate fitted to FE results so far. At each step it picks the next simulation by maximizing

$$a(x) = \Pr\!\left[\frac{M_{\text{FE}}(x)}{M_{\text{code}}(x)} < 1-\epsilon\right] + \beta\, H_{\text{regime}}(x),$$

where $M_{\text{FE}}$ is the simulated ultimate moment, $M_{\text{code}}$ is the capacity predicted by the code under test, $\epsilon$ is a tolerance on unconservatism, and $H_{\text{regime}}$ is the surrogate's uncertainty about which regime $x$ belongs to. The first term hunts for places where the code overpredicts strength. The second term keeps the search mapping regime boundaries instead of fixating on one known failure. The search domain deliberately extends past the specification's validity limits ($F_y$ up to 100 ksi, beyond the 70 ksi cap; high-strength tapered girders beyond Miller and Earls's Grade 50 study). This is where undefined behavior is most likely, and where engineering software will in practice be run without warning.

Outputs. We separate two error types that conventional testing mixes together. Implementation errors are cases where the code disagrees with the specification text. Specification errors are cases where the specification disagrees with physics. Each test the generator finds is assigned to one or both types.

Baselines

Naive reference: an existing hand-written suite with full branch coverage. For each provision we build the smallest suite that reaches 100% branch coverage on the reference implementation, one beam per compactness class and bracing zone. This is what engineering software verification typically amounts to, and it is the "do nothing" point. Every FE run it triggers is charged against the same budget as the other methods. Our method beats it if it finds unconservative regions this suite never enters. A secondary target is that the branch suite's mutant-kill rate turns out to be uncorrelated with its regime coverage, which would reproduce the software literature's coverage-efficacy gap in this domain.

Threshold boundary-value analysis. This is classical testing at and around every specification threshold ($\lambda_p$, $\lambda_r$, $L_p$, $L_r$, the 70 ksi cap) in every combination, each case labeled by FE. It is the strongest conventional competitor, because most implementation bugs live at thresholds. It runs on the same parameter ranges with the same FE budget.

Space-filling FE sampling. A Latin hypercube or Sobol design over the same domain, using the same number of FE runs, with the same surrogate fitted afterward. This is the strongest competitor on specification errors. It is what a careful mechanics researcher would do, and it is essentially how the Earls and Miller parametric databases were built. Beating it means finding more distinct unconservative regions, and a lower worst-case $M_{\text{FE}}/M_{\text{code}}$ ratio, per FE run.

Physics-side reference number. The figure to recover and then extend is Miller and Earls's FE-to-Appendix-F ratio range of 0.949 to 1.29. A search that only recovers 0.949 has found the known region. A meaningful result is a lower ratio, or unconservative regions in single angles or tees, or in high-$F_y$ tapered girders, that the existing databases did not report.

Software-side reference number. There is no reported result on Lacuna for adversarial or property-based testing of structural design code. The nearest analogues are EvalPlus's 19.3–28.9% pass@k drop under expanded tests and Code2Bench's roughly 7% near-miss rate despite 100% branch coverage. They are cited as precedent, not as directly comparable numbers.

Ablations. Three ablations isolate the key ingredients. Removing the regime-uncertainty term ($\beta = 0$) tests whether boundary mapping matters, or whether discrepancy-hunting alone suffices. Replacing FE labels with labels computed from the specification's own branches tests whether the physics oracle is what does the work. Restricting the search to $F_y \le 70$ ksi tests how much of the gain comes from extrapolating past the code's validity limits.

Why not just…?

Why not just do boundary-value testing at the specification thresholds? For implementation errors, this is a real threat, and we expect it to roughly tie our method on seeded mutants. Off-by-one comparisons and wrong constants sit exactly at the thresholds it probes. It cannot find specification errors, though, because it tests only where the specification already draws lines. The coupled failure surface Earls documents crosses those lines elsewhere: compactness that depends on $L_b$ for a fixed $b/t$. The experiment settles this directly. We report kill rates separately for the two error types. If boundary-value analysis matches us on specification errors too, the coupling claim is wrong for these sections.

Why not just run a dense space-filling FE study? This is the most serious alternative. With an unlimited budget it would find everything. The question is efficiency and coverage per run. Unconservative regions are thin slivers near curved regime boundaries, and uniform designs waste most of their runs deep inside single-regime cells. If the space-filling design at equal budget finds the same regions, the contribution shrinks to the regime-coverage metric itself. We will report that honestly, and the metric would still be useful as an adequacy criterion for suites built any other way.

Why not just apply EvalPlus-style mutation and differential testing against a reference implementation? These pipelines label tests by agreement with a trusted program. That catches code that departs from the specification, but it treats the specification as ground truth. The file lists oracle dependency as the known limit of EvalPlus, Code2Bench, VeriScale and HARDTESTGEN. In this domain the specification is the suspect, so differential testing against the specification is blind to the errors that matter most. We include differential testing as part of the boundary-value baseline so its contribution is measured rather than assumed.

Why not just use a learned physics model as a cheap oracle? The available models are hyperelastic and either restricted to circular sections or tracking only the primary bifurcation (Schommartz et al.; Pichi and Rozza). They omit yielding, which is what makes compactness depend on yield strength. Earls et al. (2026) show that text-trained language models can extrapolate simple one-dimensional PDE dynamics, but with error growing with output length. That is not a basis for labeling buckling regimes. Our surrogate only proposes where to simulate next. Every reported discrepancy is confirmed by full FE.

Proposed Experiments

Experiment 1: oracle validation. This gates everything else. The FE model must reproduce the Prawel et al. load-deflection curves, Miller and Earls's 0.949–1.29 ratios, the single-angle $\lambda_p$ trend with $F_y$, and the tee-stem $0.42\sqrt{E/F_y}$ threshold. If it cannot, the oracle is not trusted and the study stops there.

Experiment 2: the main comparison. All four test-generation strategies run under equal FE budgets of 200, 400 and 800 simulations per provision family, with five random seeds each. For each strategy and budget we report:

  • regime coverage $C_R$;
  • number of distinct unconservative regions found, at $\epsilon = 0.05$;
  • worst-case $M_{\text{FE}}/M_{\text{code}}$;
  • number of FE runs until the first unconservative case;
  • seeded-mutant kill rate;
  • the fraction of language-model-written implementations that pass the branch-adequate suite but fail the regime-guided suite.

This last fraction is the structural-engineering counterpart of EvalPlus's 11% defective references and Code2Bench's near-misses.

Experiment 3: the decisive check. Does the branch-adequate suite, and separately the boundary-value suite, leave unconservative regions undiscovered that regime-guided search finds at the same budget? And across all suites, does $C_R$ predict fault detection better than branch coverage does? We measure the second question by rank correlation between each metric and specification-error detection across many randomly drawn suites. A positive answer to both supports the core claim. Ties with space-filling sampling would narrow the claim to the metric. Ties with boundary-value analysis would refute it.

Experiment 4: extrapolation. The same search is run on high-strength tapered girders and on angles with $F_y$ above 70 ksi. These are the gaps the file identifies in Miller and Earls and in Earls (2003). The output is a map of where current provisions, extrapolated by software that does not enforce their validity limits, become unconservative.