TL;DR
Large Language Models (LLMs) can often calculate the physical limits of a mechanical system with high precision, yet they frequently fail to reject design specifications that clearly violate those limits. This "calculation-rejection divergence" means a model might correctly identify a beam's collapse load in one breath and then provide a "solved" status for a load 30% beyond that limit in the next.
What Problem Does This Paper Address?
In engineering, a calculation can be numerically "correct" according to a formula but physically impossible in reality—for example, calculating the stress in a bridge that has already collapsed. When LLMs are integrated into engineering workflows, they typically return a status like "solved" or "cannot solve." If a model returns numerical values for an impossible state with a "solved" label, downstream software or unsuspecting engineers might trust and use that data, leading to catastrophic design failures.
While existing benchmarks measure how well LLMs solve problems, they rarely test if models have the "common sense" (or mechanical intuition) to reject infeasible specifications. This paper investigates whether models that are capable of calculating a physical limit actually use that knowledge to abstain from answering impossible questions.
Methods
The researchers tested 12 state-of-the-art models (from Anthropic, OpenAI, and Google) using two classic mechanical engineering problems:
- T02: The plastic collapse of a two-span steel beam.
- T11: The burst speed of a spinning turbine disk.
They defined a "certificate" (the ground-truth physical limit) using two independent reference algorithms. They then generated "twins"—pairs of problems where the input was either just below (valid) or just above (flawed) the limit, measured by a signed margin ($m$).
The study used 4,860 model calls across various "prompt arms," including:
- Unwarned ($B_u$): Standard prompt with no mention of impossible problems.
- Explicit Rule ($S1$): Instructing the model to decline if any quantity has no valid value.
- Confidence Target ($B_t$): A scoring system that penalizes wrong answers more than abstentions.
- Feasibility Only (FO): Asking only if the design is possible, without asking for a full solution.
Datasets
The authors introduced several "frozen" item sets:
- Main set: 76 items with margins of $\pm$0.3% to 30%.
- Fine set: 64 items with tiny margins ($\pm$0.01% to 0.3%) to test resolution.
- Dense Grid: 120 items specifically for the "Fable 5.1" model to find its transition point.
- New Items: 80 additional designs to validate follow-up experiments.
Key Insights and Ideas
- The "Calculation-Rejection Gap": This is the paper's core "aha" moment. Models like Opus 5 and Gemini 3.8 Flash proved they could calculate limits precisely (with errors as low as 0.01%), yet they accepted and "solved" beam flaws 30% beyond the collapse point.
- Status Conflict: In many cases, the model’s reasoning actually noted the violation. An Opus 5 reply might say, "The load exceeds the collapse factor," but then proceed to give a numerical answer and mark the JSON status as
"solved". - Task Framing Matters: Models are much better at judging feasibility when that is the only thing they are asked to do. When asked to "solve" a problem, the pressure to provide a numerical output seems to override their internal check for physical possibility.
Key Results
- Top Performers: Opus 5.5 and GPT-6.1-Sol were highly robust, rejecting almost all flaws even at the tiny 0.01% margin with zero false alarms.
- The Divergence: Models like Opus 5 and Gemini 3.8 Flash missed nearly every beam flaw in the main set, despite passing 90% of the "capability probes" (questions asking only for the limit).
- Feasibility vs. Solving: In a head-to-head test, Opus 5 and Gemini rejected 100% of impossible designs when asked a simple Yes/No feasibility question, but accepted those same designs when asked for a full calculation.
- Incentives Work (Sometimes): Adding a confidence-based penalty ($B_t$) significantly improved rejection rates for some models (like Opus 4.8 on disks) but caused others to start "hallucinating" reasons to reject valid problems (false alarms).
Limitations and Future Work
The authors identify several constraints:
- Scope: The study only looked at two specific mechanical families with scalar limits.
- Evaluation: Grading relied on deterministic code and AI-assisted coding; no human engineers outside the research team reviewed the thousands of model replies.
- Prompt Sensitivity: While they tested several prompt styles, the "perfect" prompt for engineering safety remains elusive.
Future Work: The authors suggest that engineering AI pipelines should separate feasibility judgment from calculation. A "safety-first" architecture might involve asking a model if a design is possible before allowing it to attempt a numerical solution.
Summary updated