TL;DR
When LLM agents design mechanical parts, they often produce code that runs perfectly but creates the wrong geometry—a phenomenon the authors call "silent failure." By introducing a new CAD benchmark with an exact geometric oracle, this paper demonstrates that single-shot agents are silently wrong 14% of the time, but a simple "measure-then-claim" protocol can slash that error rate to 3% while dramatically improving the agent's self-calibration.
What Problem Does This Paper Address?
As Large Language Models (LLMs) evolve into agents that write and execute code, we face a dangerous gap: syntactic success vs. semantic correctness. In the world of Computer-Aided Design (CAD), an LLM might write Python code that executes without errors and renders a plausible-looking 3D shape. However, that shape might have a hole in the wrong place, the wrong volume, or invalid constraints.
The authors argue that "crashes are cheap to catch," but a part that builds correctly while failing its engineering requirements is an "expensive failure." Existing benchmarks often rely on visual similarity or simple execution success, which fails to capture these subtle, "silent" errors. Furthermore, LLMs are notoriously overconfident; when an agent says "Done," can an engineer actually trust it?
Methods
The researchers developed a reproducible benchmark and harness designed to measure the gap between an agent's claim of completion and the actual geometric truth.
- The Oracle: Instead of using a "judge" model (which can be "leaky" or inconsistent), they used a real CAD kernel (
build123d/OpenCascade). This allowed them to measure B-rep properties—like mass-center offsets, volume, and precise hole positions—against published tolerance bands. - Agent Conditions: They tested four models (Claude 3.5 Sonnet/Haiku, DeepSeek-v3.2, and Qwen2.5-72B-Instruct) across three architectures:
- Single-shot: The agent generates code once from a prompt.
- Tool-use (ReAct): The agent can iterate, seeing kernel measurements of its geometry. Crucially, the "contract" requires the agent to review the measurements of its final code before claiming it is "DONE."
- Multi-agent: A Planner-Coder-Checker pipeline where a separate "checker" role reviews the measurements.
- Ablation Study: They performed a single-variable ablation to see if the improvement came from the feedback itself or the claim contract (the requirement to see measurements before saying "done").
Datasets
The paper introduces a new benchmark consisting of 13 mechanical specification tasks divided into three tiers:
- T1 (Single feature): Simple parts like washers or bushings.
- T2 (Multi-feature): Complex parts like stepped shafts or L-brackets with non-coaxial holes.
- T4 (Edit-under-constraint): Modifying existing code to meet new requirements.
The authors released the full attempt corpus (519 regraded attempts), reference solutions, the grader, and a 3D viewer for inspection.
Key Insights and Ideas
- The "Done" Contract is Key: The most surprising finding was that the harness contract—forcing the agent to acknowledge measurements before claiming completion—was more important than the iterative feedback itself. When agents were allowed to claim "done" alongside unverified code, silent failures spiked.
- Silent Failures Hide in Complexity: Silent failures aren't just a result of "hard" tasks; they concentrate in internal multi-feature geometry (T2 tasks). These are the types of errors that are hardest for a human to spot in a quick visual review.
- Verification Enables Calibration: Single-shot LLMs are "grossly overconfident" (ECE 0.32). However, when forced to condition their completion claim on actual measurements, their stated confidence becomes highly informative and well-calibrated for most models.
- Oracles Need Auditing Too: The authors found that even machine-checkable oracles can be too lenient. By "hardening" their oracle (adding checks for axial extents and volume bands), they flipped 18 previous "pass" verdicts to "fail," exposing 17 previously hidden silent failures.
Key Results
- Silent Failure Reduction: Single-shot agents claimed completion on 98% of attempts but were silently wrong on 14.1%. The enforced tool-use loop cut this to 3.2%.
- Precision Boost: Claim precision (the probability that a part is actually correct given the agent says it is) jumped from 64% to 96% with enforced verification.
- Calibration Improvement: For Claude 3.5 Sonnet, the Expected Calibration Error (ECE) dropped from 0.084 (single-shot) to a near-perfect 0.016 under tool-use.
Limitations and Future Work
- Task Scale: The suite is currently small (13 tasks), which is enough to distinguish architectures but not to rank models with high granularity.
- Single-Part Focus: The current version lacks multi-part assemblies, which would introduce new failure modes like interference.
- Natural Language Ambiguity: One task (the L-bracket) admitted two valid interpretations of a dimension, highlighting that even with an exact oracle, the "intent" gap remains a challenge.
- Roadmap: Future versions plan to include parametric re-instantiation (to prevent training data contamination) and a multi-part assembly tier.
Summary updated