Done Is Not Correct: Measuring Silent Failures and Self-Verification Calibration When LLM Agents Take CAD Actions

Carson Rodrigues, Clive Rodrigues, Aravind Reddy GarXiv 2026 · arXiv

TL;DR

When LLM agents design mechanical parts, they often produce code that runs perfectly but creates the wrong geometry—a phenomenon the authors call "silent failure." By introducing a new CAD benchmark with an exact geometric oracle, this paper demonstrates that single-shot agents are silently wrong 14% of the time, but a simple "measure-then-claim" protocol can slash that error rate to 3% while dramatically improving the agent's self-calibration.

What Problem Does This Paper Address?

As Large Language Models (LLMs) evolve into agents that write and execute code, we face a dangerous gap: syntactic success vs. semantic correctness. In the world of Computer-Aided Design (CAD), an LLM might write Python code that executes without errors and renders a plausible-looking 3D shape. However, that shape might have a hole in the wrong place, the wrong volume, or invalid constraints.

The authors argue that "crashes are cheap to catch," but a part that builds correctly while failing its engineering requirements is an "expensive failure." Existing benchmarks often rely on visual similarity or simple execution success, which fails to capture these subtle, "silent" errors. Furthermore, LLMs are notoriously overconfident; when an agent says "Done," can an engineer actually trust it?

Methods

The researchers developed a reproducible benchmark and harness designed to measure the gap between an agent's claim of completion and the actual geometric truth.

  • The Oracle: Instead of using a "judge" model (which can be "leaky" or inconsistent), they used a real CAD kernel (build123d/OpenCascade). This allowed them to measure B-rep properties—like mass-center offsets, volume, and precise hole positions—against published tolerance bands.
  • Agent Conditions: They tested four models (Claude 3.5 Sonnet/Haiku, DeepSeek-v3.2, and Qwen2.5-72B-Instruct) across three architectures:
  • Single-shot: The agent generates code once from a prompt.
  • Tool-use (ReAct): The agent can iterate, seeing kernel measurements of its geometry. Crucially, the "contract" requires the agent to review the measurements of its final code before claiming it is "DONE."
    1. Multi-agent: A Planner-Coder-Checker pipeline where a separate "checker" role reviews the measurements.
  • Ablation Study: They performed a single-variable ablation to see if the improvement came from the feedback itself or the claim contract (the requirement to see measurements before saying "done").

Datasets

The paper introduces a new benchmark consisting of 13 mechanical specification tasks divided into three tiers:

  • T1 (Single feature): Simple parts like washers or bushings.
  • T2 (Multi-feature): Complex parts like stepped shafts or L-brackets with non-coaxial holes.
  • T4 (Edit-under-constraint): Modifying existing code to meet new requirements.

The authors released the full attempt corpus (519 regraded attempts), reference solutions, the grader, and a 3D viewer for inspection.

Key Insights and Ideas

  • The "Done" Contract is Key: The most surprising finding was that the harness contract—forcing the agent to acknowledge measurements before claiming completion—was more important than the iterative feedback itself. When agents were allowed to claim "done" alongside unverified code, silent failures spiked.
  • Silent Failures Hide in Complexity: Silent failures aren't just a result of "hard" tasks; they concentrate in internal multi-feature geometry (T2 tasks). These are the types of errors that are hardest for a human to spot in a quick visual review.
  • Verification Enables Calibration: Single-shot LLMs are "grossly overconfident" (ECE 0.32). However, when forced to condition their completion claim on actual measurements, their stated confidence becomes highly informative and well-calibrated for most models.
  • Oracles Need Auditing Too: The authors found that even machine-checkable oracles can be too lenient. By "hardening" their oracle (adding checks for axial extents and volume bands), they flipped 18 previous "pass" verdicts to "fail," exposing 17 previously hidden silent failures.

Key Results

  • Silent Failure Reduction: Single-shot agents claimed completion on 98% of attempts but were silently wrong on 14.1%. The enforced tool-use loop cut this to 3.2%.
  • Precision Boost: Claim precision (the probability that a part is actually correct given the agent says it is) jumped from 64% to 96% with enforced verification.
Figure 1: Success and silent-failure rates by model and agent condition (primary population).
  • Calibration Improvement: For Claude 3.5 Sonnet, the Expected Calibration Error (ECE) dropped from 0.084 (single-shot) to a near-perfect 0.016 under tool-use.
Figure 2: Reliability diagram and confidence distribution, primary population, pooled over the three conditions (hence the single pooled ECE in the panel title; per-condition ECE is in F3, per-model ECE in Table 1).

Limitations and Future Work

  • Task Scale: The suite is currently small (13 tasks), which is enough to distinguish architectures but not to rank models with high granularity.
  • Single-Part Focus: The current version lacks multi-part assemblies, which would introduce new failure modes like interference.
  • Natural Language Ambiguity: One task (the L-bracket) admitted two valid interpretations of a dimension, highlighting that even with an exact oracle, the "intent" gap remains a challenge.
  • Roadmap: Future versions plan to include parametric re-instantiation (to prevent training data contamination) and a multi-part assembly tier.

Summary updated

Abstract

When a large language model (LLM) agent writes parametric CAD code from a natural-language specification, syntactic success decouples from semantic correctness. We release a benchmark and harness that measure this gap directly: 13 specification tasks whose every requirement maps to machine-checkable properties of the produced B-rep, measured by the kernel against published tolerance bands, with reference solutions and the full graded attempt corpus. The exact, self-audited oracle isolates two quantities no prior CAD benchmark reports: the silent-failure rate (a valid, renderable solid, an explicit completion claim, and at least one failed semantic check) and completion-claim calibration. Across four models from four vendors (156 attempts per architecture), single-shot agents claimed completion on 98% of attempts and were silently wrong on 14.1%; a tool-use loop that enforces measure-then-claim cut silent failures to 3.2% and raised claim precision from 64% to 96%, and a four-model ablation attributes the change to the claim contract rather than the feedback. A planner-coder-checker pipeline matched that precision at lower coverage with no success gain. Stated confidence was miscalibrated single-shot (ECE 0.32) and became informative under verification for three of four models. Hardening our own oracle flipped 18 of 519 regraded verdicts from pass to fail and none in reverse; a blind expert audit overturned 3 of the 18 silent failures outside the one ambiguously specified task; and fourfold scaling of every tolerance band leaves the contrast intact.
BibTeX
@misc{rodrigues2026done,
  author = {Carson Rodrigues and Clive Rodrigues and Aravind Reddy G},
  title = {{Done Is Not Correct: Measuring Silent Failures and Self-Verification Calibration When LLM Agents Take CAD Actions}},
  year = {2026},
  eprint = {2611.10063},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10063},
}
Figure