Knowing When Not to Grasp: FEM-Labeled Diagnostics for Language-Model Robot Planners

Language-model planners for robots are now asked to handle soft, fragile things like fruit, sponges, and tissue. These are objects where the question is not only "where do I put the gripper" but "will squeezing hard enough to lift this crush it, given the fingers this particular robot has?" No current evaluation can say whether a planner gets this right, or why it fails when it does not. We propose a benchmark whose answers come from finite element simulation, the standard engineering method that splits an object into small volume elements and solves for how each one stretches and is stressed under load. Each task is stated in plain language. It is paired with a specific gripper, and in a deliberate fraction of cases no grasp exists that both holds the object and keeps internal stress below its damage threshold. Because the simulator computes every intermediate quantity exactly, we can grade a planner level by level. We check whether it retrieved the right physical facts, whether it judged a given squeeze correctly, and whether it noticed that the task was impossible on this hardware and could name what would need to change.

The main claim is that language-model planners will show a physical solvability gap, the robot analogue of what ToolBeHonest found for software tools. Planners will confidently propose grasps for infeasible tasks rather than abstain. We further claim that this gap does not close simply by giving the planner a fast stress-prediction tool. Planners routinely override or misread tool outputs, and the fix is a protocol in which the planner must produce a simulation-backed feasibility certificate before acting. The benchmark is the primary deliverable. The certificate protocol is the method we test on it, against strong non-language baselines that may well beat it.

Why this intersection, and what is already done

Four lines of work meet here, with unequal weight. Hierarchical task taxonomy is the evaluation lens. Benchmarks such as MVU-Eval and PhD separate perception from relational reasoning from composition so that failures can be located. ToolBeHonest adds the levels most relevant to us: deciding whether a task is solvable, finding the bottleneck sub-goal, and describing the missing tool. A known weakness of such taxonomies is that intermediate labels need expensive expert annotation or synthetic pipelines with their own artifacts. Finite element analysis removes that weakness in our setting, because the solver itself outputs the ground-truth stress field, contact region, and holding margin for every candidate grasp.

Finite element analysis is the source of ground truth and of the tool. DefGraspSim simulates grasps on 3D deformable objects with a GPU finite element solver in Isaac Gym. DefGraspNets trains a graph neural network, a model that passes messages between mesh vertices, to predict stress and deformation fields from a grasp in one step, about 1500 times faster than the solver. Inductive Biases for Predicting Deformation and Stress adds tetrahedral element features and a global context node. It also reports that rigid-body transformation error stays high, which limits how far we can trust the surrogate. We reuse these as the tool and treat their error as an experimental variable, not something to assume away.

The embodiment gap supplies the variable that makes tasks interesting. The same commanded force through a narrow finger pad produces higher peak stress than through a wide one, and different grippers have different force ranges. Existing embodiment-aware planning, notably OmniEVA, conditions plans on reach, arm length, and collisions, which are purely geometric constraints. Retargeting work such as DexMV assumes rigid objects. Bridging Embodiment Gaps on soft robots shows that rigid-arm vision-language-action models fail out of the box on a compliant arm and recover with demonstration fine-tuning, but it uses no physical model of contact stress. None of these asks whether an embodiment can do the task without damaging the object.

LLM reasoning limitations supply the hypotheses. Can LLMs Always Solve Easy Problems if They Can Solve Harder Ones? finds models that succeed on hard variants and fail on easy ones. That predicts planners whose chosen grasps look fine but whose tier-two judgments are inconsistent. Evaluating Intermediate Reasoning of Code-Assisted LLMs shows correct final answers reached through unsound steps, which is exactly what a flat success metric would miss. Divide-and-Conquer prompting acknowledges that its guarantees assume weakly coupled sub-tasks and do not cover continuum mechanics. This constrains our design: we do not ask the language model to decompose the mechanics itself, only the task around it.

The closest existing system is Make a Donut. It uses GPT-4 to decompose dough-shaping into tool stages with point-cloud subgoals, and a differentiable particle simulator for low-level control. It differs from our proposal in three ways. It has no stress or damage model, its embodiment is fixed, and it reports only end-task success, with no infeasible instances and no diagnosis. Our proposal is not "LLM plus physics simulator for deformables," which Make a Donut already is. It is a stress-labeled, embodiment-varying, solvability-aware diagnostic, together with a test of whether tool access alone fixes over-optimism.

Research Program

We build the benchmark on DefGraspSim's object set: the six objects of the inductive-biases study (polygon, cylinder, lemon, potato, sphere, strawberry) plus the real food scans used by DefGraspNets, with their published material parameters. We define a small inventory of parallel-jaw embodiments by varying finger-pad width and contact geometry, maximum grip force, and pad friction coefficient. Each embodiment is described to the planner in a text specification sheet. Each object receives a damage threshold on peak von Mises stress (a scalar summary of how hard a material element is being loaded), set per material.

For every object–embodiment pair we sweep grasp poses and forces through the full solver. We record whether the object is held against gravity, the peak stress and its location, and the margin to damage. A pair is solvable if at least one swept grasp both holds and stays under threshold. Unsolvable pairs are labeled with their bottleneck. Either the object cannot be held without exceeding the threshold, because the friction or contact area is too small, or the force range is too low to hold at all. They are also labeled with the minimal embodiment change that would restore feasibility, such as a wider pad or higher friction, found by re-running the sweep on neighboring embodiments. Tasks are then written in language, sometimes as multi-object sequences ("move the strawberries and the lemon to the tray using the narrow gripper"), so that composition and gripper selection matter.

Three tiers come directly from these labels. Tier one is retrieval. The planner must extract the relevant mass, threshold, pad area, and force limit from the object card and embodiment sheet. Tier two is relational judgment. Given a specific grasp and force, the planner must say whether it holds and whether it damages, and indicate the high-stress region. Tier three is composition and solvability. The planner must produce a feasible grasp plan for the full task, or declare it infeasible, name the bottleneck, and propose the embodiment change. Tier three mirrors ToolBeHonest's solvability, bottleneck, and missing-tool levels, with physics replacing APIs. A planner earns no credit for a correct tier-three plan whose tier-two justifications contradict the solver, which captures the unsound-reasoning failure.

The method we test is a feasibility certificate protocol. The planner may call the DefGraspNets-style surrogate, retrained across our embodiment inventory, on any candidate grasp within a fixed query budget. Before committing to a grasp, it must cite surrogate outputs showing a holding margin and a stress margin, each above a safety factor set from the surrogate's held-out error. If no candidate within budget passes, it must abstain and report which margin failed. The protocol turns the surrogate's numbers into a checkable precondition instead of advice the model may ignore. An abstention is scored correct only if the solver agrees the task is infeasible.

Baselines

The naive reference point is a language model answering every tier from the task text, the object card, and the embodiment sheet, with no tool. This measures the raw solvability gap. A second reference is the same model with chain-of-thought prompting and explicit numeric material properties. It tests whether careful written reasoning over the provided numbers is enough. We run each with at least two strong open and closed models at matched sampling settings.

The embodiment-aware prompting baseline follows OmniEVA's idea of forcing the model to reason about hardware specifications in a thinking block before acting. Because we cannot reproduce its TE-GRPO training from the file, we implement only the prompting form, extended to force and pad specifications, and we state that limitation plainly. Beating this baseline would show that knowing the hardware constraints is insufficient without stress computation.

The tool-without-protocol baseline gives the planner the same surrogate and the same query budget, with no certificate requirement. It also gets a multi-tool variant in the style of Multi-TAG, which cross-checks surrogate, Python arithmetic, and natural-language reasoning. This is the ablation that isolates our key ingredient: the only difference from our method is whether committing to a grasp is gated on cited margins.

The strongest competitor on single-grasp instances is not a language model at all. It is gradient-based grasp optimization through the DefGraspNets surrogate, as in the DefGraspNets paper, followed by a margin check, with abstention when optimization finds no passing grasp. For language-specified multi-object tasks we pair it with a language model used only to parse the task into object–embodiment pairs, so it sees the same information. It gets the same surrogate, the same query budget counted in forward evaluations, and the same safety factor. The file reports a 1500x speedup over GPU-accelerated FEM for DefGraspNets but gives no grasp-success or stress-error figure, so for this baseline's accuracy there is no reported result found on Lacuna. We will establish it ourselves. An oracle variant in which every tool call goes to the full solver gives the upper bound.

Beating the naive and prompting baselines means a large reduction in the false-feasible rate, defined as plans claimed safe that the solver says damage or drop the object, at no loss in solvable-task success. Beating the tool-without-protocol baseline means the same comparison holds at equal tool budget. Matching, rather than beating, the parse-then-optimize pipeline is an acceptable and informative outcome, discussed below.

Why not just…?

Why not just run a grasp optimizer on the surrogate and skip the language model? On single-grasp instances this will likely match or beat any language planner, and we expect it to. It handles the coupled mechanics that divide-and-conquer prompting cannot decompose. Its weaknesses lie elsewhere: it has no built-in way to read a free-form task, choose among grippers in an inventory, or explain a bottleneck in terms a person can act on. The parse-then-optimize baseline tests directly whether a thin language layer plus optimization is all that is needed. If it equals the certificate protocol on tier three, our conclusion will be that language models should parse and defer, not plan grasps. That is a useful and publishable finding, and the benchmark remains the contribution.

Why not just fine-tune a vision-language-action model on demonstrations of gentle grasps? The soft-robot transfer study shows that a few dozen demonstrations can restore task success on a new embodiment. Such a model, however, has no representation of stress and cannot distinguish infeasible from merely hard. The benchmark's unsolvable split is designed to expose exactly this, and the tier-two stress questions have no channel through which such a model could answer. A fine-tuned policy can be scored on tier-three solvable success for reference, but it cannot address the abstention question the proposal is about.

Why not just give the model the tool and trust it to use it? This is the most serious alternative, because tool access might be all that is needed. ToolBeHonest's finding that models assume capabilities exist when they do not, together with evidence of unsound intermediate reasoning in code-assisted models, suggests otherwise, but neither result is about physical tools. The tool-without-protocol ablation settles it. If the false-feasible rate on unsolvable tasks is already near the oracle's, the certificate protocol adds nothing and we will report that.

Proposed Experiments

The first decisive check measures the solvability gap. We compare solvable-task success with unsolvable-task abstention accuracy for each untooled planner. We expect a large asymmetry, and we measure it separately per embodiment to see whether narrow-pad grippers, where the embodiment gap bites hardest, produce the most overconfident plans.

The second check is hard-to-easy consistency. For each tier-three grasp a planner proposes, we ask the tier-two question about that same grasp in isolation, and we also pose easier variants with larger margins. Following Yang's finding, we predict that a meaningful fraction of correct tier-three plans come with wrong tier-two judgments. We report the rate of "right plan, wrong reason," which a flat success metric cannot see.

The third check is the core ablation: certificate protocol versus tool-without-protocol versus parse-then-optimize versus the full-solver oracle, all at matched surrogate query budgets swept from small to large. The reported metrics are false-feasible rate, tier-three success on solvable tasks, bottleneck-identification accuracy, and correctness of the proposed embodiment fix as verified by the solver.

The fourth check asks how much surrogate quality matters. The inductive-biases paper reports persistent rigid-transform error, so we degrade the surrogate by training on fewer objects and by injecting calibrated noise. We then trace how each method's false-feasible rate rises. The protocol should degrade gracefully if its safety factor tracks surrogate error. If it collapses like the unprotected tool baseline, the certificate is not doing real work.

A small physical sanity check closes the study. We take a few fruit–gripper pairs that the benchmark labels feasible, infeasible, and borderline, and attempt them on a Franka arm with two pad widths. We record visible bruising and drops, which confirms that the simulator's damage labels track reality closely enough for the benchmark's verdicts to mean something.