Talking in Jacobians: Language-Model Deliberation over Per-Part Deformation Gradients

When someone asks a 3D editing system to "make the giraffe's neck longer and slightly thicker, and bend the tail upward," every phrase describes a local, relative change: stretch along an axis, scale across it, rotate a segment. None of it says where any point should end up in absolute space. Current language-driven mesh editors still ask a vision-language model (a language model that also reads images) to produce absolute movements. In Handle-based Mesh Deformation Guided By Vision Language Model, GPT-4o draws 2D arrows on six rendered views, and a voting step merges them into a 3D displacement. In Geometric Red-Teaming for Robotic Manipulation, GPT-4o picks which points to move and a black-box numerical search decides how far. Both papers report the same weakness. The model has no sense of 3D volume, so the edits produce self-intersections and give no account of what changed or why.

We propose to give the language model a different vocabulary. A Jacobian field assigns each triangle a 3×3 matrix saying how that triangle rotates, stretches and shears; it is the differential description of a deformation. The model would write its edit as a short program of per-part Jacobian operations, such as "stretch the neck 1.4× along its axis" or "taper the tail toward its tip." A Poisson solve then stitches those local transforms into one watertight mesh. The model works in a chain of thought, meaning it writes out intermediate reasoning steps before it answers. Here each step proposes an operation, a geometry tool executes it, and the tool returns renders plus measured quantities: part lengths, smallest triangle determinant, and self-intersection count. The model checks these against the instruction and revises. Our claim is that Jacobians are the natural meeting point between language and geometry. They match how people describe shape changes, they do not depend on viewpoint, and they produce smooth, connected surfaces by construction. That leaves the reasoning loop free to spend its effort on meaning and on checking the result.

Core Claim

The claim is that measurable, compositional shape edits become more accurate and more valid under one specific design. The model does not output positions. It reasons in steps over a small set of intrinsic per-part Jacobian operators, it is grounded by measurements computed on the actual mesh, and the result is integrated by a Poisson solve. We expect this design to beat both VLM-predicted displacements and diffusion-guided Jacobian optimization. It should also cost minutes or less rather than tens of minutes, because no diffusion model sits inside the loop.

The closest work differs in where the language model touches the geometry. Sun et al. decompose the instruction into sub-instructions, which is already a light form of chain of thought. Their model then outputs 2D displacement vectors, and the mesh is solved with membrane energy on vertex coordinates. Goel et al. do use Jacobian-field deformation, but the language model is consulted once to pick points, and all quantitative choices go to a gradient-free search called TOPDM. CAD-Coder shows that a reasoning chain followed by an executable geometric program, trained with a Chamfer-distance reward, works very well. Its programs, however, are constructive CAD scripts that cannot edit an existing organic mesh. We take CAD-Coder's "plan, then write an executable geometric program, then verify" structure and move it from building CAD solids to deforming existing triangle meshes.

Research Program

The operator language. The model never names a vertex or a coordinate. Each operation names three things. The first is a semantic part. We obtain parts with the multi-view segmentation and graph-cut projection of Sun et al., which we reuse unchanged. The second is a transform type: stretch, thicken, taper, bend, twist, or rigid rotation about a joint. The third is a small set of scalar parameters, such as factor, angle, and falloff width.

The tool, not the model, computes each part's local frame from the principal axes of its surface. This removes the 3D-coordinate guessing that the research file identifies as the main spatial failure of text-only reasoning. The tool then converts each operation into target matrices $J_f$ for every face $f$. For example, a stretch is $R\,\mathrm{diag}(s,1,1)\,R^\top$, where $R$ is the part frame and $s$ is the factor. Faces outside the part keep the identity, and faces near part boundaries blend smoothly between the two. Vertices are then recovered by the standard screened Poisson problem used in APAP and DreamMesh:

$$V^* = \arg\min_V \sum_{f} a_f \,\lVert \nabla_f V - J_f \rVert_F^2 + \lambda \sum_{i \in \mathcal{A}} \lVert v_i - \bar v_i \rVert^2 .$$

Here $V$ is the set of vertex positions and $\nabla_f V$ is the gradient of the deformed positions on face $f$. The weight $a_f$ is the area of face $f$. The set $\mathcal{A}$ contains anchor vertices held at their rest positions $\bar v_i$, and $\lambda$ sets how strongly those anchors are held. Mesh connectivity never changes, so the Laplacian can be factored once and each later step costs only a back-substitution. That makes many reasoning rounds affordable, unlike the roughly 30 minutes DreamMesh reports per sample.

The grounded reasoning loop. We follow SceneCOT, which raised grounding coherence on Beacon3D from 20.4 to 34.7 by pausing generation to call 3D perception tools. We also follow ReFocus, which beat text-only reasoning by 2.6% by making image edits part of the reasoning. Our model alternates between a written rationale and an operator call. After each call it receives multi-view renders and a numeric report. The report gives measured part lengths and thicknesses relative to the original, the minimum and distribution of $\det J_f$, a flipped-triangle count, a self-intersection count from a bounding-volume-hierarchy test, and how far anchors drifted.

Self-intersections are detected but not prevented, and we report that limitation honestly. The model is expected to respond to a detected intersection by changing the operation, for instance by bending less or moving the anchor. This follows the procedural structure that SoS1 found worth up to 21% over unguided reasoning. Each round follows the same protocol: restate the target, propose an operation, read the report, compare the report to the target, then either commit or revise.

Training stage (secondary). The main system needs no training and uses an off-the-shelf vision-language model. As a second stage, we would fine-tune an open 7B model with group-relative policy optimization, the reinforcement-learning method CAD-Coder used, on the measurable benchmark described below. The reward is target-ratio error plus validity penalties. This tests whether verifiable geometric rewards transfer from CAD to deformation.

Baselines

The naive reference point is the unedited mesh, together with a direct-coordinate variant. In that variant the same language model is asked to output 3D handle displacements, and these are fed to classical as-rigid-as-possible deformation, which keeps each local patch as close to rigid as it can. The direct-coordinate variant tells us how much is lost when language must express absolute positions.

The strongest existing method for text-instructed deformation is the pipeline of Sun et al.: sub-instruction extraction, part segmentation, cone-singularity handles, six-view 2D vote, and membrane energy. Its authors report the highest GPTEval3D scores against TextDeformer and MagicClay, with an average of 8 minutes per model versus 15–74 minutes for those competitors. The file gives no numeric GPTEval3D value, so we must re-run it ourselves. For fairness, we will run it with the same underlying vision-language model, the same part segmentation, the same source meshes, and a matched budget of model calls. Beating it means better instruction-alignment scores and lower target-ratio error at equal or lower wall-clock time, with fewer self-intersections.

The second strong competitor is diffusion-guided Jacobian optimization in the style of APAP. Here the text instruction drives score-distillation gradients (gradients from a 2D image-diffusion model, backpropagated into the 3D mesh) through the same differentiable Poisson solve. On its own handle-dragging benchmark, APAP reports a k-NN GIQA image-quality score of 4.887 versus 4.753 for ARAP, and a 59.17% user preference. These numbers describe a different task and serve only as evidence that diffusion priors help plausibility. We will give this baseline the same parts and anchors and report its quality at its natural compute budget and at ours.

Three ablations isolate our ingredients. The first is single-shot operator programs: the full program is written once, with no tool feedback. The second keeps the loop but returns renders only and withholds the numeric report. The third keeps the loop and the report, but the model outputs per-part handle displacements rather than Jacobian operators, and the same Poisson backend is used. That third ablation is the decisive test of the "Jacobian vocabulary" claim.

Why not just…?

Why not just plug a Jacobian/Poisson backend into Sun et al.'s pipeline? This is our third ablation, and it is a real threat. If 2D-vote displacements fed into Poisson integration match our operator language, then our contribution reduces to the loop alone. Two kinds of edit should separate the methods. Thickening and twisting have no clear handle point to drag. Large stretches of thin parts make 2D voting ambiguous under foreshortening, when a part points toward the camera and looks shorter than it is. We will report results split by edit type so the answer is not averaged away.

Why not just run diffusion-guided Jacobian optimization from the text prompt? It has a strong plausibility prior that our system lacks. It also costs tens of minutes per edit, which DreamMesh reports as about 30 minutes, and it has no means of hitting a quantitative target like "40% longer." It is also prone to inconsistent multi-view artifacts. Our measurable benchmark tests the quantitative-target weakness directly. Plausibility on free-form edits might still favor diffusion. If so, the honest result is a hybrid: our program as the starting point, with a brief diffusion refinement afterward. We will test that hybrid.

Why not just sample many single-shot programs and keep the best one by a VLM judge or the numeric report? With cheap Poisson solves, best-of-N search could match iterative deliberation. We will compare at matched numbers of operator executions and model tokens. If best-of-N wins, the conclusion is that the operator language matters and step-by-step reasoning does not, and we would report it that way.

Proposed Experiments

Measurable-edit benchmark. We will build a benchmark from watertight, manifold meshes drawn from the APAP-BENCH shapes, YCB objects, and PartNet-Mobility, filtered for the Poisson requirements. Each mesh gets a part segmentation and 2–4-step compositional instructions with numeric targets, such as "legs 30% longer, seat 20% thinner, backrest tilted 15° back." The primary metrics are relative error between requested and measured ratios or angles, flipped-face rate, self-intersection count, drift outside the edited parts, and wall-clock time.

Free-form editing. On the instructions used by Sun et al., we report GPTEval3D and a pairwise human study modeled on APAP's user-preference protocol. The decisive checks on this benchmark are two. The first is whether the full system beats Sun et al. and the displacement-output ablation on target error at matched calls. The second is whether the numeric report, rather than renders alone, accounts for most of the drop in self-intersections.

Explained red-teaming. In the Geometric Red-Teaming suite, TOPDM drives grasping and insertion success from 90% to below 25%, and insertion as low as 22.5%. We will replace TOPDM's random proposals with reasoning-proposed operator programs that are conditioned on simulator rollouts. We then measure how many rollouts are needed to reach the same failure level. We also have humans rate whether the model's stated cause, for instance "thickening the plug tip prevents seating," predicts which held-out deformations cause failure.

A negative outcome here would be informative too. It would show that language deliberation helps where intent is semantic but not where the objective comes from physics, which is a useful boundary on how far chain-of-thought reasoning extends into geometry.