Region-Scoped State-Diff Contracts for Agents That Edit Protein Structures

Software agents are now asked to do structural biology: load a protein, mutate a residue, rebuild a loop, place a motif, clean up a structure collection. Checking whether they succeeded is harder than it looks. The leading scientific-agent benchmark checks the final 3D scene against exact coordinates or command logs and returns pass or fail. That rejects correct answers saved in a different reference frame. It also gives no credit for partial work, and it never asks whether the agent quietly damaged the parts of the protein it was told to leave alone. Enterprise agent benchmarks already handle the equivalent problem for databases. They take a snapshot before and after the agent runs, then check that the intended change happened and that nothing else changed. Protein structures have no such check.

We propose a structural version of that before/after check. We call it a region-scoped structural contract: a short, declarative statement of which residues the agent is supposed to change, what those residues should look like afterwards, and which residues and files must stay put. The verifier compares local structure using representations that ignore global rotation and translation. These are per-residue local distance agreement and Foldseek's discrete structural alphabet, which turns each residue's 3D neighborhood into one of 20 letters. The result is two scores: did the agent change the right part, and did it leave everything else intact. Each score comes with a residue-level report of where the agent went wrong. The central claim is that the scoping is what matters: separating a target region from a preserved region, inside a full snapshot of the agent's workspace. The choice of structural representation matters less. We expect the cheap discrete alphabet to be needed only when a task touches thousands of structures at once.

What exists and what is missing

The verification half of this proposal builds on Agent-Diff, which scores agents on containerized enterprise APIs. It compares before/after snapshots and asserts both the target change and the absence of unintended side effects. It also builds on Copy-on-Write Scoring, which splits a database diff into a structural score (right tables, rows and columns) and a content score (right values), and locates the step where the agent went wrong. AppWorld adds explicit collateral-damage tests. All three treat state as rows and strings. None has a notion of two states being geometrically equivalent but numerically different, which is the normal case for protein coordinates.

The closest existing bridge is ScienceBoard. It instruments ChimeraX inside a VM and reads its internal state directly. Its checks are binary and tied to fixed coordinates, so this proposal reuses its environment and replaces its checker. AgentPLM calls folding, stability and docking oracles in the middle of generating a sequence. That shows structural feedback helps, but its verification returns biophysical scalars, not a workspace diff. Its oracles cost seconds per call (AutoDock Vina at about 2.4 s), consistent with PDFBench finding structure prediction about 160 times slower than sequence scoring.

On the structure side, Foldseek provides the 3Di alphabet. It searches about 23,000 times faster than TM-align while keeping 88% of its sensitivity, which is what made the clustering of 214 million AlphaFoldDB structures by Barrio‐Hernandez et al. feasible. PLASMA aligns local functional substructures in about 10 ms using optimal transport over language-model embeddings. Both papers also constrain the design. Foldseek acknowledges that discretization discards side-chain and pocket detail. PLASMA struggles with active sites whose residues are far apart in sequence. We therefore treat these tools as candidate featurizers to be tested, not as assumed improvements.

Core Claim

A structural contract has three parts. The first is a target region $R$ (a residue set) with an acceptance condition, for example "local geometry at $R$ matches a reference motif within tolerance" or "residue 45 is now Trp and its neighborhood is not clashed." The second is a preserved region, the complement of $R$ minus a declared buffer, whose local structure must match the pre-task snapshot. The third is a workspace clause stating that no files or structures outside the declared outputs were created, deleted or modified.

Because every structural comparison is local, the contract accepts any correct solution regardless of which frame it is saved in, and it accepts any of the many sequences or conformations that satisfy the target condition. This addresses the specification-ambiguity problem that Copy-on-Write Scoring raises for single ground-truth snapshots. The two scores follow the Copy-on-Write split. The structural score asks whether changes occurred only where allowed. The content score asks whether the changed region meets its condition.

We predict three things. Region-scoped contracts will agree with expert judgment substantially better than exact-coordinate checks or global similarity thresholds. Most of the gain comes from scoping, not from the featurizer. 3Di-based contracts match local-distance-based ones on single-structure edits, and become the only practical option on collection-level tasks.

Research Program

We will build the verifier as a snapshot layer over the agent's sandbox. It captures the file tree and all structure files before and after a run, hashes them to find which entries changed, and evaluates contracts only on changed or declared entries.

Three featurizers will be interchangeable behind one interface. The first is per-residue local distance difference, computed without superposition. The second is the per-residue 3Di letter together with a windowed 3Di substitution score from the Foldseek matrices. The third is PLASMA local alignment, used for motif-type target conditions.

Contracts are written in a small declarative format: residue selections, featurizer, threshold, and buffer width. To reduce the manual-annotation burden that limits ComplexMCP to 47 tasks, most contracts will be compiled automatically from a task template plus a reference solution. The target region comes from residues the reference changes beyond a threshold. The preserved region is everything else. Thresholds are calibrated on a held-out set of known-acceptable alternatives.

There are two task families. Single-structure editing tasks run in ScienceBoard's ChimeraX environment, extended with a scripted PyMOL/Rosetta sandbox: point mutation with repacking, loop rebuilding, motif grafting, chain extraction, and frame-independent alignment to a reference. Collection tasks give the agent a few thousand structures, for instance a slice of AlphaFoldDB or a TEDBench subset (Protein Fold Classification at Scale). The agent must deduplicate, annotate folds, or remove entries meeting a structural criterion. Verifying these requires structural comparison across the collection, which is where Foldseek-speed comparison stops being optional.

Baselines

The naive reference point is ScienceBoard's own checker, run unchanged on its ChimeraX tasks and on our extended tasks: exact coordinate and state comparison with a pass/fail output. Here agents score around 15% (Claude-3.7-Sonnet at 15.79%, a GPT-4o planner with GUI-Actor-7B at 20.44%, against a human 60.27%). Beating it means higher agreement with expert labels on the same agent trajectories, not a higher agent score.

The strongest global competitor is a whole-structure contract: superimpose the final structure on a reference with TM-align and accept if TM-score and RMSD clear tuned thresholds. This is what a careful benchmark designer would do first, and it already fixes the frame problem. It gets the same trajectories and its thresholds are tuned on the same calibration set we use.

The strongest local competitor is an unscoped per-residue local-distance diff against the reference, accepting when the median or minimum per-residue agreement is high. It is the same as our method except for the target/preserved split, so it serves as the ablation that isolates scoping. A second ablation removes the workspace clause, measuring how much file-level collateral damage alone contributes, in the spirit of the path-tracking failures SP-Mind reports.

A model-judge baseline gives a strong VLM the final screenshot and the saved file summary. On collection tasks we also compare Foldseek-based contract checks against all-pairs TM-align checks on the same hardware budget.

For the structure-search components, the reference numbers are Foldseek's reported 86% of Dali's and 88% of TM-align's sensitivity. For the central quantity, agreement between an automatic structural verifier and expert judgment, we found no reported result on Lacuna, so the baselines must be measured here.

Why not just…?

Why not just threshold the global TM-score of the final structure? It handles reference frames and gives continuous credit, and on tasks where the whole protein should change it may be all one needs. Its weakness is averaging. Unfolding a 12-residue helix next to the edited site barely moves the TM-score of a 300-residue protein, and a correct motif graft can lower global similarity to the original. The synthetic perturbation suite described below tests this directly. If global TM-score detects localized damage as well as scoped contracts do, the scoping claim is wrong.

Why not just use per-residue local distance difference and skip structural alphabets? This is a real threat, and we expect it to win on single-structure edits. Local distance agreement is already independent of reference frame and of residue position, and it keeps detail that 3Di discards, as the Foldseek authors acknowledge. The proposal therefore makes no claim that 3Di is a better single-structure verifier. 3Di is evaluated as the featurizer that keeps collection-level contracts tractable. If all-pairs local-distance checking fits the time budget on our collection tasks, the alphabet adds nothing and we will report that.

Why not just let a vision-language model or the agent itself judge success? ComplexMCP documents agents explaining failures away in persuasive text. ScienceBoard shows current VLMs handle dense 3D molecular scenes poorly. A judge also cannot see files the agent deleted. The model-judge baseline measures this rather than assuming it.

Why not just call biophysical oracles, as AgentPLM does? Stability and docking scores answer a different question: is the result physically good, rather than did the agent do what was asked without breaking anything else. At seconds per call they are expensive for scoring many rollouts. We treat an optional energy-clash term as a complement inside the preserved-region clause and report whether adding it changes verdicts.

Proposed Experiments

The first decisive check is a controlled perturbation suite. Starting from verified correct solutions, we apply transformations that should pass: random rigid motions, alternative valid rotamers or sequences at the target, and renamed output files. We also apply transformations that should fail: local unfolding of 5 to 20 residues at varying distance from the target, off-target point mutations, clashes, and deletion of unrelated files. Each verifier is scored on false accept and false reject rates as a function of perturbation size and location. The prediction is that scoped contracts reject local damage the global and unscoped verifiers accept, with no rise in false rejects of valid alternatives.

The second check uses real agent trajectories from several agents on ScienceBoard's biochemistry tasks plus our extended tasks. Two structural biologists label each trajectory as acceptable or not and mark damaged regions. We report Cohen's kappa between each verifier and the experts, along with inter-annotator kappa as the ceiling. We also compute the fraction of ScienceBoard binary failures that experts consider acceptable, and that our verifier recovers.

The third check covers collection tasks. We compare verifier wall-clock time and verdict agreement between Foldseek-based contracts and all-pairs TM-align or local-distance contracts as collection size grows from hundreds to tens of thousands of structures. This identifies the size at which the alphabet becomes necessary, and how many verdicts flip because of its lost detail.

The fourth check tests automatic contract compilation. We take contracts compiled from a template plus reference solution and measure how often their verdicts match hand-written contracts on a 50-task subset. This establishes whether the method actually lowers the annotation cost that limits current state-diff benchmarks.

The proposal fails in a clear way if unscoped local-distance checks match scoped contracts on expert agreement and perturbation detection. It also fails if compiled contracts disagree with hand-written ones often enough that manual writing is still needed.