Axis-Explicit Tool Contracts for Agents That Write Their Own Code

Language-model agents increasingly solve scientific problems by writing small programs on the fly: they break a question into steps, write a function for each step, run it, and keep the functions that work for later reuse. The weak point is the code itself. Tensor code is full of silent mistakes. Averaging over the wrong axis, or letting a batch dimension broadcast against a feature dimension, produces a program that runs, returns a plausible number, and is wrong. The code is also tied to one library: a tool written for PyTorch will not run in a JAX pipeline. We propose that tool-writing agents should not emit imperative tensor code at all. Instead, the agent that plans a step should write a short declarative specification in einx notation, a string language that names every axis and states which axes are reduced, combined, or kept. A separate checking agent then verifies that specification against the task's stated meaning before anything runs, and the same specification compiles to whichever library the pipeline uses.

The bet is narrow and testable. Making axis roles explicit in the contract between planner and coder should catch a class of errors that execution feedback cannot catch, because those errors never crash. It should also make evolved tool libraries portable across backends for free. We test this where silent axis errors are most common and most costly: tools that measure and manipulate the geometry of multimodal embeddings, such as the modality gap between image and text clusters. Every quantity there is a reduction over batch, modality, and feature axes that are easy to confuse.

Core Claim

Beyond Static Tools: Test-Time Tool Evolution for Scientific Reasoning supplies the method we build on. Its agents split tasks into sub-goals, write code tools on demand, check that they execute, break verified tools into reusable atomic pieces, and prune redundant ones. On its SciEvo benchmark this raises accuracy from 0.56 for the best baseline to 0.62. Its verification is runtime execution, and its tools are ordinary imperative Python. Our claim is that replacing the imperative tool body with an axis-explicit declarative specification, and giving verification of that specification to a dedicated role, will lower the rate of tools that run but return wrong values. It should do so without lowering task accuracy, and it should yield tools that give numerically matching results under PyTorch and JAX without any translation step.

The notation comes from two papers. Einops introduced string specifications of input and output axes that run on several backends. It's All Just Vectorization: einx, a Universal Notation for Tensor Operations generalized this to reductions, dot products, and concatenations marked with brackets. It reports 7 to 24 ms first-call compile cost, under 0.1 ms cached, and no overhead under JAX jit. Neither paper uses the notation as a communication format between agents. That is the gap we target.

Research Program

The system has three roles. We keep it at three because MAM found that collaboration in its diagnostic system peaks at about three specialized roles and degrades at five or more. That finding constrains our design. The planner takes a sub-goal and writes a tool contract. The contract has a typed signature in which every input and output axis carries a semantic label (for example b for paired samples, m for modality, d for embedding dimension), an einx expression for each tensor step, and a one-sentence statement of intent. The checker sees only the contract and the task text. Its job is to confirm that the reduced axes match the intent, for instance "mean over samples within each modality" rather than "mean over features." It also runs a symbolic shape pass, which einx provides, on synthetic inputs whose axis sizes are all distinct primes, so that accidental broadcasting cannot go unnoticed. The executor compiles the accepted contract to the active backend and runs it. Operations that einx cannot express fall back to imperative code wrapped in a contract that states only shapes. We report what fraction of tools needed this fallback. The notation's coverage is bounded by what the backend supports, a ceiling the einx paper itself acknowledges.

Tools that pass verification enter a library keyed by contract, following the reuse-and-prune loop of the tool-evolution work. Because contracts are backend-independent, the library behaves like the shared intermediate schema in OmniEvaluator. That system cut the cost of connecting inference backends to evaluation frameworks from one integration per pair to one per component. We want the same effect for tool libraries and runtimes.

The application regime is a new task suite we will build, called GapOps. Each task asks an agent to compute or change a property of paired multimodal embeddings from released CLIP-style encoders on MSCOCO, MSR-VTT, and CIFAR10. The tasks include the gap distance between modality centroids, the effect of shifting embeddings along the gap vector on zero-shot accuracy, the alignment and centroid-uniformity losses of ModGap, and V-Measure clustering scores. The published values serve as reproduction anchors. Closing the Modality Gap Aligns Group-Wise Semantics reports a standard-CLIP gap of 0.47 on MSCOCO with V-Measure around 13.0. Mind the Gap reports that adjusting the gap moves CIFAR10 top-1 accuracy from 0.9013 to 0.9081. These are exactly the computations where swapping the sample and feature axes still yields a number.

We will not claim to reproduce these figures exactly. Checkpoints and preprocessing may differ. Ground truth for each task therefore comes from reference implementations written by hand independently in PyTorch and JAX and checked against each other. The published numbers serve only as sanity bounds. The modality-gap direction thus enters as the evaluation regime, not as something we try to resolve. We take no side in the bug-versus-feature dispute between those two papers. That dispute does make the tasks useful, because agents are asked both to widen and to shrink the gap, and the sign of a mistake matters.

Baselines

The naive reference point is a single model with chain-of-thought prompting that writes imperative NumPy or PyTorch for each task and runs it once. It shows how often unaided code is silently wrong. The strongest existing method is test-time tool evolution as published in Beyond Static Tools, which reports 0.62 SciEvo accuracy against 0.56 for its best baseline and a tool reuse rate TRR@1 of about 1.0. It is the strongest competitor because it already decomposes, verifies by execution, and reuses tools, so any gain we report must come from the contract format and the checker. We will rerun it ourselves with the same base language model, the same token budget per task, and the same number of repair attempts. A gain only counts if it comes from the representation and not from extra compute.

A stronger version of that competitor adds execution-feedback repair, meaning tracebacks and shape errors are fed back for retries until the budget runs out. Beating it means a lower silent-error rate on GapOps at equal or better SciEvo accuracy. For portability, the baseline is tool evolution in PyTorch followed by language-model translation of each tool into JAX. Beating it means higher cross-backend numerical agreement at lower token cost.

Two ablations isolate the key ingredient. In the first, the planner writes einx contracts but no separate checker exists, so the planner verifies its own work. This tests whether the separate role matters. In the second, a checker reviews imperative code with semantic axis comments instead of einx contracts. This tests whether the notation matters, or whether any careful review would do. If the second ablation matches the full system, the notation is not the active ingredient, and we will say so.

Why not just…?

Why not just use execution feedback and retries? This is the most serious threat. On SciEvo-style tasks, where answers are checked against final values, retries may close most of the accuracy gap, because a wrong final answer is itself a signal when the agent has some way to verify it. Retries cannot help when nothing fails, though, and on GapOps the agent has no reference answer at run time. We therefore split GapOps errors into crashes and silent wrong values and compare the two systems on each type. If the retry baseline matches us on silent errors as well, the proposal reduces to its portability claim and we will report it as that.

Why not just write plain NumPy, or convert with ONNX? The einx paper argues that NumPy-style code relies on positional axes with inconsistent conventions (axis=1 versus dim=-1) and on implicit broadcasting, which is the source of silent shape errors. Plain NumPy does not run inside JAX-traced or GPU-native pipelines without rewriting. Graph converters operate on exported models, not on short scripts an agent writes and discards mid-task. The NumPy arm of the naive baseline measures the first issue directly.

Why not just let one strong model write einx itself? That is the first ablation. The prior evidence is mixed. Role separation helped in Self-collaboration Code Generation via ChatGPT and in MACM, but more roles also adds cost. We report tokens and wall-clock time alongside accuracy, and if self-checking is as good, we adopt it.

Why not just translate PyTorch tools into JAX with the language model when needed? This is the portability baseline. Translation can introduce new silent errors and costs tokens on every port. Our contracts pay nothing extra per backend. Cross-backend agreement settles the question.

Proposed Experiments

The decisive experiment is the silent-error rate on GapOps: the fraction of tools that run to completion but disagree with the reference by more than numerical tolerance, compared across the full system, the retry baseline, and the two ablations at matched budgets. We will also run a seeded-bug detection test. We take verified reference tools, inject axis mutations that still run (swapped reduction axes, transposed pairing, broadcast of a per-sample term against a per-feature term), and present the same bug to the checker either as an einx contract or as imperative code. We then measure detection rates. If the detection rate is not clearly higher for contracts, the core claim fails.

Secondary checks cover four things. On SciEvo, accuracy should not fall below the reproduced tool-evolution number, compared against the reported 0.62. The library evolved under PyTorch should agree with itself under JAX within floating-point tolerance on all GapOps tasks. We also report the fraction of tools that needed imperative fallback, and the token and latency overhead per task, since the tool-evolution work already flags cost as a weakness. A result where contracts cut silent errors substantially, keep SciEvo accuracy, and port without translation would show that the format agents use to talk to each other about tensors is a design choice with measurable effects on correctness.