Executable Handoff Contracts for Multi-Agent Data Pipelines

When a team of language-model agents builds a data pipeline, with one agent finding the right files, another writing the joins, and another computing the final analysis, a single silent mistake early on can poison everything after it. Most such mistakes are not crashes. A wrong table chosen, a join that silently duplicates rows, a column misread as a date: each of these runs without error and hands later agents a plausible-looking artifact they have no reason to doubt. The benchmark evidence is stark. On DAComp, models write individual transformation steps reasonably well, yet strict end-to-end success on the data-engineering track stays below 10% for almost all models. The benchmark scores every node below a failed node as zero, which makes this cascade visible (DAComp). We propose that each agent boundary carry a small, executable contract. The consuming agent states, as runnable checks, what it needs from its input, for example "the key customer_id is unique" or "the date column parses with fewer than 1% nulls." The producing agent states what it guarantees, and it also keeps its runner-up candidates (the second-best file, the alternative join key) instead of discarding them. A cheap checker runs the contract before the downstream agent consumes anything. A violation sends work back to the node that produced the bad artifact rather than letting it flow on.

The idea borrows a lens from the study of human workflow fragmentation, which documents what people lose when they act as "manual clipboards" that carry state between disconnected tools (The TaskTracker System). A decomposed agent pipeline fragments its own workflow in the same way. Every handoff is a lossy copy-paste in which the downstream agent receives a hard artifact stripped of the upstream agent's uncertainty. Our central claim is that cascades in agent pipelines come mainly from these unchecked, lossy handoffs, not from weak individual steps. If that is right, making handoffs checkable should do two things. It should shrink cascades, and it should shift the best number of agents upward, so that finer decomposition pays off where today it hurts.

Core Claim

The proposal has three testable parts. First, consumer-stated executable preconditions catch a large share of the silent upstream errors that self-debugging driven by crashes cannot see. Second, passing the top few alternatives across a boundary, instead of a single hard choice, lets a violated contract be repaired by local backtracking rather than full re-planning. This addresses the "information starvation" that pipeline work has long identified, in which early discrete decisions discard alternatives that later stages needed (The University of Sheffield System at TAC KBP 2010; Entity-Focused Dense Passage Retrieval). Third, contracts change the shape of the tradeoff between decomposition depth and accuracy. The Price of Over-Delegation models how splitting work across too many agents raises cascading failure when handoff costs are not internalized. Contracts act directly on the per-handoff error cost in that model, so they should push the best depth deeper. That paper's analysis covers only line and tree workflows and explicitly does not handle reconverging DAGs. The real data pipelines in DAComp have exactly that reconverging shape, so our depth sweep also tests empirically whether the theory's qualitative prediction survives that extension.

Relation to Prior Work

Three lines of work motivate the gap. Post-hoc attribution systems locate the origin of a failure after the run is finished. Aegis trains diagnostic models on 9,533 deliberately sabotaged multi-agent trajectories. CORRECT retrieves cached "error schemas" to help an LLM find the failing step. Both work on finished logs, and Aegis reports that saying why a step failed is much harder than saying which agent failed. Our approach sidesteps the "why" problem at runtime by making each boundary's expectations explicit, so a violation arrives already labeled with the property that broke. ReCAPA does intervene during execution. It aligns actions, subgoals, and trajectories in embodied agents through learned contrastive and optimal-transport objectives. It works in continuous simulated environments, though, and requires training. Our contracts are training-free and symbolic, because data artifacts, unlike robot trajectories, can be checked exactly. We do borrow ReCAPA's idea of measuring error propagation as a function of lag, and we adapt it to depth in a DAG.

The method's closest relative is Seeing Before Agreeing. It stops agents in visual question answering from converging on wrong answers by requiring them to commit to atomic, evidence-grounded claims before arbitration. We carry that move into code and data. The "atomic claim" becomes a runnable assertion, and a deterministic checker plays the role of the grounding step. The constraining evidence comes from notebook agents. DatawiseAgent shows that a structured single agent with incremental execution, self-debugging, and pruning of failed traces does well on data science benchmarks. That result warns us that much of the gain may come from execution feedback alone, so we treat it as a primary baseline and not a straw man. Its pruning also creates a tension that contracts help resolve. Pruning keeps the coder's context clean but throws away diagnostic evidence. A contract is a compact, persistent record of what each boundary promised, so it survives pruning. The LLM-based Multi-Agent Blackboard System supplies a discovery stage we reuse, since it improves data discovery in unorganized data lakes. CoDA-Bench shows why discovery matters: agents fail to find the right files in about 20% of tasks, and oracle data lifts execution accuracy by 25 to 28%. Discovery is exactly the boundary where keeping runner-up candidates should pay off most.

Research Program

We will build a pipeline harness in which tasks are broken into a DAG of agent nodes, with the granularity set as an experimental control. At one end is a single agent; at the other, separate agents for discovery, schema mapping, each transformation, and analysis. At every edge, the consuming agent is prompted to write preconditions as short Python or SQL checks against its expected input. The producing agent writes postconditions and returns its top three candidates for any discrete choice, such as file, join key, or filter interpretation. A checker that uses no language model runs both sides. On a violation, the harness first tries the producer's next candidate. Only if every candidate fails does it re-invoke the producer, passing along the violation message. Contracts are bounded in size so that their token overhead can be reported and controlled. The overhead matters because coordination cost is a documented weakness of multi-agent designs: the blackboard system reports roughly 1.8 times the tokens of a centralized master agent.

A secondary human-facing component tests the fragmentation claim directly. Debug Smarter, Not Harder found that an agentic debugger inside a notebook raised negative-productivity ratings (1.57 versus 1.31) because its interface exposed too much. Our hypothesis is that contract violations are the right unit to show a user: one line per broken promise at the cell where it broke. We will compare three notebook conditions built on the same backend. One shows the full agent trace, one shows only the final output, and one shows only contract violations.

Baselines

The naive reference point is the same decomposed DAG pipeline with contracts removed. Each agent receives the upstream artifact and proceeds, and crashes are reported but not routed. Every other condition is measured against it, and it isolates what the contracts add.

The monolithic reference is a single frontier agent with the whole task in context. On DAComp, the authors report a GPT-5 aggregate data-engineering score of 42.88% and a data-analysis score of 50.84%, with strict data-engineering success below 10% for almost all models (DAComp). On CoDA-Bench, the best reported result is 61.1% execution accuracy on the full benchmark and 49.6% on CoDA-HARD, from Mini-SWE-agent with GPT-5.5 (CoDA-Bench). We will rerun these on our own model backbone rather than compare against the reported numbers directly, and we cite the reported figures only to calibrate the difficulty.

The strongest competitor on our specific question is execution-driven self-debugging in the style of DatawiseAgent. It runs each step incrementally, repairs on exceptions, and prunes failed traces. Execution feedback is the cheapest signal available, and it is a well-established source of gains on data tasks. The file gives no number for DatawiseAgent on DAComp or CoDA-Bench (no reported result found on Lacuna for DatawiseAgent on these benchmarks), so we will reimplement its loop inside our harness.

The second strong competitor is an online LLM critic. We run a CORRECT-style detector, with retrieved error schemas, at every edge before handoff (CORRECT). CORRECT reports up to 28.7% gains in step-level error localization on its own benchmark. That result is post-hoc, and we know of no reported online result. The critic tests whether a model reading the trace can do the contract's job without executable checks.

For fairness, all methods share one backbone model, identical tool access and sandbox, and the same task splits. Each receives the same per-task token and wall-clock budget, with the critic and contract-writing calls counted against it. Each also gets the same prompt-tuning budget on a held-out development split. Beating the strongest baseline means higher strict DAComp success and a lower downstream zero-out rate under the cascading failure score at matched or lower token cost, with gains concentrated on silent rather than crashing errors.

Ablations isolate the key ingredients. We compare consumer-written versus producer-written contracts, contracts with versus without runner-up candidates, and executed checks versus the same checks read and judged by an LLM. The last comparison separates "making expectations explicit" from "checking them exactly."

Why not just…?

Why not just rely on execution feedback and self-debugging? This is the most serious threat. If most DAComp cascades start as runtime exceptions, a self-debugging loop catches them and contracts add little but cost. Prior work hints otherwise. CoDA-Bench's failures in file discovery and its large oracle-data jump describe errors that execute cleanly on the wrong data. Still, we will not assume this. Before building anything else, we will label baseline failures on DAComp and CoDA-HARD as crashing or silent. If silent errors are a small share, we will narrow the claim to discovery boundaries or drop it.

Why not just put an LLM critic at every handoff? A critic is more flexible than a fixed assertion and needs no executable form. Aegis's finding that models struggle to say why a step failed suggests that critics reading traces will miss property-level errors. The executed-versus-LLM-judged ablation tests this at matched budget. If the LLM-judged contracts match the executed ones, the contribution reduces to "ask the consumer to state its needs," and we will report it that way.

Why not just use one strong agent and skip decomposition? If self-written assertions help a single agent as much as they help a pipeline of agents, decomposition is beside the point. Our depth sweep includes depth one with contracts, so this outcome would be visible and would be reported. It would still be a useful result about checking, though not about handoffs.

Why not just fix discovery with oracle-quality retrieval? Better discovery, for example the blackboard design, addresses one boundary. It does not address the transformation boundaries where DAComp's cascades also occur. We include the blackboard discovery stage in every condition so that contracts are credited only with gains beyond it.

Proposed Experiments

The first decisive check is error containment under injection. Following Aegis's sabotage idea, we inject controlled silent errors into a single upstream node of DAComp-DE and CoDA-HARD pipelines: a wrong file, a duplicating join, or a mistyped column. We then measure the fraction that reach the final output as a function of DAG depth below the injection point. This is a DAG analogue of ReCAPA's error propagation rate. Contracts should flatten that curve relative to all baselines.

The second check is the depth sweep. We run 1, 2, 4, and 8 agent nodes per task, with and without contracts. We report strict success and the cascading failure score. The over-delegation model predicts an accuracy peak at shallow depth without contracts. Our claim predicts that the peak moves deeper with contracts. If the peak does not move, the claim that contracts make decomposition safe fails, even if overall accuracy rises.

The third check is the natural-error benchmark comparison against all baselines at matched budget. We report DAComp strict success, aggregate scores, and the cascading failure score, along with CoDA-Bench execution accuracy and token cost. We also report how often a violated contract names the node that a human annotator judges to be the true origin.

Finally, a small within-subjects study with notebook users compares the three exposure conditions on real debugging tasks. It uses the error-resolution and negative-productivity ratings from Debug Smarter, Not Harder. The prediction is that showing only violations keeps the resolution benefit of the agentic debugger without its added overload.