SCAFFOLD: Toward Warrant-Preserving Evidence Transport for Reusable LLM-Agent Skills

Nami Cho, Shailja GuptaarXiv 2026 · arXiv

TL;DR

SCAFFOLD introduces a formal framework for managing the "temporal mismatch" in LLM-agent skills, where evidence of success in a past context may not justify execution in a new one. It proposes Warranted Reuse Claims (WRC) and Warrant-Preserving Evidence Transport (WPET) as a discipline to determine which historical evidence remains admissible after a change in environment, policy, or dependencies.

What Problem Does This Paper Address?

As LLM agents become more autonomous, they rely on "skills"—reusable scripts, tool-use recipes, and workflows. However, a skill that succeeded yesterday isn't necessarily safe today.

The authors identify a critical gap: validation happens in the past, but authorization is demanded in the present. Between these two points, the world changes. An API might update its semantics, a new privacy policy might be enacted, or a underlying library might change how it rounds numbers. Current approaches like "design by contract" or simple versioning often fail to capture why a skill was considered safe and whether those reasons still hold. The problem isn't just "does the skill run?" but "is the evidence from its last run still relevant enough to justify this specific future use?"

Methods

The paper contributes a conceptual and methodological framework rather than an empirical study. The core components are:

  1. Warranted Reuse Claim (WRC): Instead of a simple "pass/fail" certificate, a WRC is a structured record $\langle s, c, \phi, E, A, D \rangle$ containing the skill version ($s$), target context ($c$), the specific claim being made ($\phi$), the evidence ledger ($E$), applicability assumptions ($A$), and known "defeaters" ($D$)—reasons the claim might be invalid.
  2. Warrant-Preserving Evidence Transport (WPET): This is the logic used to decide if evidence $e$ acquired in context $c_1$ is still admissible in context $c_2$. It relies on checking a set of premises $R(e, \phi)$ against the documented changes between contexts.
  3. Three-Way Authorization Policy: The framework moves beyond binary "Allow/Deny" to include a "Test/Review" state for unresolved uncertainties.
Fig. 1. Claim-specific evidence transport and reuse authorization. Illustration of the proposed WRC / WPET process (Sections III–V). Evidence transport does not establish claim truth or guarantee safe execution. Allow depends on all policy requirements being satisfied, no active hard prohibitions, and no unacceptable unresolved uncertainty.

Datasets

Not discussed in the paper. As a conceptual and methodological contribution, the authors do not perform experiments on existing datasets. Instead, they provide "constructed scenarios" (like a tax-calculation agent) to illustrate how the framework would function.

Key Insights and Ideas

  • Trust is Defeasible: The authors argue that trust shouldn't be a permanent quality label. It must be "defeasible"—meaning it can and should be retracted the moment its underlying assumptions fail.
  • Admissibility $\neq$ Truth: WPET distinguishes between whether evidence is admissible (relevant and preserved) and whether the safety claim is actually true. This allows for a conservative approach to safety without requiring a "proof of universal agent safety."
  • Selective Revalidation: By mapping specific evidence to specific premises, developers can avoid "blanket retesting." If a library change only affects tax rounding, you only need to re-run the tax-related tests, not the entire suite (see Figure 2).
  • The Provenance Trap: The paper warns against "shared-failure-mode dependence." If the same LLM writes a skill, generates the test cases, and then judges those tests, the resulting "evidence" is highly suspect because it likely inherits the same underlying errors in goal interpretation.
Fig. 2. Selective evidence transport after localized tax-engine change. Constructed invoice-reconciliation scenario (Section VI-B). This illustration does not report measured results. Selective reuse requires an accurate claim-dependency map, trustworthy change records, unchanged relevant premises, and current defeater checks. A common end-to-end integration check may still be required. Strong dependency-aware baselines can make the same selection; no superiority is demonstrated.

Key Results

The paper provides formal propositions rather than quantitative benchmarks:

  • Proposition 1 (Conditional Preservation): Defines the exact conditions under which evidence remains admissible across context changes.
  • Proposition 2 (Scope Fidelity): Asserts that if an "Allow" decision is bound to an enforced capability boundary (like a sandbox), the execution cannot escape that boundary.
  • Cost-Benefit Logic: The authors provide a formula (Equation 5) for a "suitable benchmark" to evaluate future implementations, focusing on minimizing harm while staying within a budget for human/compute review.

Limitations and Future Work

The authors are remarkably candid about the "stringent" nature of their proposal:

  • Incomplete Premises: If a developer fails to identify a critical dependency (a "hidden dependency"), WPET will return an incorrect positive verdict.
  • Overhead: The cost of maintaining a detailed dependency map and checking premises might exceed the cost of simply re-running all tests.
  • Trustworthy Registries: The system assumes the existence of tamper-proof registries for versions, policies, and identities, which are themselves difficult to implement.
  • Race Conditions: The framework does not yet resolve "time-of-check/time-of-use" (TOCTOU) races where a context changes in the milliseconds between authorization and execution.

The next step for this research is a longitudinal evaluation comparing SCAFFOLD-guided policies against strong, version-aware baselines to see if the added complexity actually results in better safety-to-utility trade-offs.

Summary updated

Abstract

An agent skill that succeeded yesterday may be executable today without being justified for reuse today. Tests may cover only observable contract clauses, outcome oracles may be absent, evaluations may inherit the same erroneous specification, and permissions or dependencies may change between validation and invocation. We argue that the appropriate unit of prospective trust is a Warranted Reuse Claim (WRC): a defeasible, claim-specific authorization tied to a versioned skill, target context, evidence, assumptions, and identified defeaters. The paper sharpens this position through Warrant-Preserving Evidence Transport (WPET), a proposed discipline for deciding which historical evidence remains admissible after context change. It distinguishes evidence admissibility from truth of the supported safety or effectiveness claim; states a conditional preservation result and its limits; specifies three-valued applicability and asymmetric context-transfer judgments; and illustrates how provenance dependence and policy-invalidating changes alter authorization. We explicitly relate the proposal to assurance cases, contracts, and skill-library maintenance rather than claiming novelty for their ingredients. A falsifiable longitudinal evaluation would compare WRC-guided policies with strong version-aware baselines at matched coverage, utility, and review cost. This paper contributes a formalized problem statement, testable design obligations, and reporting protocol, not an implemented verifier, empirical superiority result, or proof of universal agent safety.
BibTeX
@misc{cho2026scaffold,
  author = {Nami Cho and Shailja Gupta},
  title = {{SCAFFOLD: Toward Warrant-Preserving Evidence Transport for Reusable LLM-Agent Skills}},
  year = {2026},
  eprint = {2611.23990},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23990},
}
Figure