CRAFT: A Robot-Agent Harness with Scope-Aware Adaptation

Peng Zhang, Jiawen Wang, Cheng Xue, Jochen RenzarXiv 2026 · arXiv

TL;DR

CRAFT is a new framework that helps robots learn "correction reach"—deciding whether a human's "Not there!" applies to a specific spot or represents a broader rule. By using a "scope ladder" that prioritizes specific memories for familiar spots while mixing rules and learners for new ones, robots can adapt to local requirements more efficiently and with fewer annoying questions.

What Problem Does This Paper Address?

Imagine a robot assistant in your home. It places a glass of water near your laptop, and you say, "Not there." Does that mean "don't put drinks near the laptop," or "don't put drinks near the edge of the table"?

This is the problem of correction reach. If a robot interprets your feedback too narrowly, it will make the same mistake in a slightly different spot. If it interprets it too broadly, it might stop putting drinks on the table entirely.

Current robots often have "frozen" skills—they are excellent at the physical act of moving objects—but they struggle to learn the unstated, site-specific rules of a new environment. The authors tackle the challenge of how a robot should generalize sparse "yes/no" feedback to ensure its actions are not just physically successful, but also contextually acceptable.

Methods

The authors introduce CRAFT (Contextual Requirement Adaptation with Frozen Tools), a harness that separates physical execution from task acceptability. Within this framework, they propose the Scope Ladder, an arbitration method that chooses how to interpret feedback using three levels:

  1. Memory Level (M): Uses k-Nearest Neighbors to recall specific past verdicts.
  2. Rule Level (G): Uses Bayesian inference over a pre-defined grammar of spatial and attribute-based rules (e.g., "Keep away from red objects").
  3. Open Level (O): Uses a logistic regression learner to catch patterns that don't fit the explicit grammar.

The Secret Sauce: The Gate and Weighting The system doesn't just average these levels. It uses a specialist gate: if the robot finds itself in a "near-identical" situation it has seen before, it trusts its Memory implicitly. If the situation is new, it uses an evidence-weighted mixture. Each predictor (Rule, Open, Memory) is weighted based on how accurately its past predictions matched the feedback eventually received. This allows the robot to "learn who to trust" as it gains experience at a specific site.

Datasets

The research relies entirely on simulated environments to maintain controlled, reproducible studies:

  • Decision-Level Simulations: 150 "fresh" tabletop sites with procedurally generated rules and contexts.
  • Continuing-Service Pilot: A more complex simulation involving a fixed-pelvis R1 robot performing pushing tasks.
  • Baselines: The authors compared the Scope Ladder against kNN-memory, LLM-based agents (including a frontier model referred to as gpt-6.1-sol), and adaptations of existing frameworks like DROC and CarryOn.

Key Insights and Ideas

  • Physical Success $\neq$ Acceptability: By treating "did the arm reach the target?" and "was the target a good choice?" as separate signals, the robot avoids learning "false rules" from mechanical failures.
  • The "Specialist" Priority: One of the biggest "aha" moments was that rules are great for new places, but they are often noisy. If you are in the exact same spot where a human previously complained, your specific memory of that complaint should override any general rule.
  • Prequential Arbitration: Instead of picking one "best" model, the system constantly re-evaluates its predictors. If the "Rule" level starts failing because the human's preferences are quirky, the "Open" or "Memory" levels naturally gain more influence.

Key Results

  • Better Utility: In confirmatory studies, the Scope Ladder significantly outperformed standard kNN-memory, showing a utility gain of +0.074 on new contexts and +0.052 on repeated ones.
  • Fewer Mistakes, Fewer Questions: When allowed to ask for help, the Ladder+ask configuration took 14% fewer wrong actions on new contexts and 27% fewer on repeated ones compared to memory-only baselines, all while asking fewer questions.
  • LLM Comparison: While frontier LLMs were capable of complex reasoning, the Scope Ladder was more efficient. In the continuing-service pilot, the ladder completed more jobs (96/120 vs 94/120) and asked significantly fewer optional questions (6 vs 11) than the LLM-based agent.
  • The "Gate" Matters: Removing the memory gate (the rule that prioritizes exact repeats) caused performance to drop on familiar tasks, proving that generalization shouldn't come at the expense of specific experience.

Limitations and Future Work

  • Unannounced Rule Changes: The system still struggles when a human changes their mind without saying so (e.g., "I used to want drinks on the left, now I want them on the right"). The "staleness" of old memories remains a difficult nut to crack.
  • Simulation Gap: All results are currently in simulation. Real-world human feedback is often noisier, delayed, or ambiguous, which will provide a much tougher test for the Scope Ladder.
  • Grammar Constraints: The "Rule" level is only as good as the grammar provided to it. Future work could involve using LLMs to dynamically expand the grammar when the robot encounters a requirement it can't currently express.

For those interested in the implementation, the authors have committed to releasing the code at github.com/zp312/CRAFT.

Summary updated

Abstract

A robot may know how to move an object yet still need to learn where it belongs. When a person rejects a placement, the robot must decide whether the correction applies only to that situation or expresses a broader rule. We study this problem in CRAFT (Contextual Requirement Adaptation with Frozen Tools), a framework for learning local requirements without retraining robot skills. It records physical success separately from whether an action meets the requirement. Our scope ladder uses memory when an action has already been judged in a nearly identical situation. Elsewhere, it combines memory, explicit rules and a learner for requirements beyond those rules, weighting their predictions by how well they predicted earlier feedback. Controlled studies show gains over correction memory alone in utility, which rewards acceptable actions and charges for mistakes and questions. In camera-grounded robot simulations, the ladder asks fewer questions than language-model agents with persistent memory, but some of those agents take fewer wrong actions and succeed more often after requirements change. All execution failures and timeouts remain in the evaluation. These results support combining specific memories with broader rules, while leaving reliable adaptation to changing requirements unresolved. All evidence comes from simulation with generated requirements and simulated feedback.
BibTeX
@misc{zhang2026craft,
  author = {Peng Zhang and Jiawen Wang and Cheng Xue and Jochen Renz},
  title = {{CRAFT: A Robot-Agent Harness with Scope-Aware Adaptation}},
  year = {2026},
  eprint = {2611.19952},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19952},
}
Figure