HospitalBench: Diagnosing Where and Why Embodied Agents Fail on Long Hospital Task Chains

Gaytri Jena, Kapil Wanaskar, Aman Chadha, Vasu Sharma, Vinija Jain, Amitava DasarXiv 2026 · arXiv

TL;DR

HospitalBench is a new diagnostic benchmark designed to uncover exactly where and why embodied agents fail during long-horizon hospital chores. By using "causal chains"—where the physical state or information left by one sub-task dictates the success of the next—it reveals how small errors in positioning or memory compound into total task failure.

What Problem Does This Paper Address?

Current benchmarks for embodied AI (robots driven by Large Language Models or Vision-Language Models) often focus on "end-task success." If a robot fails to set up a surgical tray, we know it failed, but we don't necessarily know why. Did it miss a grasp? Did it forget which tool the surgeon asked for five minutes ago? Or did it park its trolley three centimeters too far to the left, making the next reach impossible?

The authors argue that long-horizon tasks fail at the "seams" between skills. Most existing benchmarks place sub-tasks side-by-side in a "clean" state. HospitalBench instead forces agents to operate in a continuous simulation where errors propagate. This "link dependence" is critical for diagnosing the robustness of agents intended for complex, real-world environments like hospitals.

Methods

The researchers developed HospitalBench within NVIDIA Isaac Sim. The core framework consists of:

  • Causal Task Chains: Each of the 10 chores consists of five scored sub-tasks. Crucially, the "pass band" of one task is wider than the "re-grip band" of the next, meaning the robot must perceive and adapt to the specific result of its previous action.
  • Link Kinds: The paper defines four types of dependencies (links) between sub-tasks:
  • V (Vehicle/Cart pose): The position of a mobile base affects subsequent reaches.
  • C (Choice): A decision made in step 1 (e.g., which bay to attend) sets the target for step 2.
  • P (Placement): Where an object was placed determines where it must be picked up.
  • F (Fact seen once): A piece of information (like a "KEEP" card) is shown only once and must be remembered for a later step.
  • Skill-Level API: Agents interact via high-level calls like pick, place, drive_to, and open_drawer.
  • Scripted Robot Proof: To ensure tasks are actually solvable, the authors wrote "scripted" programs that use the same API as the AI models. If the script can't pass, the task is poorly defined.

Datasets

The paper introduces the HospitalBench benchmark:

  • 10 Hospital Chores: Ranging from instrument counting to biohazard bagging and infusion alarm rounds.
  • Environment: Six distinct hospital areas built with 74 unique NVIDIA SimReady assets.
  • Comparison: The authors compared HospitalBench against 17 existing benchmarks (e.g., CALVIN, BEHAVIOR-1K, LIBERO), highlighting that HospitalBench is unique in its combination of continuous chaining, "fact-based" memory links, and explicit "link testing."

Key Insights and Ideas

  • The Memory Bottleneck: One of the most striking findings was the failure of "F" (Fact) links. In task T1, agents were shown a card indicating which forceps to keep. Many models failed the subsequent sub-task because they either forgot the instruction or couldn't map the visual "fact" to the physical action of picking the other forceps.
  • Compounding Physical Errors: Small drifts in the robot's own odometry (position estimation) led to "tool-use breakdowns." If the robot thinks it is 5cm away from where it actually is, its pick calls will consistently fail, even if its high-level reasoning is perfect.
  • Recovery is Hard: While the models often tried a different call after a failure (168 out of 189 failed calls were followed by a different attempt), they rarely successfully "unstuck" themselves to complete the entire chain.
  • The "Seam" Diagnosis: By tracking exactly which link (V, C, P, or F) caused the break, the benchmark provides a "passed-in-a-row" score that is far more informative than a binary success/fail metric.

Key Results

The paper presents preliminary results from 15 "walks" (full task attempts) using models like Claude 3.5 Opus (aliased as Opus 5.5) and GPT-4o (aliased as GPT-6.1):

  • Low Success Rate: Only 2 out of 15 runs passed all five sub-tasks. Both successes were on the same task (T1).
  • Early Attrition: On average, models only passed 1.6 to 2.4 sub-tasks before failing.
  • State Drift: The median drift between the robot's estimated position and its true position was 6.7 cm, which is enough to cause failures in precision tasks like grasping surgical tools.
  • Failed Calls: 19% of all API calls failed. Of those, 28% were "repeated" calls where the model simply tried the exact same failing action again.

Limitations and Future Work

  • Scale: This version reports on only 15 of 60 planned runs and covers only 3 of the 10 chores with model data.
  • Model Diversity: No open-weight models (like Llama) or specialized Vision-Language-Action (VLA) models have been tested yet.
  • Link Testing: The rigorous "Link Test" (systematically moving objects to see if the robot can still find them) has only been fully executed on 1 of the 40 links.
  • Sim-to-Real: As with most simulation benchmarks, there is an unaddressed gap between the Isaac Sim environment and physical hospital hardware.

The authors plan to release the full suite of scenes, task chains, and the evaluation harness to the research community to encourage more diagnostic-heavy embodied AI research.

Summary updated

Abstract

Long-horizon embodied tasks fail in ways that end-task success cannot see. HospitalBench is a diagnostic benchmark of ten hospital chores in NVIDIA Isaac Sim, each two unscored warm-up steps followed by a causal chain of five scored sub-tasks in one continuous simulation: the state one step leaves (an object pose, a choice, a fact seen once) determines what the next step must do, unlike benchmarks that place sub-tasks side by side. Vision-language agents act through skill-level tools from camera images. A scripted robot that makes only the calls a model gets is the solvability check of a chore. From each trajectory the benchmark reads where a chain breaks, what the agent did after a failed call, and how far its own position estimate drifts from the true one. This version reports first results only: 15 of 60 planned runs (three models at high effort, one run each on chores T1 and T10 from both start states and T3 from one), with scripted-robot proofs on three of the ten chores. Of the 15 runs, 2 passed all five sub-tasks. Runs still going after sub-tasks 1 to 5 were 15, 8, 4, 2, 2. Of 189 failed calls, 168 were followed by a different call and 111 by a call that did not fail. These are first results and not a full benchmark. We will release the scenes, task chains, scripted robot, evaluation harness and trajectories.
BibTeX
@misc{jena2026hospitalbench,
  author = {Gaytri Jena and Kapil Wanaskar and Aman Chadha and Vasu Sharma and Vinija Jain and Amitava Das},
  title = {{HospitalBench: Diagnosing Where and Why Embodied Agents Fail on Long Hospital Task Chains}},
  year = {2026},
  eprint = {2611.19274},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19274},
}
Figure