TL;DR
HospitalBench is a new diagnostic benchmark designed to uncover exactly where and why embodied agents fail during long-horizon hospital chores. By using "causal chains"—where the physical state or information left by one sub-task dictates the success of the next—it reveals how small errors in positioning or memory compound into total task failure.
What Problem Does This Paper Address?
Current benchmarks for embodied AI (robots driven by Large Language Models or Vision-Language Models) often focus on "end-task success." If a robot fails to set up a surgical tray, we know it failed, but we don't necessarily know why. Did it miss a grasp? Did it forget which tool the surgeon asked for five minutes ago? Or did it park its trolley three centimeters too far to the left, making the next reach impossible?
The authors argue that long-horizon tasks fail at the "seams" between skills. Most existing benchmarks place sub-tasks side-by-side in a "clean" state. HospitalBench instead forces agents to operate in a continuous simulation where errors propagate. This "link dependence" is critical for diagnosing the robustness of agents intended for complex, real-world environments like hospitals.
Methods
The researchers developed HospitalBench within NVIDIA Isaac Sim. The core framework consists of:
- Causal Task Chains: Each of the 10 chores consists of five scored sub-tasks. Crucially, the "pass band" of one task is wider than the "re-grip band" of the next, meaning the robot must perceive and adapt to the specific result of its previous action.
- Link Kinds: The paper defines four types of dependencies (links) between sub-tasks:
- V (Vehicle/Cart pose): The position of a mobile base affects subsequent reaches.
- C (Choice): A decision made in step 1 (e.g., which bay to attend) sets the target for step 2.
- P (Placement): Where an object was placed determines where it must be picked up.
- F (Fact seen once): A piece of information (like a "KEEP" card) is shown only once and must be remembered for a later step.
- Skill-Level API: Agents interact via high-level calls like
pick,place,drive_to, andopen_drawer. - Scripted Robot Proof: To ensure tasks are actually solvable, the authors wrote "scripted" programs that use the same API as the AI models. If the script can't pass, the task is poorly defined.
Datasets
The paper introduces the HospitalBench benchmark:
- 10 Hospital Chores: Ranging from instrument counting to biohazard bagging and infusion alarm rounds.
- Environment: Six distinct hospital areas built with 74 unique NVIDIA SimReady assets.
- Comparison: The authors compared HospitalBench against 17 existing benchmarks (e.g., CALVIN, BEHAVIOR-1K, LIBERO), highlighting that HospitalBench is unique in its combination of continuous chaining, "fact-based" memory links, and explicit "link testing."
Key Insights and Ideas
- The Memory Bottleneck: One of the most striking findings was the failure of "F" (Fact) links. In task T1, agents were shown a card indicating which forceps to keep. Many models failed the subsequent sub-task because they either forgot the instruction or couldn't map the visual "fact" to the physical action of picking the other forceps.
- Compounding Physical Errors: Small drifts in the robot's own odometry (position estimation) led to "tool-use breakdowns." If the robot thinks it is 5cm away from where it actually is, its
pickcalls will consistently fail, even if its high-level reasoning is perfect. - Recovery is Hard: While the models often tried a different call after a failure (168 out of 189 failed calls were followed by a different attempt), they rarely successfully "unstuck" themselves to complete the entire chain.
- The "Seam" Diagnosis: By tracking exactly which link (V, C, P, or F) caused the break, the benchmark provides a "passed-in-a-row" score that is far more informative than a binary success/fail metric.
Key Results
The paper presents preliminary results from 15 "walks" (full task attempts) using models like Claude 3.5 Opus (aliased as Opus 5.5) and GPT-4o (aliased as GPT-6.1):
- Low Success Rate: Only 2 out of 15 runs passed all five sub-tasks. Both successes were on the same task (T1).
- Early Attrition: On average, models only passed 1.6 to 2.4 sub-tasks before failing.
- State Drift: The median drift between the robot's estimated position and its true position was 6.7 cm, which is enough to cause failures in precision tasks like grasping surgical tools.
- Failed Calls: 19% of all API calls failed. Of those, 28% were "repeated" calls where the model simply tried the exact same failing action again.
Limitations and Future Work
- Scale: This version reports on only 15 of 60 planned runs and covers only 3 of the 10 chores with model data.
- Model Diversity: No open-weight models (like Llama) or specialized Vision-Language-Action (VLA) models have been tested yet.
- Link Testing: The rigorous "Link Test" (systematically moving objects to see if the robot can still find them) has only been fully executed on 1 of the 40 links.
- Sim-to-Real: As with most simulation benchmarks, there is an unaddressed gap between the Isaac Sim environment and physical hospital hardware.
The authors plan to release the full suite of scenes, task chains, and the evaluation harness to the research community to encourage more diagnostic-heavy embodied AI research.
Summary updated