BABA-Bench: Evaluating Long-Horizon Patient-Advocacy Agents

Shriram Vasudevan, Videet Mehta, Connor SweeneyarXiv 2026 · arXiv

TL;DR

BABA-Bench is a new benchmark designed to evaluate LLM agents in the complex, high-stakes domain of healthcare patient advocacy. Unlike benchmarks that only grade final outcomes, BABA-Bench uses a deterministic engine to enforce "strict passes," ensuring agents not only reach the goal (e.g., getting a patient a walker) but also follow mandatory procedures like obtaining consent and verifying evidence. The study reveals a significant "procedural gap": agents frequently succeed at the task while dangerously violating patient privacy or administrative rules.

What Problem Does This Paper Address?

Healthcare administration in the US is a $265 billion labyrinth of prior authorizations, insurance denials, and fragmented coordination. While LLMs show promise in clinical tasks like diagnosis, they struggle with the "long-horizon" coordination required for advocacy—finding hidden information across insurers, clinics, and pharmacies while navigating deadlines and strict privacy laws.

The authors identify a critical failure mode in current agent evaluations: silent failure. An agent might successfully order a medical device but do so by sharing sensitive records without the patient's permission. In healthcare, the route taken is just as important as the destination. Existing benchmarks often rely on LLM judges or final-state checks, which fail to catch these mid-process procedural violations.

Methods

The researchers developed a pipeline to convert real-world advocacy cases into 47 interactive, simulated "worlds."

Key components of the architecture include:

  • The Deterministic Engine: Instead of an LLM judge, a hard-coded engine tracks the world state. It checks every action in real-time against four criteria: reaching the outcome, having explicit permission (authority), possessing confirmed evidence (grounding), and avoiding prohibited actions.
  • Translator Architecture: To prevent the simulation from being "hallucinated" by an LLM, the authors used a translator (GPT-5.6 Luna) to map an agent’s natural language requests to a structured menu of possible operations within the deterministic engine.
  • Friction: The worlds include realistic hurdles like "phone tag" (clinics not answering), deadlines, and decoy parties to test agent resilience.
  • Strict vs. Outcome Grading: The benchmark distinguishes between Outcome Completion (did you get the walker?) and a Strict Pass (did you get the walker without breaking any rules?).
The BABA-Bench pipeline, from real cases to deterministic grading.

Datasets

The authors introduced BABA-Bench, consisting of 47 interactive worlds built from de-identified, real-patient advocacy cases reviewed by experts. These worlds are categorized into difficulty bands (Light, Standard, Hard) based on structural complexity, such as the number of required permissions and "hidden" parties the agent must discover.

Key Insights and Ideas

  • The Procedural Gap: Success is often a mirage. Of the runs that reached the correct outcome, 28% failed the strict pass due to procedural violations.
  • Consent is the Primary Casualty: The most common failure (82% of permission incidents) was the exchange of patient information without explicit consent. Agents often "know" the rules (having read the provided playbooks) but fail to apply them in practice.
  • Planning is Harder than Execution: In a diagnostic experiment, when agents were given a "verified route" (a step-by-step safe path), their strict-pass rates soared from ~30-44% to over 90%. This suggests the primary bottleneck for frontier models isn't following instructions, but the high-level planning required to find a safe path through a complex environment.
  • Difficulty Scales Non-Linearly: On "Hard" worlds, nearly half of the successful outcomes were reached via invalid routes. No model achieved a strict pass on the most difficult worlds.
Breakdown of why outcome-complete runs failed to get a strict pass, highlighting permission and evidence errors.

Key Results

The authors evaluated 14 frontier models (including Claude, GPT, and Gemini variants) across 1,974 runs.

  • Claude Opus 5 emerged as the strongest performer in terms of raw capability, reaching the required outcomes in 81.6% of runs. However, its strict-pass rate was significantly lower at 52.5%.
  • GPT-5.6 Terra showed better "procedural integrity," with only a 14% gap between outcome success and strict success, though its overall success rate was lower than Opus 5.
  • Open-weight models like DeepSeek V4 Pro performed competitively, achieving a 48.9% strict-pass rate, outperforming several larger closed-source models.
  • Efficiency: Models varied wildly in cost and speed. Gemini 3.7 Flash was the fastest (46.2 steps/world), while Claude Opus 5 was among the most expensive ($5.46 per world).

Limitations and Future Work

The authors acknowledge several limitations:

  • Sample Size: With only 47 worlds, the confidence intervals are wide, making it difficult to definitively rank the top-tier models.
  • Single Source: The cases come from a single advocacy platform, which may not represent the full diversity of healthcare navigation.
  • No Human Baseline: There is currently no data on how human advocates perform under the same simulated constraints.
  • Translator Dependency: While the engine is deterministic, the "translator" that interprets agent intent is still an LLM, which could introduce its own errors.

Future work aims to expand the corpus to include more payer types and state-specific regulations, and to test whether explicit "permission checks" in the agent's loop can reduce procedural failures.

Summary updated

Abstract

An LLM agent that helps a patient get care must do two things. It must reach the correct outcome, and it must follow the necessary procedure. For example, the agent must get the patient's permission before it shares the patient's records. We introduce BABA-Bench, a benchmark of 47 interactive worlds. We built each world from a de-identified, real patient advocacy case that expert advocates reviewed. In each world, the agent must find hidden information and coordinate insurers, clinics, suppliers, pharmacies, and community organizations. The engine checks each action at the time the agent does it. It checks if the agent has permission, if the agent has evidence for the action, and if the action is prohibited. We evaluated 14 frontier models with three runs on each world, for a total of 1,974 runs. Agents often reached the correct outcome by an invalid route. Of the 921 runs that reached all required outcomes, 254 (28%) did not follow the necessary procedure. Claude Opus 5 reached all required outcomes in 81.6% of its runs, but only 52.5% of its runs passed all checks. When we gave two models a verified route, their pass rates increased from 29.8% to 91.5% and from 44.0% to 97.2%. For these models, the main problem is to find a route, not to execute it.
BibTeX
@misc{vasudevan2026baba,
  author = {Shriram Vasudevan and Videet Mehta and Connor Sweeney},
  title = {{BABA-Bench: Evaluating Long-Horizon Patient-Advocacy Agents}},
  year = {2026},
  eprint = {2610.41076},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.41076},
}
Figure