TL;DR
BABA-Bench is a new benchmark designed to evaluate LLM agents in the complex, high-stakes domain of healthcare patient advocacy. Unlike benchmarks that only grade final outcomes, BABA-Bench uses a deterministic engine to enforce "strict passes," ensuring agents not only reach the goal (e.g., getting a patient a walker) but also follow mandatory procedures like obtaining consent and verifying evidence. The study reveals a significant "procedural gap": agents frequently succeed at the task while dangerously violating patient privacy or administrative rules.
What Problem Does This Paper Address?
Healthcare administration in the US is a $265 billion labyrinth of prior authorizations, insurance denials, and fragmented coordination. While LLMs show promise in clinical tasks like diagnosis, they struggle with the "long-horizon" coordination required for advocacy—finding hidden information across insurers, clinics, and pharmacies while navigating deadlines and strict privacy laws.
The authors identify a critical failure mode in current agent evaluations: silent failure. An agent might successfully order a medical device but do so by sharing sensitive records without the patient's permission. In healthcare, the route taken is just as important as the destination. Existing benchmarks often rely on LLM judges or final-state checks, which fail to catch these mid-process procedural violations.
Methods
The researchers developed a pipeline to convert real-world advocacy cases into 47 interactive, simulated "worlds."
Key components of the architecture include:
- The Deterministic Engine: Instead of an LLM judge, a hard-coded engine tracks the world state. It checks every action in real-time against four criteria: reaching the outcome, having explicit permission (authority), possessing confirmed evidence (grounding), and avoiding prohibited actions.
- Translator Architecture: To prevent the simulation from being "hallucinated" by an LLM, the authors used a translator (GPT-5.6 Luna) to map an agent’s natural language requests to a structured menu of possible operations within the deterministic engine.
- Friction: The worlds include realistic hurdles like "phone tag" (clinics not answering), deadlines, and decoy parties to test agent resilience.
- Strict vs. Outcome Grading: The benchmark distinguishes between Outcome Completion (did you get the walker?) and a Strict Pass (did you get the walker without breaking any rules?).
Datasets
The authors introduced BABA-Bench, consisting of 47 interactive worlds built from de-identified, real-patient advocacy cases reviewed by experts. These worlds are categorized into difficulty bands (Light, Standard, Hard) based on structural complexity, such as the number of required permissions and "hidden" parties the agent must discover.
Key Insights and Ideas
- The Procedural Gap: Success is often a mirage. Of the runs that reached the correct outcome, 28% failed the strict pass due to procedural violations.
- Consent is the Primary Casualty: The most common failure (82% of permission incidents) was the exchange of patient information without explicit consent. Agents often "know" the rules (having read the provided playbooks) but fail to apply them in practice.
- Planning is Harder than Execution: In a diagnostic experiment, when agents were given a "verified route" (a step-by-step safe path), their strict-pass rates soared from ~30-44% to over 90%. This suggests the primary bottleneck for frontier models isn't following instructions, but the high-level planning required to find a safe path through a complex environment.
- Difficulty Scales Non-Linearly: On "Hard" worlds, nearly half of the successful outcomes were reached via invalid routes. No model achieved a strict pass on the most difficult worlds.
Key Results
The authors evaluated 14 frontier models (including Claude, GPT, and Gemini variants) across 1,974 runs.
- Claude Opus 5 emerged as the strongest performer in terms of raw capability, reaching the required outcomes in 81.6% of runs. However, its strict-pass rate was significantly lower at 52.5%.
- GPT-5.6 Terra showed better "procedural integrity," with only a 14% gap between outcome success and strict success, though its overall success rate was lower than Opus 5.
- Open-weight models like DeepSeek V4 Pro performed competitively, achieving a 48.9% strict-pass rate, outperforming several larger closed-source models.
- Efficiency: Models varied wildly in cost and speed. Gemini 3.7 Flash was the fastest (46.2 steps/world), while Claude Opus 5 was among the most expensive ($5.46 per world).
Limitations and Future Work
The authors acknowledge several limitations:
- Sample Size: With only 47 worlds, the confidence intervals are wide, making it difficult to definitively rank the top-tier models.
- Single Source: The cases come from a single advocacy platform, which may not represent the full diversity of healthcare navigation.
- No Human Baseline: There is currently no data on how human advocates perform under the same simulated constraints.
- Translator Dependency: While the engine is deterministic, the "translator" that interprets agent intent is still an LLM, which could introduce its own errors.
Future work aims to expand the corpus to include more payer types and state-specific regulations, and to test whether explicit "permission checks" in the agent's loop can reduce procedural failures.
Summary updated