Suppose a hospital has thousands of logged conversations between patients and an older medical chatbot, and wants to know how a new chatbot would have fared on them without putting it in front of patients. This is off-policy evaluation: estimating how well a new decision-making policy would do using only data collected by a different one. The standard tool reweights each logged conversation by how much more or less likely the new policy was to produce it. For a language model, "produce it" means every token of every reply. The reweighting factor is therefore a product of hundreds or thousands of per-token probability ratios, and it is almost always either vanishingly small or astronomically large. This exponential blow-up with sequence length is the curse of horizon. At the token level it is not a nuisance. It makes the estimate useless.
We propose to measure the conversation at the level where the clinical evaluation metric already operates. Modern medical dialogue metrics break a conversation into atomic clinical facts and checklist items, such as "asked about chest pain onset" or "checked for anticoagulant use" (Automated Rubrics for Reliable Evaluation of Medical Dialogue Systems; MedQA-CS). We treat the set of rubric items satisfied so far as the state of the encounter, and the clinical intent of each turn (ask about X, recommend Y, close) as the action. Off-policy reweighting then runs over a few dozen clinical intents per turn instead of a vocabulary per token. The central question is how much variance this removes, and how much bias it introduces when the patient's reply depends on phrasing rather than intent.
Core Claim
The claim is that the domain-specific evaluation metric supplies the right state abstraction for off-policy evaluation of clinical dialogue agents. If the reward depends only on which rubric items are satisfied, and the patient's next reply depends on the agent's turn mainly through its clinical intent, then importance weights and visitation ratios computed over rubric states and intents are nearly unbiased. They are also many orders of magnitude less variable than token-level weights. Concretely, with $c_t$ the rubric state at turn $t$ (the set of satisfied checklist items), $z_t$ the intent class of the agent's turn, $\pi$ the new policy and $\beta$ the logging policy, the abstracted trajectory weight is
$$w = \prod_{t=1}^{T} \frac{\hat\pi(z_t \mid c_t)}{\hat\beta(z_t \mid c_t)},$$
where $T$ is the number of turns, not tokens. We estimate $\hat\pi(z \mid c)$ by sampling the target model several times at each logged conversation prefix and classifying the sampled replies into intents. This is possible because the target model is available to query, even though it cannot be deployed on patients. When the behavior policy is unknown, as with real logs, the same abstraction serves as the state space for behavior-agnostic stationary-ratio estimation.
The contribution does not depend on any one estimator. What is new is the metric-induced abstraction, together with a measured bias–variance curve as the abstraction is made coarser or finer. The closest prior work is Concept-driven Off Policy Evaluation. It bins ICU vital signs into clinical severity concepts and reweights actions within those bins, which reduced estimator variance by one to two orders of magnitude on MIMIC-III. That work motivates ours, but its concepts are hand-chosen summaries of numeric patient state and its actions are already discrete treatments. In our setting the actions are free text, the concepts are derived from the evaluation rubric itself, and the abstraction is applied to both states and actions. Aligning LLMs Toward Multi-Turn Conversational Outcomes Using Iterative PPO also works at turn level rather than token level. It compresses future conversation into a learned terminal value model, but it learns from on-policy rollouts with business outcomes such as suggestion acceptance, and it does not evaluate a new policy from someone else's logs.
Research Program
The first component is the abstraction itself. For each case in MedQA-CS, which provides 44 USMLE Step 2 Clinical Skills cases with case-specific checklists spanning information gathering, physical exam, closure, and differential diagnosis, a turn-level judge marks which checklist items each agent turn satisfies. The judge decomposes each turn into atomic facts in the style of Chen et al., whose rubric auditing reached a near-miss AUROC of 0.977 against a GPT-4o judge. Intents come from a small fixed taxonomy per encounter phase. We will study three abstraction granularities. The coarse version uses the phase and the count of satisfied items. The middle version uses the exact satisfied set. The fine version uses the satisfied set plus phrasing features such as question type and inclusion of lay explanation. Moving from coarse to fine traces the path from low variance and high bias toward the token-level estimator.
The second component is a set of estimators placed on top of each abstraction, so that abstraction and estimator effects can be separated. We will run abstracted importance sampling and per-decision importance sampling, fitted Q-evaluation over rubric states, a doubly robust combination, and a DualDICE-style stationary ratio estimator (DualDICE). DualDICE is the behavior-agnostic option. It learns the ratio of state-action visitation frequencies through a minimax objective, and the original paper tested it only on Taxi, Cartpole, and Reacher. Our finite, structured rubric states are a natural place to test whether its saddle-point instability, a known weakness, becomes manageable. For the doubly robust variant we reuse the orthogonal-loss idea of the DRQ-learner (An Orthogonal Learner for Individualized Outcomes in Markov Decision Processes). That estimator is built so that first-order errors in the auxiliary estimates (behavior policy, visitation ratio, baseline value) do not propagate to the value estimate, but it has been validated only on tabular Taxi and Frozen Lake.
The third component is an analysis of when the abstraction is sufficient. Bias enters exactly when two replies with the same intent lead the simulated patient to disclose different information. Evaluating large language models as agents in the clinic reports that phrasing strongly shifts diagnostic trajectories, which makes this a live concern and not a technicality. We will measure it directly. At logged prefixes, we sample many same-intent replies, observe the spread in next-state distributions, and relate that spread to the estimator bias observed at each granularity.
Baselines
The naive reference point is token-level trajectory importance sampling, which uses the exact log-probabilities of both models over every generated token. We expect its effective sample size to collapse to near one. We will report effective sample size and error as the turn budget grows, to show the curse of horizon empirically in this setting. Beating it is a low bar, but it documents the problem.
The second baseline is one-step replay scoring. At each logged prefix, the target model generates only the next turn, which is scored with the same rubric judge, and the scores are averaged. This is what current single-turn clinical metrics effectively do. It uses the same data, the same judge, and the same target-model sampling budget as our method. Beating it means higher rank correlation with ground-truth policy values, specifically on pairs of policies that differ in information-gathering strategy rather than per-turn quality.
The strongest existing competitor is fitted Q-evaluation on dense embeddings of the dialogue history. It regresses future rubric reward on an LLM embedding of the conversation so far, with no hand-built abstraction, and is the standard deep off-policy evaluation baseline that the DRQ paper compares against. It gets the same logs, the same reward labels, and a matched hyperparameter search budget, tuned on a held-out set of target policies whose ground truth is not reused for the final comparison. Beating it means lower normalized absolute error and higher Spearman correlation across target policies at equal or smaller data size.
The key ablation is the same estimator with random, non-rubric clusters. Conversation states and turns are clustered in embedding space into the same number of cells as the rubric abstraction. If random clusters do as well, the benefit comes from coarsening in general, not from the clinical metric, and the core claim fails.
For reported numbers, no result was found on Lacuna for off-policy evaluation of multi-turn clinical dialogue policies on any benchmark. The only comparable figure is the "one to two orders of magnitude" variance reduction from concept abstraction on MIMIC-III treatment data reported in Concept-driven Off Policy Evaluation. We treat it as the order of effect to expect, not as a number on our task.
Why not just…?
Why not just roll out each new agent against an LLM patient simulator? In our main testbed we do exactly this, because the simulator serves as the ground-truth environment. The alternative is therefore not being dismissed. It is being used to validate our estimator. The case for logs is that real deployment asks about real patients, and simulated patients distort behavior under phrasing changes (Evaluating large language models as agents in the clinic). In addition, fully synthetic loops erode pathological variability (AI-generated data contamination erodes pathological variability and diagnostic reliability). This is a real threat to the usefulness of the method in the simulator setting, and we will say plainly that the method's value depends on logs with real patient responses.
Why not just use fitted Q-evaluation on rubric states and skip importance weighting? This may work as well, and the design is built to find out. Every estimator is run on every abstraction. If regression on rubric states matches the doubly robust and stationary-ratio estimators, the finding is that the metric-derived state is what matters, and the simplest estimator should be recommended. The DRQ results suggest regression degrades under low overlap. We test this by including target policies whose intent choices are rare under the logging policy.
Why not just sum per-turn rubric scores on replayed turns? This ignores that a new agent changes what the patient says next. An agent that asks the right early question uncovers information that later turns depend on. The replay baseline measures this failure directly. Our prediction is that it ranks correctly among policies that differ only in phrasing quality and misranks policies that differ in questioning order.
Proposed Experiments
The main testbed is semi-synthetic, with MedQA-CS cases as the environment. A fixed LLM patient simulator is conditioned on each case's hidden findings. The behavior data comes from a mixture of three prompted open-weight models, with a few thousand logged encounters. The targets are twelve policies that vary the model, a thoroughness prompt against a diagnose-early prompt, and temperature. Each target's true value is its mean checklist-plus-diagnosis score over at least 500 on-policy rollouts in the same simulator. Metrics are normalized absolute error, Spearman correlation across the twelve targets, regret of the top-ranked policy, effective sample size, and interval coverage when CoinDICE-style empirical-likelihood intervals are added (CoinDICE).
Three checks decide the claim. The first is a horizon sweep, truncating encounters at 5, 10, 20, and 30 turns. Token-level error should grow roughly exponentially while rubric-level error stays nearly flat, since the weight product now has $T$ factors over a small intent space. The second is the granularity curve. It should show a clear interior minimum in error, and the location of that minimum should track the measured same-intent next-state spread. The third is the random-cluster ablation, which must lose to the rubric abstraction at equal cell count.
A separate stress test measures judge noise. We inject controlled rates of false positives and false negatives into item detection and measure how value error grows.
Finally, on MedDialog-EN, which has no ground-truth values, we will test only coverage: what fraction of the rubric states and intents visited by target policies appear in real logs. A general medical rubric from HealthBench stands in for case checklists. This check tells us whether real conversation logs are rich enough to support the method outside simulation. We state in advance that this is a feasibility result, not an accuracy result.
References
- Automated Rubrics for Reliable Evaluation of Medical Dialogue Systems
- MedQA-CS: Benchmarking Large Language Models Clinical Skills Using an AI-SCE Framework
- Concept-driven Off Policy Evaluation
- Aligning LLMs Toward Multi-Turn Conversational Outcomes Using Iterative PPO
- DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections
- An Orthogonal Learner for Individualized Outcomes in Markov Decision Processes
- Evaluating large language models as agents in the clinic
- AI-generated data contamination erodes pathological variability and diagnostic reliability
- CoinDICE: Off-Policy Confidence Interval Estimation