TL;DR
Standard medical AI benchmarks evaluate model proposals in isolation, ignoring the fact that clinical workflows frequently override AI decisions due to local constraints like hospital capacity or safety policies. This paper introduces AuthEval, a framework that logs and evaluates both AI proposals and final clinical actions under the same local criteria. Their findings reveal that ignoring this "authority gap" can lead to highly deceptive evaluation metrics, even reversing whether an AI assistant appears beneficial or detrimental to a workflow.
What Problem Does This Paper Address?
When a medical Vision-Language Model (VLM) assesses a skin lesion image, it might propose an urgency level for specialist review. However, in a real clinic, the model does not have the final say. A local service or clinician acts as a gatekeeper, accepting or replacing the AI's proposal based on factors the model cannot see—such as current hospital capacity, safety protocols, or localized patient context.
Because of this, evaluating the model's proposal quality is fundamentally different from evaluating the final workflow action. Traditional benchmarks only look at the model's raw output. This paper addresses the challenge of measuring the authority gap—the difference between the score of the model's proposal and the score of the action actually taken by the workflow—and demonstrates how failing to account for local constraints invalidates workflow-level claims.
Methods
The authors formalize the human-AI clinical workflow and introduce AuthEval, a logging protocol and evaluation framework. AuthEval treats the model proposal ($\hat{a}$) and the final selected action ($a_{\text{sel}}$) as distinct objects.
Crucially, AuthEval factorizes the authority gap ($\Delta_{\text{authority}}$) into a simple, interpretable product: $$\Delta_{\text{authority}} = p \cdot \delta$$ where:
- $p$ is the proposal-change rate (how often the local review overrides the AI).
- $\delta$ is the mean score change on those modified cases.
To test this framework in a controlled setting, the authors simulate a local review process ("gate") using a metadata proxy to represent decision-time risk context. They evaluate three distinct gate constraint regimes:
- Capacity: Capping how many high-urgency escalations the system can handle.
- Safety: Imposing a strict safety floor to prevent under-triage.
- Mixed: Applying both capacity and safety constraints simultaneously.
Datasets
The framework was validated using the ISIC 2019 dermoscopy training set (comprising MSK, HAM10000, and BCN20000 sources). Diagnoses were mapped to four study-defined risk levels (very low to high) and four corresponding routing actions (observation to urgent specialist assessment).
To prevent data leakage, the authors enforced a strict lesion-aware split (17,629 train / 3,735 validation / 3,967 test images), ensuring that multiple images of the same lesion or perceptual duplicates did not cross between the training and testing sets.
Key Insights and Ideas
- Override frequency is a deceptive metric: A high override rate ($p$) does not inherently tell you if a workflow is improving or degrading. Two different clinical environments can have identical change rates but completely opposite authority gaps depending on whether the overrides shift actions toward safety floors or capacity ceilings.
- The Evaluation Unit Illusion: How you define a "case" drastically changes your conclusions. Weighting every image equally showed an optimistic authority gap ($+0.374$), implying the model's raw proposals scored better than the final workflow actions. However, weighting every lesion cluster equally flipped the gap to negative ($-0.206$).
- Why the flip happens: High-risk lesions are photographed more frequently than low-risk ones. Image-equal weighting accidentally over-emphasizes repeated high-risk lesions—precisely where capacity constraints force downward substitutions, making the model's unconstrained proposals look artificially superior.
Key Results
The authors evaluated MedGemma 1.5 4B as the primary VLM assistant. Under the mixed constraint simulation, the local review changed the model's proposal 47.0% of the time.
- Under Capacity Constraints: The image-equal authority gap was $+0.744$. Proposal-only scoring is highly optimistic here because it completely ignores the reality that a maxed-out clinic will be forced to downgrade triage levels.
- Under Safety Constraints: The gap shifted to $-0.051$ (image-equal) and $-0.224$ (cluster-equal), capturing the workflow's defensive adjustments toward higher-urgency actions.
Limitations and Future Work
- Simulated Local Review: Because the ISIC dataset lacks real-world clinical deployment metadata (like actual hospital capacity or clinician override logs), the local review gate and patient contexts had to be simulated.
- Static Utility Scores: The evaluation relies on a study-defined static scoring rule rather than tracking actual, long-term patient clinical outcomes.
- Single Modality Focus: The study is limited to a single dermoscopic imaging task. Future work is required to see how these authority gaps behave in more complex, multi-modal clinical environments like radiology or emergency triaging.
Summary updated