TL;DR
This paper explores why AI decision-support tools often fail to transition from the lab to the clinic, identifying a critical "sociotechnical gap" where algorithmic outputs clash with a physician's legal and ethical responsibilities. By interviewing 19 U.S. physicians using scenario-based design, the researchers demonstrate that for AI to be truly "actionable," it must provide specific contextual information—such as societal guidelines and institutional policies—that allows doctors to navigate legal liability and maintain their "ethics of care."
What Problem Does This Paper Address?
Despite the proliferation of high-accuracy AI models for medical diagnosis and drug recommendation, clinical adoption remains low. The authors argue that AI developers often overlook the "care ecologies" that physicians inhabit—complex networks of patients, colleagues, hospital policies, and, crucially, legal infrastructures.
Physicians operate under a legally defined "duty to care." When an AI tool provides a recommendation, the legal burden of a bad outcome doesn't fall on the algorithm or its developers; as one participant put it, "It’s still 100% on me." Current Explainable AI (XAI) focuses on how a model reached a conclusion (e.g., feature importance), but it fails to address the legal ambiguity that prevents a doctor from acting on that conclusion. The paper seeks to identify what information physicians actually need to bridge this gap.
Methods
The researchers employed Scenario-Based Design (SBD), a qualitative method that immerses participants in a familiar task to elicit nuanced needs.
- Participants: 19 U.S.-based physicians ranging from first-year residents to seasoned clinicians.
- The Scenario: Participants acted as an attending physician at a large hospital using a mock AI-based drug recommendation tool integrated into an Electronic Health Record (EHR) interface.
- The Probe: A non-interactive visual mock-up of an EHR that displayed various types of AI explanations and buttons (e.g., "Request Audit," "View Guidelines").
- Iterative Design: The study was conducted in two rounds, with the visual probe updated between rounds based on initial feedback to better reflect the information physicians requested.
Datasets
As a qualitative human-computer interaction (HCI) study, the paper does not use traditional machine learning training datasets. Instead, the "data" consists of:
- Audio/Video Recordings: Transcribed and anonymized from 19 interviews (averaging 58–72 minutes each).
- Thematic Analysis: The authors used an iterative coding approach based on the "Actionability Framework" (Mansi et al. 2026) to categorize concerns and information needs.
Key Insights and Ideas
- AI as a "Consultant," Not a Decision-Maker: Physicians view AI as a limited input. They perform a "mental process" of comparing AI outputs against comorbidities, race, and social factors that the AI often misses (e.g., a patient’s lack of transportation to a pharmacy).
- The "Black Box Warning" Analogy: Doctors are conditioned by existing EHR "black box warnings" (FDA safety alerts) where clicking a button constitutes a formal acceptance of legal responsibility. They fear that interacting with AI feedback features might unintentionally create a "legal trail" of liability.
- Collective Care: Responsibility is not just individual. Physicians worry about how their use of the tool affects the "collective" (trainees, colleagues, and the broader medical profession). For instance, they fear AI might cause "deskilling" in residents or create "self-fulfilling prophecies" by reinforcing biased historical data.
- Strategic Non-Transparency: Some physicians expressed a desire to hide AI use from patients to preserve trust, fearing that admitting a computer helped make the decision would make them seem "no better than the computer."
Key Results
The study identified four essential types of information that resolve legal ambiguity and make AI actionable:
- Societal Guidelines & Peer-Reviewed Literature: Doctors need the AI to show how its recommendation aligns (or conflicts) with established standards like AHA guidelines or PubMed studies.
- Institutional Policies: Physicians feel "backed up" when an AI output aligns with their specific hospital’s protocols, which provides a layer of legal protection.
- Patient Education Materials: Tools that help doctors explain the "why" to patients (via diagrams or simplified chunks of info) reduce malpractice risk by building trust and shared decision-making.
- Human Resource Connections: The ability to see the credentials of the person who "audited" the AI or a way to contact a human specialist for validation was deemed more valuable than the algorithmic explanation itself.
Limitations and Future Work
- Geographic Scope: The study is limited to the U.S. healthcare and legal system. Malpractice concerns and "duty to care" definitions vary significantly in other countries.
- Task Specificity: The findings are grounded in drug selection and dosage. Other high-stakes areas like medical imaging or emergency triage might reveal different legal pressures.
- Future Directions: The authors suggest expanding this research to other healthcare professionals (nurses, technicians) and exploring how AI impacts different clinical applications like transcription or clinical trial matching. They also call for AI evaluations that specifically assess a physician's ability to navigate institutional infrastructures, rather than just their trust in the model.
Summary updated