Abstract
We use machine learning as a tool to study decision making, focusing specifically on how physicians diagnose heart attack. An algorithmic model of a patient's probability of heart attack allows us to identify cases where physician testing decisions deviate from predicted risk. We then use actual health outcomes to evaluate whether those deviations represent mistakes or physicians' superior knowledge. This approach reveals two inefficiencies. Physicians over-test: predictably low-risk patients are tested, but do not benefit. At the same time, physicians undertest: predictably high-risk patients are left untested, and then go on to suffer adverse health events including death. A natural experiment using shift-to-shift testing variation confirms these findings. Simultaneous over-and under-testing cannot easily be explained by incentives alone, and instead point to systematic errors in judgment. We provide suggestive evidence on the psychology underlying these errors. First, physicians use too simple a model of risk. Second, they overweight factors that are salient or representative of heart attack, such as chest pain. We argue health care models must incorporate physician error, and illustrate how policies focused solely on incentive problems can produce large inefficiencies.
Introduction
In the landscape of health economics, "low-value care" is typically attributed to moral hazard. The standard narrative suggests that because patients are insured and doctors are often compensated for volume (or fear malpractice litigation), the medical system defaults to over-testing. The policy prescription has followed suit: raise the threshold for testing to reduce waste.
However, this "less is more" approach rests on a critical assumption: that physicians are experts at ranking patient risk and simply choose a threshold that is too low. In "Diagnosing Physician Error," Sendhil Mullainathan and Ziad Obermeyer challenge this assumption. Using machine learning to audit emergency room decisions, they suggest that the primary driver of inefficiency isn't just an incentive problem, but a prediction problem.
Methodology: Auditing the Expert
To determine if physicians are making systematic errors, the authors developed a machine learning pipeline to predict the "yield" of diagnostic tests for Acute Coronary Syndrome (heart attack).
The Predictor
The researchers utilized an ensemble model (Gradient Boosted Trees and LASSO) trained on Electronic Health Records (EHR) from a major urban hospital. The dataset included over 246,000 emergency visits and 16,000 patient characteristics, ranging from vitals to a two-year medical history. The model's goal was to predict the probability that a test (such as cardiac catheterization) would find a blockage.
Addressing "Private Information"
A common defense of physician judgment is that doctors possess "private information"—nuances like a patient’s pallor or frailty—that algorithms cannot see. To test this, the authors looked at the outcomes for the untested. If a physician’s decision to skip a test was based on superior private information, those untested patients should remain healthy. If the algorithm was correct and the physician was wrong, those high-risk untested patients would suffer adverse events.
Key Findings: The Dual Inefficiency
The study reveals that the medical system suffers from a "misallocation" of resources rather than a simple surplus of care.
1. Over-testing the Low-Risk
The audit confirmed that many tests are indeed low-value. Approximately 62% of tests performed were found to cost more than $150,000 per life-year saved. These patients were predictably low-risk, yet they underwent invasive and expensive procedures.
2. Under-testing the High-Risk
More strikingly, the researchers found a massive "under-testing" problem. The highest-risk decile of patients who were not tested by physicians had a 15.6% rate of Major Adverse Cardiac Events (MACE), including a 3.3% death rate, within 30 days. This is nearly eight times higher than the 2% clinical threshold at which testing is standardly recommended.
3. The Natural Experiment
To validate these findings, the authors exploited the quasi-random assignment of patients to ER shifts. They found that high-risk patients who happened to arrive during "high-testing" shifts (shifts with a higher propensity to test) saw a 32% reduction in one-year mortality compared to similar patients in low-testing shifts. This confirms that for the high-risk group, more testing is not "waste"—it is life-saving.
The Psychology of Error
Why do highly trained physicians miss these high-risk cases? The authors point to two cognitive factors:
- Bounded Rationality: Physicians appear to use a simplified mental model. While the ML model utilized 224 variables to achieve peak accuracy, the model that best predicted physician behavior used only 49. Doctors ignore the complex, high-dimensional history available in the EHR in favor of a few "classic" indicators.
- Salience Bias: Doctors over-weight "representative" symptoms. For example, a patient presenting with chest pain was 578% more likely to be tested than a patient with the same algorithmic risk but different symptoms (like shortness of breath or nausea).
Policy Implications
The findings suggest that current policies aimed at reducing "low-value care" may be dangerously blunt instruments. If a policy simply incentivizes doctors to "test less," doctors—who are already struggling to identify risk—may cut testing for the very high-risk patients who need it most.
Instead, the authors propose precision pricing (tying reimbursement to the predicted risk of the patient) and algorithmic decision support to help physicians identify high-risk patients who do not fit the "classic" profile of a heart attack.
Conclusion
By reframing medical inefficiency as a prediction error rather than just an incentive problem, Mullainathan and Obermeyer provide a new path for healthcare reform. Improving the accuracy of physician audits through machine learning doesn't just save money; it saves lives by ensuring that care is directed to those who will actually benefit from it.
Summary updated