Abstract
Using a risk-stratification model to guide clinical practice often requires the choice of a cutoff—called the decision threshold—on the model’s output to trigger a subsequent action such as an electronic alert. Choosing this cutoff is not always straightforward. We propose a flexible approach that leverages the collective information in treatment decisions made in real life to learn reference decision thresholds from physician practice. Using the example of prescribing a statin for primary prevention of cardiovascular disease based on 10-year risk calculated by the 2013 Pooled Cohort Equations, we demonstrate the feasibility of using real-world data to learn the implicit decision threshold that reflects existing physician behavior. Learning a decision threshold in this manner allows for evaluation of a proposed operating point against the threshold reflective of the community standard of care. Furthermore, this approach can be used to monitor and audit model-guided clinical decision-making following model deployment.
Introduction
Risk-stratification models have become a cornerstone of modern medicine, providing clinicians with quantitative estimates of patient risk—such as the probability of a cardiovascular event within the next decade. However, a significant gap remains between generating a probability and taking a clinical action. To bridge this gap, developers must set a decision threshold: a specific cutoff score that triggers an intervention, such as an electronic health record (EHR) alert or a medication recommendation.
Traditionally, these thresholds are determined through Medical Decision Analysis, which requires assigning precise "utility values" to every possible outcome (e.g., the numeric cost of a false positive versus the benefit of a true positive). In the complex environment of clinical practice, these values are notoriously difficult to quantify. Alternatively, thresholds are often set using hypothetical clinical vignettes, which may not accurately reflect how physicians behave when faced with real patients. This paper proposes a third way: learning "implicit" decision thresholds by analyzing the aggregate behavior of clinicians as recorded in EHR data.
Methodology: Reverse-Engineering Clinical Wisdom
The researchers developed a mathematical framework to identify the thresholds already embedded in clinical practice. By treating the aggregate decisions of many clinicians as a "wisdom of the crowd," the framework maps calculated risk scores to actual treatment outcomes.
The Mathematical Framework
The core of the approach uses a sigmoid function to link a patient’s risk score (the independent variable) to the probability of receiving treatment (the dependent variable). The researchers tested two versions of the "net utility" function within this sigmoid:
- Linear Model: Assumed a direct, linear relationship between risk and treatment probability.
- Logarithmic Model: Assumed that clinicians’ sensitivity to risk changes as the absolute risk increases. This transformation reflects the psychological reality that a 5% increase in risk may be perceived differently at low risk levels than at very high risk levels.
Defining Thresholds
From this mapping, the researchers derived two key reference points:
- The Popular Vote: The risk score where the probability of treatment matches the overall average treatment rate of the population.
- The Majority Vote: The "tipping point" where a patient has a greater than 50% chance of being treated by a clinician.
Empirical Evaluation: Cardiovascular Risk
To validate this approach, the study utilized real-world data from the Stanford Medicine Research Repository (STARR), focusing on the primary prevention of cardiovascular disease (CVD). The study analyzed two cohorts:
- Pre-2013 Cohort: 4,705 patients (2009–2013).
- Post-2013 Cohort: 23,291 patients seen after the release of the 2013 Pooled Cohort Equations (PCE) guidelines.
The 2013 PCE is a standard tool for predicting 10-year atherosclerotic cardiovascular disease risk. By applying the framework to these cohorts, the researchers could see how well the learned thresholds aligned with official medical guidelines.
Key Results and Insights
Model Performance and Risk Sensitivity
The logarithmic model significantly outperformed the linear model, achieving a better fit (Brier score of 0.159 vs. 0.165). This suggests that clinicians do not view risk increases linearly; they are more sensitive to changes at lower risk levels when deciding whether to initiate therapy, a finding that aligns with established risk-tolerance studies.
Alignment with Guidelines
The learned thresholds showed a remarkable alignment with expert-derived guidelines. For the pre-2013 group, the "popular vote" threshold was identified at a 3.6% risk, while the "majority vote" was 23.0%. These figures correspond closely to the 2013 guidelines, which suggest treatment for "intermediate" risk starting at 7.5% and "high" risk at 20%. The "majority vote" effectively captured the high-risk category where clinical consensus on the benefits of treatment is strongest.
Stability Over Time
One of the most striking findings was the stability of these thresholds. Despite the publication of new official guidelines in 2013, the learned thresholds remained consistent. This suggests that the "community standard of care" is resilient and evolves more gradually than the literature might suggest.
Discussion and Future Applications
This framework offers a robust method for "shadow-testing" new AI models before deployment. By ensuring that a model's triggers align with existing clinical standards, developers can increase the likelihood of clinician adoption and reduce alert fatigue.
Furthermore, this method provides a powerful tool for bias auditing. By comparing learned thresholds across different patient demographics (e.g., race or gender), researchers can identify if certain groups are being held to different standards of care despite having identical risk scores.
While the study is limited by the inherent quality of EHR data—which may not always capture patient refusals or specific contraindications—it demonstrates that aggregate clinician behavior is a viable and valuable proxy for determining the utility of medical interventions. Future work may extend this to evaluate individual physician patterns, providing a path toward personalized feedback and the identification of clinical outliers.
Summary updated