TL;DR
This study investigates the "Equal Z" (EZ) standard-setting method for medical school exams, finding it to be a valid and efficient way to determine passing scores. The key takeaway is that a panel of just 10 experts is sufficient to achieve reliable cut scores, making the EZ method a practical solution, especially when expert availability is limited.
What Problem Does This Paper Address?
Setting reliable "cut scores" for high-stakes examinations, particularly in medical education, is crucial for accurately distinguishing competent from incompetent individuals. Traditional methods like the Angoff and Hofstee methods, while widely used, have faced criticism for being resource-intensive, time-consuming, and potentially susceptible to expert judgment biases. The "Equal Z" (EZ) method, a more recent development, aims to address these limitations by providing a valid and effective alternative. However, a critical unanswered question was the minimum number of experts required for the EZ method to yield acceptable and reliable cut scores. This paper tackles this gap by simulating different panel sizes to determine this optimal number.
Methods
The study employs a simulation-based approach using primary data collected from a standard-setting exercise for mock Objective Structured Clinical Examinations (OSCEs) at Taipei Veterans General Hospital. The core of the EZ method involves each expert independently identifying the "highest failing mark" (L) and the "lowest passing mark" (H) for each station. These individual judgments are then used to calculate the mean L (X_L) and mean H (X_H), along with their respective standard errors (SE_L and SE_H).
The novel aspect of the EZ method lies in its calculation of a standardized Z-score that represents a point equidistant from X_L and X_H in terms of their standard errors. This is achieved through the equation: $Z = (X_H - X_L) / (SE_L + SE_H)$. The cut score is then determined as $X_L + Z * SE_L$ (or equivalently, $X_H - Z * SE_H$).
For this simulation study, the researchers generated 1,000 random samples of panel sizes ranging from 5 to 25 experts from the original dataset of 31 panelists. Analysis of Variance (ANOVA) was used to assess whether different panel sizes led to statistically significant differences in the cut scores across all stations.
Datasets
The primary data for this study were collected from a standard-setting exercise involving 31 senior clinicians who assessed 12 stations of a mock OSCE. This dataset, comprising the highest failure marks and lowest pass marks for each station from each expert, was used to generate simulated data for the study. The simulated dataset, containing 1,000 sub-samples of varying panel sizes, is available from Harvard Dataverse.
Key Insights and Ideas
- Efficiency is Key: The EZ method offers a significant time advantage. A panel of 12 experts can assess 12 OSCE stations in under an hour, a stark contrast to the hours often required by traditional methods.
- Simplicity of Calculation: The statistical skills required for the EZ method are basic, making it accessible and less demanding on the expertise of the facilitators.
- Built-in Reliability: The EZ method inherently incorporates a reliability measure through the Z-score calculation, which is influenced by the variance of expert judgments (SE_L and SE_H) and the range of perceived borderline scores (X_H - X_L).
- Expertise vs. Numbers: The study suggests that while expert judgment is crucial, the number of experts doesn't necessarily improve cut score reliability beyond a certain point. Diversity of expertise might be more impactful than sheer panel size.
Key Results
- A panel of 10 or more experts yielded cut scores with a 1-sided confidence of 90% or higher (Z-score ≥ 1.64).
- A panel of 15 experts achieved a 1-sided confidence of 95% or higher (Z-score ≥ 1.96).
- Crucially, ANOVA revealed no statistically significant differences in cut scores across panel sizes ranging from 5 to 25 experts (P ≥ 0.243). This indicates that once a minimum threshold is met, increasing panel size doesn't substantially alter the cut score.
Limitations and Future Work
- Resampling Size: The study acknowledges that the number of re-samples per station per sample size (19-51) might be considered moderate. A larger number of panels could potentially lead to higher significance levels, making the current findings even more robust.
- Holistic Assessment: The EZ method assesses stations at a whole-station level rather than individual items. While this is a limitation for granular analysis, it aligns with the holistic approach of the Hofstee method and has been shown to yield comparable results to other methods.
- Generalizability: The authors suggest that further research is needed to assess the generalizability and utility of the EZ method across different examination types, contexts, and populations.
The study strongly advocates for the EZ method as a practical, valid, and reliable tool for setting cut scores in high-stakes examinations, particularly in resource-constrained environments.
Summary updated