Impact of prevalence and case distribution in lab-based diagnostic imaging studies

Brandon D. Gallas, Weijie Chen, Elodia B. Cole, Robert Ochs, Nicholas Petrick, Etta D. Pisano, Berkman Sahiner, F. W. Samuelson, Kyle J. MyersJournal of Medical Imaging 2019 · DOI

Abstract

We investigated effects of prevalence and case distribution on radiologist diagnostic performance as measured by area under the receiver operating characteristic curve (AUC) and sensitivity-specificity in lab-based reader studies evaluating imaging devices. Our retrospective reader studies compared full-field digital mammography (FFDM) to screen-film mammography (SFM) for women with dense breasts. Mammograms were acquired from the prospective Digital Mammographic Imaging Screening Trial. We performed five reader studies that differed in terms of cancer prevalence and the distribution of noncancers. Twenty radiologists participated in each reader study. Using split-plot study designs, we collected recall decisions and multilevel scores from the radiologists for calculating sensitivity, specificity, and AUC. Differences in reader-averaged AUCs slightly favored SFM over FFDM (biggest AUC difference: 0.047, <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:mi>SE</mml:mi> <mml:mo>=</mml:mo> <mml:mn>0.023</mml:mn></mml:mrow> </mml:math> , <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:mi>p</mml:mi> <mml:mo>=</mml:mo> <mml:mn>0.047</mml:mn></mml:mrow> </mml:math> ), where standard error accounts for reader and case variability. The differences were not significant at a level of 0.01 (0.05/5 reader studies). The differences in sensitivities and specificities were also indeterminate. Prevalence had little effect on AUC (largest difference: 0.02), whereas sensitivity increased and specificity decreased as prevalence increased. We found that AUC is robust to changes in prevalence, while radiologists were more aggressive with recall decisions as prevalence increased.
BibTeX
@article{gallas2019impact,
  author = {Brandon D. Gallas and Weijie Chen and Elodia B. Cole and Robert Ochs and Nicholas Petrick and Etta D. Pisano and Berkman Sahiner and F. W. Samuelson and Kyle J. Myers},
  title = {{Impact of prevalence and case distribution in lab-based diagnostic imaging studies}},
  journal = {Journal of Medical Imaging},
  year = {2019},
  doi = {10.1117/1.jmi.6.1.015501},
}

Loading related content...

Figure