Paired split-plot designs of multireader multicase studies

Weijie Chen, Qi Gong, Brandon D. GallasJournal of Medical Imaging 2018 · DOI

TL;DR

This paper introduces and analyzes the "paired split-plot" (PSP) design for multi-reader, multi-case (MRMC) studies, offering a more efficient and flexible alternative to the traditional "fully crossed" (FC) design. PSP designs can achieve the same statistical power as FC designs with fewer reader readings, but at the cost of requiring more patient cases, enabling a crucial trade-off between study resources.

What Problem Does This Paper Address?

Comparing medical imaging modalities often relies on MRMC studies where multiple radiologists (readers) evaluate multiple patient cases. The standard approach, the fully crossed (FC) design, requires every reader to examine every case for both modalities. While this design maximizes the information gained from a given set of readers and cases, it can lead to substantial reader workload and long study durations. The challenge is to find study designs that maintain statistical rigor while reducing costs and increasing flexibility, particularly concerning reader time and the number of patient cases needed.

Methods

The authors employ a U-statistic based variance analysis to analytically compare the statistical efficiency of the paired split-plot (PSP) design against the fully crossed (FC) design. They decompose the variance of the area under the ROC curve (AUC) into components related to reader variability, case variability, and their interactions. The PSP design partitions readers and cases into groups, with readers within a group reading a subset of cases. This structure allows for analytical comparisons of variance and statistical power under various scenarios. They also validate their findings using a simulation model and demonstrate the practical application with a real-world reader study.

Datasets

The paper uses simulated data based on the Roe and Metz model to explore different levels of data correlation and reader variability. Additionally, they analyze data from the VIPER study, a retrospective reader study comparing full-field digital mammography (FFDM) with screen-film mammography (SFM), to compare the PSP and FC designs using real-world variance parameters.

Key Insights and Ideas

  • Efficiency Gain: PSP designs can be statistically more efficient than FC designs when the number of readers and the workload per reader are held constant. This efficiency stems from a reduction in the case variability component of the overall variance.
  • Cost-Benefit Trade-off: The PSP design offers a clear trade-off: increased statistical precision (or power) can be achieved with fewer reader readings, but this comes at the expense of needing a larger number of truth-verified patient cases. This allows researchers to tailor study designs based on available resources.
  • Reduced Reader Workload: By having readers examine different subsets of cases, PSP designs can significantly reduce the time commitment for individual readers, which is often a major cost factor in MRMC studies.
  • Flexibility: PSP designs can offer practical advantages, such as potentially shorter study durations and the ability to conduct studies across multiple institutions without the need to ship cases.
  • Software Validation: The paper validates the iMRMC software's ability to reliably analyze data from both FC and PSP designs, crucial for practical implementation.

Key Results

  • Analytical derivations show that PSP designs can achieve the same statistical power as FC designs with fewer reader readings.
  • Simulations (Table 2) demonstrate that for various data correlation and reader variability structures, PSP designs (PSP2, PSP4, PSP8) consistently yield higher statistical power than the FC design for the same number of readings. For instance, under the "HH" structure with AUCs of 0.90 and 0.95, the FC design achieves 68.0% power, while PSP8 reaches 93.0% power with the same number of readings.
  • Table 3 illustrates the trade-off: to achieve 80% power, the FC design requires significantly fewer cases per reader (e.g., 1197 for HH structure) compared to PSP designs (e.g., 198 for PSP8), but the total number of cases for PSP designs increases with the number of groups.
  • Analysis of the VIPER study data (Table 4) showed that an FC study would require 56% more uncertainty (larger SE of AUC difference) than a PSP4 study, despite the FC study using 25% fewer cases.

Limitations and Future Work

  • The paper acknowledges that the Roe and Metz simulation parameters may not always be perfectly realistic, motivating the use of real-world data from the VIPER study.
  • The current iMRMC software primarily supports binary performance endpoints and AUC estimated by the trapezoidal/Wilcoxon method. Future work could extend this to other endpoints or estimation methods.
  • The authors suggest that future research could explore the connection between U-statistic parameters and Obuchowski-Rockette (OR) model parameters more broadly, and survey published studies analyzed by both methods.
  • While the paper focuses on the trade-off between reader workload and case collection costs, other cost factors could be considered in future analyses.

Summary updated

Abstract

The widely used multireader multicase ROC study design for comparing imaging modalities is the fully crossed (FC) design: every reader reads every case of both modalities. We investigate paired split-plot (PSP) designs that may allow for reduced cost and increased flexibility compared with the FC design. In the PSP design, case images from two modalities are read by the same readers, thereby the readings are paired across modalities. However, within each modality, not every reader reads every case. Instead, both the readers and the cases are partitioned into a fixed number of groups and each group of readers reads its own group of cases-a split-plot design. Using a [Formula: see text]-statistic based variance analysis for AUC (i.e., area under the ROC curve), we show analytically that precision can be gained by the PSP design as compared with the FC design with the same number of readers and readings. Equivalently, we show that the PSP design can achieve the same statistical power as the FC design with a reduced number of readings. The trade-off for the increased precision in the PSP design is the cost of collecting a larger number of truth-verified patient cases than the FC design. This means that one can trade-off between different sources of cost and choose a least burdensome design. We provide a validation study to show the iMRMC software can be reliably used for analyzing data from both FC and PSP designs. Finally, we demonstrate the advantages of the PSP design with a reader study comparing full-field digital mammography with screen-film mammography.
BibTeX
@article{chen2018paired,
  author = {Weijie Chen and Qi Gong and Brandon D. Gallas},
  title = {{Paired split-plot designs of multireader multicase studies}},
  journal = {Journal of Medical Imaging},
  year = {2018},
  doi = {10.1117/1.jmi.5.3.031410},
}
Figure