Sample Size Estimates for DBM MRMC based on Analysis of Published Data
One of the aims of our grant R01 EB 00863 from the NIBIB is to provide estimates of minimum reader and case sample sizes needed to detect performance differences. Sample size estimation is important in the planning of future experiments. Data from pilot experiments, or comparable experiments, is required to make these calculations credible. By providing sample size estimates based on published articles with multireader ROC data sets, we hope to assist in the planning of radiology experiments so that reader and case sample sizes are appropriate.
Our research for sample size estimation using the DBM method is described in Hillis & Berbaum (2004). We also developed refinements of the DBM method and showed the connection with the Obuchowski-Rockette corrected F method, with this research described in Hillis, Obuchowski, Schartz & Berbaum (2005). We also showed how the variance components can be estimated, which are then used for sample size computation as described in Hillis & Berbaum (2004). The DBM MRMC 2.1 program takes into account the refinements developed in Hillis, Obuchowski, Schartz & Berbaum (2005) and provides estimates of variance components that will allow users to make sample size and power estimates as described in Hillis & Berbaum (2004). Hillis (2006) derives a new denominator degrees of freedom (ddfHillis) for both the DBM and Obuchowski-Rockette procedures and empirically shows that new model simplification plus ddfHillis performs better than new model simplification. We have updated the Hillis & Berbaum (2004) procedure to base the power computation on new model simplification plus ddfHillis.
For a given effect size there will be several different reader-case sample size combinations that yield the desired power. It is important that the pilot study or previous study used for the computations be comparable to the planned study with respect to the modalities, reader expertise, case selection, ratio of normal to abnormal cases, and how the rating data are fitted.
Cautionary note: For some data, the estimated required number of cases when only cases are treated as random sometimes exceeds the required number when both cases and readers are treated as random. If the variance components were all known, this would not be possible. It happens here because we are using estimates of the variance components. To be conservative, we recommend that the higher of the two estimates be used. Alternatively, it is possible to pool information from similar studies, resulting in more precise variance component estimates and hence more precise sample size estimates.
The table below provides links to pages providing details on previous studies. Researchers planning a study can search through the links in the following table to find a previous study that is similar to the planned study. For each of these studies, a brief description is provided that lists the number of readers, the number of cases, the treatments/modalities used, the ROC model used, the ROC parameter used, and components of variance from the ANOVA. Following this description are three tables: one for an AUC difference of .03, one for an AUC difference of .05, and one for an AUC difference of .10 (a fourth table for an AUC difference of .20 is provided for some previous studies). In each of these tables, the rows indicate number of readers in the study. There is a column for readers and cases random, a column for only cases random, and a column for only readers random.
Questions regarding sample size estimation should be directed to Kevin Berbaum (firstname.lastname@example.org).
Hillis SL. A comparison of denominator degrees of freedom methods for multiple observer ROC analysis. Statistics in Medicine 2006; in press.
Hillis SL, Berbaum KS. Power estimation for the Dorfman-Berbaum-Metz method. Academic Radiology 2004;11:1260-1273.
Hillis SL, Berbaum KS. Monte Carlo validation of the Dorfman-Berbaum-Metz method using normalized pseudovalues and less data-based model simplification. Academic Radiology 2005;12:1534-1542.
Hillis SL, Obuchowski NA, Schartz KM, Berbaum KS. A comparison of the Dorfman-Berbaum-Metz and Obuchowski-Rockette Methods for receiver operating characteristic (ROC) data. Statistics in Medicine 2005;24:1579-1607.