This proposal addresses a
fundamental methodological problem in diagnostic radiology research: how to
evaluate the diagnostic performance of imaging modalities. Receiver operating
characteristic (ROC) analysis is an accepted procedure for measuring how well
imaging modalities allow physicians to detect disease. Traditional
statistical comparison of different imaging modalities only analyzes ROC
measures of a single radiologist from each modality. Dorfman, Berbaum and
Metz (1992) developed a statistical methodology that overcomes this
limitation and provides a comprehensive framework within which to analyze ROC
studies with multiple readers, taking into account both reader-sample and
case-sample variation. The Dorfman-Berbaum-Metz (DBM) methodology allows
greater statistical power to detect clinically meaningful differences in
diagnostic performance than single-radiologist approaches. Its validity has been
tested extensively using computer simulations and has wide acceptance in
diagnostic radiology experiments. Because DBM methodology relies on ROC
analysis for its dependent measures, the robustness and accuracy of the ROC
analysis affect the statistical precision and power of DBM methodology. DBM
analysis of some experiments fails because of flaws in traditional ROC
measures. Since the DBM methodology was developed, there have been
fundamental advances in ROC analysis, which provide more interpretable measures.
DBM could be more robust and statistically powerful if it took advantage of
these advances. To accomplish this we propose building an integrated,
public-domain software resource to bring together methodological advantages
previously only available in separate programs.
The
specific aims are: (1) develop a new modular DBM software architecture that is
freely accessible by other software and allows greater flexibility in ROC
analysis; (2) develop “proper” ROC analysis modules that provide more interpretable
ROC results and study the resulting DBM statistical power; (3) develop modules
for types of ROC analysis that take into account location of reported disease
and multiple reports of disease, features that have been shown to improve
statistical power; (4) develop modules and procedures to merge DBM methodology
with standard statistical software to answer more complex experimental
questions; and (5) use DBM re-analysis of published data to provide estimates
of minimum reader and case sample size combinations needed to detect various
performance differences with power for various imaging areas. Completing these
aims will enhance the effectiveness of the primary method used to assess the
diagnostic performance of new imaging systems. This will lead to better
radiology research, which will raise the quality and lower the cost of health
care.
|