Knowledge based computerized audit of radiologists' performance

This study investigated a new approach to measure the quality of X-ray reports in radiology department and evaluate the performance of radiologists. This approach involves the use of Medical Information System (MIS), expert system technology, information theory, and other techniques to generate relative measures of radiologists' performance. Eight algorithms are: 1) Outcome Directed Information Current (Score 1); 2) Outcome Directed Probability Change (Score2); 3) Outcome Directed Weight Evidence (Score3); 4) Disease Specific Information Content (Score4); 5) Ranking of Information Content (RI); 6) Ranking of Absolute Value of Information Content (RI2); 7) Ranking Probability (RP); and 8) Ranking of Likelihood Ratio (RLL). ANOVA analysis were used to identify the algorithms that can best distinguish the performance of radiologists The result of the ANOVA showed the Score1, Score2, and Score3 had more power to distinguish the performance of radiologists both for cases across different diseases and for cases with a particular disease such as chronic bronchitis and nonpulmonary disease. However, they may not be a good measure in distinguishing radiologist performance for cases with disease such as congestive heart failure. In addition RI2 seemed to be a relatively good measure for distinguishing the performance radiologists across pulmonary diseases. No scoring algorithm was shown to be able to significantly distinguish radiologist performance for cases with congestive heart failure. Discharge diagnoses of patients significantly influenced the performance of radiologists. While one radiologist may do well in cases with some diseases, he or she may do poorly in cases with other diseases. In addition, whether the performance of radiologist can be significantly distinguished by a specific scoring algorithm was also influenced by the disease. As a result, the evaluation of radiologist should be disease specific. First of all, different scoring algorithms might be required to distinguish radiologist performance for different disease categories. Furthermore, the performance of radiologist in reading X-ray reports for patients with one disease might be quite different from that in reading X-ray reports for patients with another disease. Using average performance for different diseases might lead to confusing performance results.