Usability evaluation of a vendor's emergency department computerized provider order entry application at the University of Utah Hospital

Electronic Health Record (EHR) adoption rates have been low in the United States. A key reason for this low adoption rate is poor EHR usability. Currently no standards exist for design, testing and monitoring the usability of EHRs. Therefore, we conducted a usability evaluation of a vendor's product in the Emergency Department at the University of Utah. In the first objective of this study, we evaluated a newly implemented computerized provider order entry application. Four usability experts used the Zhang et al 14 heuristics and 23 predefined tasks to perform the evaluation. The experts found 48 usability problems categorized into 51 heuristic violations. There were 4 cosmetic, 120 minor, 64 major, and 4 catastrophic problems identified. The interrater reliability was 0.81 using Fleis' Kappa, showing a high level of consistency in ratings across evaluators. For the second objective, we used an electronic version of Questionnaire of User Interaction Satisfaction (QUIS 7.0) to evaluate physician satisfaction with the CPOE application in the ED. The physician response rate was 50% (25/50). The total survey mean was 4.87, lower than the -a priori‖ definition for acceptable satisfaction score of 5.0 (of a possible 9). The lowest scale scores were for overall user reaction and learning iv and the highest were for screen, terminology and system capabilities. Further analyses were completed to determine any differences for satisfaction scores between physician trainees and attending. A multifactor ANOVA was performed to examine the combined effect of the different experience levels and sections of the QUIS. The results were significant at -1.43 (p < 0.05) for screen and terminology and system capabilities. In this setting, the ED CPOE application had a high level of usability issues and low mean satisfaction scores among physician end-users. The responsibility for improved usability lies with both vendors developing the product and facilities implementing the product and both should be educated on usability principles. The combination of a user-based and expert-based inspection method yielded congruent findings and was an accurate and efficient means of evaluation.