Advancing clinical decision support (CDS) and electronic clinical quality measurement (ECQM)

Clinical decision support (CDS) and electronic clinical quality measurement (eCQM) are 2 important computerized strategies aimed at improving the quality of healthcare. Unfortunately, computer-facilitated quality improvement faces many barriers. One problem area is the lack of integration of CDS and eCQM, which leads to duplicative efforts, inefficiencies, misalignment of CDS and eCQM implementations, and lack of appropriate automated feedback on clinicians' performance. Another obstacle in the acceptance of electronic interventions can be the inadequate accuracy of electronic phenotyping, which leads to alert fatigue and clinicians' mistrust of eCQM results. To address these 2 problems, the research pursued 3 primary aims: Aim 1. Explore beliefs and perceptions regarding the integration of CDS and eCQM functionality and activities within a healthcare organization. Aim 2. Evaluate and demonstrate feasibility of implementing quality measures using a CDS infrastructure. Aim 3. Assess and improve strategies for human validation of electronic phenotype evaluation results. To address Aim 1, a qualitative study based on interviews with domain experts was performed. Through semistructured in-depth and critical incident interviews, stakeholders' insights about CDS and eCQM integration were obtained. The experts identified multiple barriers to the integration of CDS and eCQM and offered advice for addressing those barriers, which the research team synthesized into 10 recommendations. To address Aim 2, the feasibility of using a standards-based CDS framework aligned with anticipated electronic health record (EHR) certification criteria to implement electronic quality measurement (QM) was evaluated. The CDS-QM framework was used to automate a complex national quality measure at an academic healthcare system which had previously relied on time-consuming manual chart abstractions. To address Aim 3, a randomized controlled study was conducted to evaluate whether electronic phenotyping results should be used to support manual chart review during single-reviewer electronic phenotyping validation. The accuracy, duration, and cost of manual chart review were evaluated with and without the availability of electronic phenotyping results, including relevant patient-specific details. Providing electronic phenotyping results was associated with improved overall accuracy of manual chart review and decreased review duration per test case. Overall, the findings informed new strategies for enhancing efficiency and accuracy of computer-facilitated quality improvement.

ADVANCING CLINICAL DECISION SUPPORT (CDS) AND ELECTRONIC CLINICAL QUALITY MEASUREMENT (ECQM) by Polina Kukhareva A dissertation submitted to the faculty of The University of Utah in partial fulfillment of the requirements for the degree of Doctor of Philosophy Department of Biomedical Informatics The University of Utah December 2017 Copyright © Polina Kukhareva 2017 All Rights Reserved The University of Utah Graduate School STATEMENT OF DISSERTATION APPROVAL The dissertation of Polina Kukhareva has been approved by the following supervisory committee members: , Chair Kensaku Kawamoto 09/05/2017 Date Approved , Member Catherine J. Staes 09/05/2017 Date Approved , Member Charlene R. Weir 09/05/2017 Date Approved , Member Stanley M. Huff 09/05/2017 Date Approved , Member Howard R. Weeks 09/05/2017 Date Approved and by Wendy W. Chapman the Department/College/School of , Chair/Dean of Biomedical Informatics and by David B. Kieda, Dean of The Graduate School. ABSTRACT Clinical decision support (CDS) and electronic clinical quality measurement (eCQM) are 2 important computerized strategies aimed at improving the quality of healthcare. Unfortunately, computer-facilitated quality improvement faces many barriers. One problem area is the lack of integration of CDS and eCQM, which leads to duplicative efforts, inefficiencies, misalignment of CDS and eCQM implementations, and lack of appropriate automated feedback on clinicians' performance. Another obstacle in the acceptance of electronic interventions can be the inadequate accuracy of electronic phenotyping, which leads to alert fatigue and clinicians' mistrust of eCQM results. To address these 2 problems, the research pursued 3 primary aims: Aim 1. Explore beliefs and perceptions regarding the integration of CDS and eCQM functionality and activities within a healthcare organization. Aim 2. Evaluate and demonstrate feasibility of implementing quality measures using a CDS infrastructure. Aim 3. Assess and improve strategies for human validation of electronic phenotype evaluation results. To address Aim 1, a qualitative study based on interviews with domain experts was performed. Through semistructured in-depth and critical incident interviews, stakeholders' insights about CDS and eCQM integration were obtained. The experts identified multiple barriers to the integration of CDS and eCQM and offered advice for addressing those barriers, which the research team synthesized into 10 recommendations. To address Aim 2, the feasibility of using a standards-based CDS framework aligned with anticipated electronic health record (EHR) certification criteria to implement electronic quality measurement (QM) was evaluated. The CDS-QM framework was used to automate a complex national quality measure at an academic healthcare system which had previously relied on time-consuming manual chart abstractions. To address Aim 3, a randomized controlled study was conducted to evaluate whether electronic phenotyping results should be used to support manual chart review during single-reviewer electronic phenotyping validation. The accuracy, duration, and cost of manual chart review were evaluated with and without the availability of electronic phenotyping results, including relevant patient-specific details. Providing electronic phenotyping results was associated with improved overall accuracy of manual chart review and decreased review duration per test case. Overall, the findings informed new strategies for enhancing efficiency and accuracy of computer-facilitated quality improvement. iv TABLE OF CONTENTS ABSTRACT ....................................................................................................................... iii LIST OF TABLES ........................................................................................................... viii LIST OF FIGURES ........................................................................................................... ix ACKNOWLEDGMENTS .................................................................................................. x Chapters 1 INTRODUCTION ........................................................................................................... 1 1.1 Clinical Quality Improvement Strategies................................................. 1 1.1.1 Clinical Decision Support (CDS)................................................. 1 1.1.2 Electronic Clinical Quality Measurement (eCQM) ..................... 2 1.2. Challenges Facing Quality Improvement Efforts ................................... 4 1.3 Potential Solutions ................................................................................... 5 1.3.1 Integration of CDS and eCQM .................................................... 6 1.3.2 Improving Strategies for Validating Results of Electronic Phenotyping .......................................................................................... 7 1.4 Dissertation Aims..................................................................................... 8 1.5 References .............................................................................................. 10 2 CHALLENGES AND RECOMMENDATIONS FOR INTEGRATION OF CLINICAL DECISION SUPPORT AND ELECTRONIC CLINICAL QUALITY MEASUREMENT: INSIGHTS FROM DOMAIN EXPERTS ........................................ 14 2.1 Abstract .................................................................................................. 14 2.2 Background ............................................................................................ 15 2.2.1 Objective .................................................................................... 18 2.3 Materials and Methods ........................................................................... 18 2.3.1 Study Design .............................................................................. 18 2.3.2 Research Team ........................................................................... 18 2.3.3 Subjects ...................................................................................... 18 2.3.4 Interviews ................................................................................... 19 2.3.5 Data Analysis ............................................................................. 21 2.4 Results .................................................................................................... 22 2.4.1 Participants ................................................................................. 22 2.4.2 Similarities and Differences in CDS and eCQM Practice ......... 23 2.4.3 Potential Benefits of Integration of CDS and eCQM ................ 24 2.4.4 Barriers for Integrating CDS and eCQM ................................... 24 2.4.5 Recommendations for Integration of CDS and eCQM .............. 24 2.5 Discussion .............................................................................................. 26 2.6 Conclusion ............................................................................................. 31 2.7 References .............................................................................................. 40 3 CLINICAL DECISION SUPPORT-BASED QUALITY MEASUREMENT (CDS-QM) FRAMEWORK: PROTOTYPE IMPLEMENTATION, EVALUATION, AND FUTURE DIRECTIONS .................................................................................................. 43 3.1 Abstract .................................................................................................. 44 3.2 Introduction ............................................................................................ 44 3.2.1 Overview of Clinical Quality Measurement (QM). ................... 44 3.2.2 Previous Work in Automating QM ............................................ 45 3.2.3 The Problem: Duplicative, Divergent Implementation of QM and CDS. .................................................................................................... 45 3.2.4 Potential Solution: Leverage a Standards-based CDS Web Service Across a Population for QM. ................................................. 45 3.3 Methods.................................................................................................. 46 3.3.1 Identification of Opportunities to Enhance Quality Improvement Using CDS-QM .................................................................................. 46 3.3.2 Design and Implementation of CDS-QM Framework ............... 46 3.3.3 Evaluation of CDS-QM Approach for Representative National Quality Measure .................................................................................. 46 3.4 Results .................................................................................................... 47 3.4.1 Opportunities to Enhance Quality Improvement Using CDS-QM ............................................................................................. 47 3.4.2 Design and Implementation of CDS-QM Framework ............... 48 3.4.3 Evaluation of CDS-QM Approach for Representative National Quality Measure .................................................................................. 49 3.5 Discussion .............................................................................................. 50 3.6 Conclusion ............................................................................................. 51 3.6.1 Acknowledgements. ................................................................... 51 3.7 References .............................................................................................. 51 4 SINGLE-REVIEWER ELECTRONIC PHENOTYPING VALIDATION IN OPERATIONAL SETTINGS: COMPARISON OF STRATEGIES AND RECOMMENDATIONS .................................................................................................. 54 4.1 Introduction ............................................................................................ 56 4.2 Objectives .............................................................................................. 57 4.3 Materials and Methods ........................................................................... 57 4.3.1 Design ........................................................................................ 57 4.3.2 Setting ........................................................................................ 57 4.3.3 Study Population ........................................................................ 57 vi 4.3.4 Intervention ................................................................................ 57 4.3.5 Manual Chart Review ................................................................ 57 4.3.6 Electronic Phenotyping .............................................................. 57 4.3.7 Quality Measures ....................................................................... 57 4.3.8 Validation Framework ............................................................... 57 4.3.9 Validation Process and Development of the Reference Standard .............................................................................................. 59 4.3.10 Statistical Analysis ................................................................... 59 4.4 Results .................................................................................................... 60 4.4.1 Accuracy of Manual Chart Review............................................ 60 4.4.2 Duration of Manual Chart Review ............................................. 60 4.4.3 Cost of Detecting One Erroneous Electronic Phenotyping Result .................................................................................................. 61 4.4.4 Ability to Detect Errors in Electronic Phenotyping Results ...... 61 4.4.5 Description of Human Errors with Manual Chart Review ........ 61 4.5 Discussion .............................................................................................. 61 4.6 Conclusion ............................................................................................. 63 4.7 Acknowledgments.................................................................................. 64 4.8 References .............................................................................................. 64 5 DISCUSSION ................................................................................................................ 65 5.1 Concurrent Efforts by Others ................................................................. 66 5.2 Context Within Continuous Clinical Quality Improvement .................. 67 5.3 Significance............................................................................................ 67 5.4 Innovation .............................................................................................. 68 5.5 Limitations ............................................................................................. 69 5.6 Future Directions ................................................................................... 69 5.7 References .............................................................................................. 71 6 CONCLUSION .............................................................................................................. 73 vii LIST OF TABLES Tables 1.1 Challenges facing electronic quality improvement efforts and potential solutions ...... 9 2.1. Participants ................................................................................................................. 32 2.2. Similarities and differences between CDS and eCQM .............................................. 33 2.3. Benefits of an integrated approach to CDS and eCQM ............................................. 36 2.4. Barriers to the integration of CDS and eCQM........................................................... 37 3.1. Comparison of the denominator (inclusion/exclusion) classification, using two methods ............................................................................................................................. 49 3.2. Comparison of the numerator (passed/failed) criteria, using two methods ............... 50 4.1. Quality measure groups and chart review sample size for each stratum ................... 58 4.2. Performance of intervention and control strategies ................................................... 60 4.3. Ability of manual chart review to detect errors in electronic phenotyping for control and intervention chart review strategies ........................................................................... 62 4.4. Types and examples of human errors in manual chart review results ....................... 63 LIST OF FIGURES Figures 3.1 OpenCDS architecture: high-level interaction for CDS ............................................. 45 3.2 Comparison of traditional and CDS-QM approaches to quality measurement and improvement ..................................................................................................................... 47 3.3 Major systems and processes involved in the CDS-QM approach ............................. 48 3.4 SCIP-VTE-2 Business Logic represented in Guvnor ................................................. 49 4.1 A. Screenshot of a spreadsheet for control review strategy. B. Screenshot of a spreadsheet for intervention review strategy. ................................................................... 58 4.2 Quality measure evaluation process............................................................................ 59 4.3 Validation process and development of the reference standard.................................. 59 4.4 Impact of the availability of electronic phenotyping results across 26 HEDIS quality measure groups on: (A) average accuracy of review and (B) average time to review a test case. ................................................................................................................................... 61 ACKNOWLEDGMENTS Firstly, I would like to express my genuine gratitude to my scientific advisor Dr. Kensaku Kawamoto for the constant support of my PhD study and related research, for his patience, inspiration, and enormous knowledge. His guidance helped me in all the time of research and writing of this dissertation. I could not have imagined having a better advisor and mentor for my PhD study. I would also like to particularly thank Dr. Catherine Staes for support, advice and ideas which helped me with this dissertation. I would like to thank the rest of my dissertation committee: Dr. Charlene Weir, Dr. Stanley M. Huff, and Dr. Howard R. Weeks for their insightful comments and encouragement. I thank my fellow colleagues from the Knowledge Management and Mobilization Team for the useful advice, shared work experience, and for all the fun we have had in the last four and a half years. Additionally, I would like to thank all the professors, postdocs and students at the University of Utah Department of Biomedical Informatics for creating a healthy and supportive environment. Finally yet importantly, I would like to thank my family: my husband, my parents and my sister for supporting me spiritually throughout writing this dissertation. CHAPTER 1 INTRODUCTION 1.1 Clinical Quality Improvement Strategies Delivering quality healthcare is challenging due to ongoing and ubiquitous variation in health system processes that may lead to errors.1 Measuring and reducing variation from evidence-based clinical recommendations have been shown to improve quality and decrease costs of healthcare.2 The increasing adoption of electronic health records (EHRs) and associated interoperability standards in recent years has created a foundation upon which structured electronic data can be used to facilitate quality improvement strategies such as CDS and clinical quality measurement (CQM). 1.1.1 Clinical Decision Support (CDS) A substantial body of evidence shows that, if correctly implemented, CDS could be effective in improving clinical and process outcomes.3 Initially, many large academic hospitals developed their own EHRs and their own CDS Systems (CDSSs). Later, when home-grown EHR systems were replaced by commercial EHR systems, those CDSSs could not be easily adopted because they were tightly coupled with the home-grown EHR systems for which they were developed. Currently, many CDSSs are built on top of the 2 customized implementations of commercial EHRs specific to a given healthcare organization. Kawamoto et al. have previously suggested that a standards-based, serviceoriented architecture could be used to make CDS logic sharable between different EHRs.4 In pursuing this potential approach to CDS, a promising resource is an open-source, standards-based, service-oriented framework for CDS known as OpenCDS.5 An EHR system can submit patient data to OpenCDS and obtain patient-specific assessments and recommendations that are provided to clinicians via alerts, reminders, or other CDS modalities.6 OpenCDS is compliant with the HL7 Virtual Medical Record (vMR) and HL7 Decision Support Service (DSS) standards, and it leverages various open-source component resources, including the JBoss Drools knowledge management platform and Apelon Distributed Terminology System. 1.1.2 Electronic Clinical Quality Measurement (eCQM) Clinical quality measures are measures of processes, experiences, and/or outcomes of patient care. Having a means to assess healthcare quality is essential for identifying deviations from evidence-based best practices and mitigating preventable errors.7 Currently, CQM is required by public and private payers, regulators, accreditors and others that certify performance levels for consumers, patients and payers.8 Current quality measurement systems in many hospitals include time-consuming manual paper and electronic record abstraction by a quality improvement specialist.9,10 At large academic medical centers such as University of Utah Health Care (UUHC), manual data abstraction is often followed by data analysis performed by an external organization 3 such as the University HealthSystem Consortium.11,12 There are several limitations with this process. For example, (a) 3 to 6 months may elapse between the time of a clinical procedure (eg, a surgery) and the time when feedback is given to a clinician; (b) human errors may be introduced during manual record review; and (c) only a subset of the patients and clinical events is oftentimes selected for review, leading to gaps in quality assessment coverage. Theoretically, the above problems could be solved using electronic clinical quality measurement (eCQM). There are increasing mandates and financial incentives to use EHRs to measure quality as opposed to employing traditional manual processes for QM.7,13 For example, the Meaningful Use (MU) recommendations issued by the federal Health Information Technology Policy Committee in November 2012 require the implementation of eCQM as well as CDS for high-priority conditions, and the use of related standards.13,14 One of the promises of implementing EHRs is the possibility for automatic generation of eCQM.10 A MU-certified EHR must be able to export standardized quality reports, which can then "be fed into a calculation engine to compute various aggregate scores."15 Following these recommendations, major EHR vendors such as Epic have started to integrate eCQM logic into their products.16 Currently, however, only some EHR vendors offer quality measurements embedded in their system, and the scope of measures supported is not always comprehensive.10,16 For example, a study in 2010 found that KPHealth Connect had automated 6 of 13 Joint Commission measurement sets.10 As well, EpicCare Inpatient 2014 and EpicCare Ambulatory 2014 offered 56 National Quality Forum (NQF) quality measures required for MU certification out of over 700 NQF-endorsed measures on their 4 website.16 While vendor-based solutions may be comprehensive in the scope of patients analyzed, their implementation may be a "black box" where the inner workings of the algorithms employed are difficult to discern. Also, it is not always clear which version of each rule has been implemented or whether the quality measure logic is up-to-date. In addition, users may not have control over the logic to customize quality measurement. Even so, automated eCQM has the potential to provide quality reports on demand, may avoid human errors in manual abstraction, and can analyze 100% of relevant patients and their encounters, as opposed to analyzing only a subset when using manual phenotyping. Most ongoing efforts to produce automated quality measures are tied to a specific EHR system, and the executable logic for the quality measure is not sharable between different EHR systems.10 1.2 Challenges Facing Quality Improvement Efforts Despite multiple efforts undertaken to improve healthcare quality since the publication of the Institute of Medicine reports "To Err Is Human"17 and "Crossing the Quality Chasm"18, the quality of healthcare in the United States continues to be compromised by unnecessary variation in the implementation of clinical practice guidelines. Deficiencies in CDS and eCQM design, implementation, and maintenance, as well as misaligned incentives, can cause the low effectiveness of CDS and eCQM. For example, a meta-analysis of 26 papers showed that usability flaws in medication alerting systems have negative impact on workflow, technology effectiveness, medication management processes, and patient safety.19 Informatics-based quality improvement efforts often fail to reach their goal because of multiple issues as summarized in Table 5 1.1. This dissertation research addressed 2 of these challenges: the lack of integration between CDS and eCQM and the low accuracy of phenotyping. CDS and eCQM were traditionally implemented in silos and discussed separately in the medical literature. Only 3 out of 160 randomized clinical trials described in a systematic review by Lobach et al. describe CDSSs accompanied by periodic performance feedback, possibly because feedback requires additional development effort and could not be easily integrated with CDS.3 Once implemented, both CDS and eCQM need to be regularly reviewed and potentially updated. When they are programmed separately, maintenance of the logic requires duplication of effort. In addition, CDS and eCQM logic may get updated asynchronously or differently, which could cause confusion among clinicians. These issues may be exacerbated by differences in the background of personnel performing quality oversight compared to the technical personnel tasked with implementing decision support. Integration may be difficult when the incentives and mission are misaligned between the 2 teams. Furthermore, validation processes for both CDS and eCQM need to be improved. Studies have shown that electronically reported MU quality measures have low accuracy.20 Similarly, studies have shown that CDS use is compromised by alert fatigue and low attention of clinicians to some CDS alerts, partly due to poor accuracy of alerts.19 1.3 Potential Solutions Potential solutions have been mapped to the challenges described above (Table 1.1). 6 1.3.1 Integration of CDS and eCQM CDS and eCQM are highly related, as eCQM focuses on who is eligible for a needed intervention (denominator identification) and who among them has received the needed intervention (numerator identification), whereas CDS focuses on who is eligible for a needed intervention and has not received the needed intervention (equivalent to numerator identification). However, to the best of our knowledge, there have been limited reports of evaluation and validation in the literature concerning how technical approaches for one problem space can be reused in the other, especially pertaining to standards-based approaches. This finding is important because EHR certification criteria will likely require more automation and need for validation in the future. It has been previously suggested that CDS and eCQM could be combined.21 Furthermore, there has been a trend towards viewing CDS and eCQM as two sides of the same coin. There was a qualitative field study performed at the Regenstrief Institute, Partners Health Care System, and Veterans Health Administration that showed a paradigm shift from viewing CDS and performance measures as 2 separate approaches to viewing a clinical reminder as a real time performance measure with an "n of one."22 It has been shown that clinical reminders corresponding to performance measures could improve organizational performance.3 Diabetes care was shown to improve significantly when a multifaceted intervention combining reminders and performance feedback was introduced.23 This finding is congruent with the findings from a systematic review by Forrest et al. In this systematic review of randomized controlled trials for patients with type 2 diabetes, Forrest et al. found that CDS only improves patient outcomes when combined with feedback on performance.24 7 New methods are currently being developed to unify CDS and eCQM and follow the success of clinical pathways implementation.25 There have been efforts to combine CDS and eCQM logic. For example, one of the proposed solutions is to use the National Quality Forum (NQF) Quality Data Model (QDM) and JBoss Drools rules engine.26-28 However, these efforts are often not standards-based, and no conceptual framework was developed.29-31 We hypothesized that technical integration of CDS and eCQM could be reached by leveraging a standards-based CDS Web service across a population for both eCQM and CDS. In pursuing the integration of CDS and eCQM, it is important to understand the viewpoint and experience of different stakeholders, such as members of institutional quality teams and CDS teams. Thus, we proposed to investigate both cultural and technical challenges preventing CDS and eCQM integration and to develop a framework which would allow implementing CDS and eCQM simultaneously. 1.3.2 Improving Strategies for Validating Results of Electronic Phenotyping Computable phenotyping entails automatic identification of patient records satisfying specific conditions. Computable phenotyping is essential for CQM, CDS, risk adjustment, clinical registries, predictive analytics, public reporting, and cohort identification for clinical trials and research.32 Accuracy of such phenotyping is essential for CDS and eCQM to be optimally effective. For example, a time-series analysis at a large internal medicine practice using a commercial EHR showed that making point-of care reminders and feedback more accurate accelerated the rate of quality improvement.33 The testing of electronic phenotyping algorithms is important to detect errors and 8 provide high quality results over time. Double human chart review is often considered a "gold standard" of phenotyping validation in research and academic settings34-37; however, it is too expensive and slow to be used in operational settings.38 Human review is subject to error and produces both false negative and false positive results when used to detect errors. This dissertation aims to develop a single human review-based phenotyping validation approach that is both pragmatic and high-performing. Currently, there is no standard framework for electronic phenotyping validation. Newton et al. presented recommendations for phenotyping algorithms validation but did not focus on human expert review.39 While validating quality measures for enterprise implementation at UUHC, our group initially developed an ad hoc validation methodology that was not sufficiently robust. We neither selected cases randomly, nor did we ensure an adequate mix of positive and negative results. To improve the quality of our validation strategy, we developed and formally evaluated a new and more robust electronic phenotyping validation framework. 1.4 Dissertation Aims To address the problems raised above, the research had 3 primary aims: Aim 1. Explore beliefs and perceptions regarding the integration of CDS and eCQM functionality and activities within a healthcare organization. Aim 2. Evaluate and demonstrate feasibility of implementing quality measures using a CDS infrastructure. Aim 3. Assess and improve strategies for human validation of electronic phenotype evaluation results. 9 Table 1.1 Challenges facing electronic quality improvement efforts and potential solutions Challenge Poor user interface design Lack of interoperability40 Lack of technical approaches to co-implement CDS and eCQM Lack of organizational integration between CDS and eCQM teams Clinicians do not see a need for computerized quality improvement.41 Low accuracy of electronic phenotyping Misaligned incentives43 Outcomes of interventions often not monitored or evaluated Poor timing (reactive versus proactive) High cost Description Unclear text, too many clicks to access the information, requested actions do not correspond to what the user requested. Most existing CDS and eCQM systems and their knowledge bases have limited portability. Alerts are often not updated properly. The lack of standardization and poor versioning causes divergent CDS and eCQM implementations. Clinicians do not get feedback on their decisions. Quality teams include analysts with a mission to evaluate and improve care quality. CDS teams include technical implementers with a mission to develop and implement functionality. Potential Solutions Conduct usability testing. Clinician self-assessment of delivered care quality is often higher than their true performance. Provide feedback on performance. It has been shown that feedback on performance lowers canceling of alerts by junior-level physicians.42 Improve validation strategies. High number of false positive results causes alert fatigue and mistrust of quality measures. Fee-for-service reimbursement models are still the predominant form of US healthcare reimbursement. Changes caused by quality improvement interventions are often not analyzed properly, thereby limiting opportunities for learning and continuous improvement. CDS often appears after the user has already made a decision. Feedback from the QM can also be delayed and may be delivered months after the fact. Implementing CDS and eCQM capabilities is oftentimes difficult and costly, with the need for highly skilled personnel. Employ standard-based approaches. Develop and evaluate technical approaches for integrating CDS and eCQM. Pursue efficient integration of quality and CDS teams. Align financial incentives with quality and outcomes (eg, via pay-forperformance). Improve outcome evaluation. Improve the timing for presenting feedback within the user's workflow. Increase interoperability and collaboration to efficiently share CDS and eCQM capabilities (eg, both within and across institutions). 10 1.5 References 1. Landrigan CP, Parry GJ, Bones CB, Hackbarth AD, Goldmann DA, Sharek PJ. Temporal trends in rates of patient harm resulting from medical care. N Engl J Med. 2010;363(22):2124-2134. doi:10.1056/NEJMsa1004404. 2. James BC, Savitz LA. How Intermountain trimmed health care costs through robust quality improvement efforts. Health Aff (Millwood). 2011;30(6):1185-1191. doi:10.1377/hlthaff.2011.0358. 3. Lobach D, Sanders GD, Bright TJ, et al. Enabling health care decisionmaking through clinical decision support and knowledge management. Evid Rep Technol Assess (Full Rep). 2012;(203):1-784. 4. Kawamoto K, Jacobs J, Welch BM, et al. Clinical information system services and capabilities desired for scalable, standards-based, service-oriented decision support: consensus assessment of the Health Level 7 clinical decision support Work Group. AMIA Annu Symp Proc. 2012;2012:446-455. 5. OpenCDS Home. http://www.opencds.org/. Accessed January 30, 2014. 6. Kawamoto K, Shields D, Del Fiol G. OpenCDS: enabling clinical decision support at scale through open-source, standards-based software and resources. AMIA poster. 2011:1830. 7. Blumenthal D, Tavenner M. The "meaningful use" regulation for electronic health records. N Engl J Med. 2010;363(6):501-504. doi:10.1056/NEJMp1006114. 8. Meyer GS, Nelson EC, Pryor DB, et al. More quality measures versus measuring what matters: a call for balance and parsimony. BMJ Qual Saf. 2012;21(11):964-968. doi:10.1136/bmjqs-2012-001081. 9. Silow-Carroll S, Edwards JN, Rodin D. Using electronic health records to improve quality and efficiency: the experiences of leading hospitals. Issue brief (Commonwealth Fund). 10. Garrido T, Kumar S, Lekas J, et al. e-Measures: insight into the challenges and opportunities of automating publicly reported quality measures. J Am Med Inform Assoc. 2014;21(1):181-184. doi:10.1136/amiajnl-2013-001789. 11. University HealthSystem Consortium (UHC). https://www.uhc.edu/. Accessed January 30, 2014. 12. University of Utah Health Care - Salt Lake City, Utah. http://healthcare.utah.edu/about/. Accessed January 30, 2014. 13. Blumenthal D. Launching HITECH. N Engl J Med. 2010;362(5):382-385. doi:10.1056/NEJMp0912825. 11 14. Joseph AM. The stage 3 meaningful use preliminary recommendations: concerns are being raised. MLO Med Lab Obs. 2013;45(7):64. 15. Dolin RH. Accuracy of electronically reported "meaningful use" clinical quality measures. Ann Intern Med. 2013;159(1):73. doi:10.7326/0003-4819-159-1-20130702000017. 16. Epic: Meaningful Use Stage 2 Certification Details. https://www.epic.com/software-certification.php. Accessed February 3, 2014. 17. National Research Council. To Err Is Human: Building A Safer Health System Institute of Medicine. Washington, DC: The National Academies Press; 2000. http://www.iom.edu/Reports/1999/to-err-is-human-building-a-safer-health-system.aspx. Accessed February 26, 2014. 18. National Research Council, Committee on Quality of Healthcare in America, Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: The National Academies Press, D.C.: NATIONAL ACADEMY PRESS; 2001. http://www.nap.edu/catalog.php?record_id=10027. Accessed February 26, 2014. 19. Marcilly R, Ammenwerth E, Roehrer E, Pelayo S, Vasseur F, Beuscart-Zéphir MC. Usability flaws in medication alerting systems: impact on usage and work system. Yearb Med Inform. 2015;10(1):55-67. doi:10.15265/IY-2015-006. 20. Kern LM, Malhotra S, Barrón Y, et al. Accuracy of electronically reported "meaningful use" clinical quality measures: a cross-sectional study. Ann Intern Med. 2013;158(2):77-83. doi:10.7326/0003-4819-158-2-201301150-00001. 21. Lobach DF, Kawamoto K, Anstrom KJ, et al. Proactive population health management in the context of a regional health information exchange using standardsbased decision support. AMIA Annu Symp Proc. January 2007:473-477. 22. Haggstrom DA, Saleem JJ, Militello LG, Arbuckle N, Flanagan M, Doebbeling BN. Examining the relationship between clinical decision support and performance measurement. AMIA Annu Symp Proc. 2009;2009:223-227. 23. Weber V, Bloom F, Pierdon S, Wood C. Employing the electronic health record to improve diabetes care: a multifaceted intervention in an integrated delivery system. J Gen Intern Med. 2008;23(4):379-382. doi:10.1007/s11606-007-0439-2. 24. Cleveringa FGW, Gorter KJ, van den Donk M, van Gijsel J, Rutten GEHM. Computerized decision support systems in primary care for type 2 diabetes patients only improve patients' outcomes when combined with feedback on performance and case management: a systematic review. Diabetes Technol Ther. 2013;15(2):180-192. doi:10.1089/dia.2012.0201. 25. Konrad R, Tulu B, Lawley M. Monitoring adherence to evidence-based practices: 12 a method to utilize HL7 messages from hospital information systems. Appl Clin Inform. 2013;4(1):126-143. doi:10.4338/ACI-2012-06-RA-0026. 26. Pathak J, Bailey KR, Beebe CE, et al. Normalization and standardization of electronic health records for high-throughput phenotyping: the SHARPn consortium. J Am Med Inform Assoc. 2013;20(e2):e341-8. doi:10.1136/amiajnl-2013-001939. 27. Li D, Endle CM, Murthy S, et al. Modeling and executing electronic health records driven phenotyping algorithms using the NQF Quality Data Model and JBoss® Drools Engine. AMIA Annu Symp Proc. 2012;2012:532-541. 28. Thompson WK, Rasmussen L, Pacheco JA, et al. An evaluation of the NQF quality data model for representing electronic health record driven phenotyping algorithms. AMIA Annu Symp Proc. 2012;2012:911-920. 29. Raja AS, Gupta A, Ip IK, Mills AM, Khorasani R. The use of decision support to measure documented adherence to a national imaging quality measure. Acad Radiol. 2014;21(3):378-383. doi:10.1016/j.acra.2013.10.017. 30. Lakshminarayan K, Rostambeigi N, Fuller CC, Peacock JM, Tsai AW. Impact of an electronic medical record-based clinical decision support tool for dysphagia screening on care quality. Stroke. 2012;43(12):3399-3401. doi:10.1161/STROKEAHA.112.662536. 31. Goldstein MK, Tu SW, Martins S, et al. Automating performance measures and clinical practice guidelines: differences and complementarities. AMIA Annu Symp Proc. 2014:37-38. 32. Richesson RL, Hammond WE, Nahm M, et al. Electronic health records based phenotyping in next-generation clinical trials: a perspective from the NIH Health Care Systems Collaboratory. J Am Med Inform Assoc. 2013;20(e2):e226-31. doi:10.1136/amiajnl-2013-001926. 33. Persell SD, Kaiser D, Dolan NC, et al. Changes in performance after implementation of a multifaceted electronic-health-record-based quality improvement system. Med Care. 2011;49(2):117-125. doi:10.1097/MLR.0b013e318202913d. 34. Shelton JB, Skolarus TA, Ordin D, et al. Validating electronic cancer quality measures at Veterans Health Administration. Am J Manag Care. 2014;20(12):1041-1047. 35. Richesson RL, Smerek M. Electronic health records-based phenotyping. NIH Health Care Systems Research Collaboratory. http://sites.duke.edu/rethinkingclinicaltrials/ehr-phenotyping/. Published 2014. Accessed July 22, 2015. 36. Brown SH, Elkin PL, Rosenbloom ST, Fielstein E, Speroff T. eQuality for all: extending automated quality measurement of free text clinical narratives. AMIA Annu Symp Proc. January 2008:71-75. 13 37. Linder JA, Kaleba EO, Kmetik KS. Using electronic health records to measure physician performance for acute conditions in primary care: empirical evaluation of the community-acquired pneumonia clinical quality measure set. Med Care. 2009;47(2):208216. doi:10.1097/MLR.0b013e318189375f. 38. Pakhomov S, Bjornsen S, Hanson P, Smith S. Quality performance measurement using the text of electronic medical records. Med Decis Making. 2008;28(4):462-470. doi:10.1177/0272989X08315253. 39. Newton KM, Peissig PL, Kho AN, et al. Validation of electronic medical recordbased phenotyping algorithms: results and lessons learned from the eMERGE network. J Am Med Inform Assoc. 2013;20(e1):e147-54. doi:10.1136/amiajnl-2012-000896. 40. Osheroff JA, Teich JM, Middleton B, Steen EB, Wright A, Detmer DE. A roadmap for national action on clinical decision support. J Am Med Inform Assoc. 2007;14(2):141-145. doi:10.1197/jamia.M2334. 41. Cabana MD, Rand CS, Powe NR, et al. Why don't physicians follow clinical practice guidelines? A framework for improvement. JAMA. 1999;282(15):1458-1465. 42. Redwood S, Ngwenya NB, Hodson J, Ferner RE, Coleman JJ. Effects of a computerized feedback intervention on safety performance by junior doctors: results from a randomized mixed method study. BMC Med Inform Decis Mak. 2013;13(1):63. doi:10.1186/1472-6947-13-63. 43. Petersen LA, Woodard LD, Urech T, Daw C, Sookanan S. Does pay-forperformance improve the quality of health care? Ann Intern Med. 2006;145(4):265-272. CHAPTER 2 CHALLENGES AND RECOMMENDATIONS FOR INTEGRATION OF CLINICAL DECISION SUPPORT AND ELECTRONIC CLINICAL QUALITY MEASUREMENT: INSIGHTS FROM DOMAIN EXPERTS 2.1 Abstract Objective of this study was to assess barriers and develop recommendations for the integration of clinical decision support (CDS) and electronic clinical quality measurement (eCQM). Leading experts in CDS and eCQM were recruited using targeted invitations and an open solicitation on listservs for professional national informatics organizations. Through semistructured in-depth and critical incident interviews using online meeting software, we obtained stakeholders' insights about CDS and eCQM integration, with a focus on key differences and similarities between CDS and eCQM, benefits and barriers of integration, and potential solutions. Fifteen experts were recruited, including executives and other leaders from academia, healthcare organizations, government, consulting companies, and commercial Health IT vendors. The experts identified multiple barriers to the integration of CDS and eCQM and offered advice for addressing those barriers, which the research team 15 synthesized into 10 recommendations. In particular, experts suggested improving the availability and adoption of standards, improving the approach to developing clinical practice guidelines and eCQM specifications, addressing cultural and structural differences between CDS and eCQM teams, and, finally, aligning financial reimbursement models with quality of care. Integration of CDS and eCQM will likely require substantial effort including developing technical capabilities and changing organizational structures and cultures to align CDS and eCQM. Integrating CDS and eCQM will require addressing several barriers. We anticipate that the expert insights elucidated in this study will facilitate CDS and eCQM integration and ultimately improvements in care quality and value. 2.2 Background Clinical decision support (CDS) systems and electronic clinical quality measurement (eCQM) are 2 important computer-based strategies aimed at improving the quality of healthcare.1 For the purposes of this study, we define CDS as the provision of pertinent knowledge and person-specific information to clinical decision makers to enhance health and healthcare.2 Examples of CDS tools include alerts, order sets, care plans, protocols, documentation templates and tools, relevant data summaries, and dashboards. Such CDS tools can help clinicians provide evidence-based care for a specific individual or for a population of patients.3 In turn, we define eCQM as the measurement and tracking of the quality of healthcare services using electronic data. Clinical quality measurement results are used in 16 reports and feedback on clinician performance, accreditation reviews and institutional performance metrics. Clinical quality measurement (QM) is traditionally conducted using manual chart abstraction, but this domain is transitioning towards electronic data extraction. In conjunction with CDS, or on its own, eCQM can help to improve quality by providing feedback to relevant stakeholders.4 CDS and eCQM fundamentally address the same issue of identifying patients who should receive particular health or administrative interventions and determining whether they have received that intervention.5-7 Coordination of vision, processes, and technologies, or integration, of CDS and eCQM domains has the potential to improve healthcare value.8-10 CDS can facilitate the collection of data elements needed for the quality measures, and eCQM results can support iterative, data-driven refinement of the CDS. Other potential positive outcomes of integrating CDS and eCQM include reducing duplication of effort and minimizing inconsistencies in guidance recommendations. Recognizing the importance of such integration, groups including the US federal government and the Institute of Medicine (IOM) are advising healthcare providers to tighten the feedback loop between CDS and eCQM. 11,12 An improved ability to establish such a virtuous feedback loop between quality improvement and continuous performance measurement is an important enabler for becoming a Learning Health Care System. Notably, the US Office of the National Coordinator (ONC) for Health Information Technology (IT) and the Centers for Medicare & Medicaid Services (CMS) have sponsored the public-private Clinical Quality Framework (CQF) initiative to harmonize health IT standards for CDS and eCQM to facilitate their integrated implementation.13 Several standards and technological solutions have been suggested to enable integration 17 of CDS and eCQM.14-19 Despite this recognition of the importance of integrating CDS and eCQM, fully integrated quality improvement approaches are still rarely used and only sporadically reported in the literature. For example, only 3 out of 160 randomized clinical trials included in a review of CDS systems by Lobach et al. were accompanied by periodic performance feedback.3 Additionally, when an integrated approach is used, it is often incomplete. For example, coordinated CDS ad eCQM efforts based on commercial EHR systems oftentimes use different tools for CDS and for eCQM implementations.10,20 Moreover, family physicians report a lack of quality improvement infrastructure to codeliver CDS and eCQM in their practices.21 Finally, aspects of organizational culture and structure that inhibit integration of CDS and eCQM are poorly described in the literature. In summary, there is a need for research to better characterize how CDS and eCQM can be better integrated to improve care. To address this need, we sought insight from experts in the field to characterize the current state of CDS and eCQM integration and to identify potential approaches for advancing such integration moving forward. 2.2.1 Objective This study aimed to explore the beliefs and perceptions regarding the integration of CDS and eCQM functionality and activities within healthcare organizations, using qualitative methods that engage subject matter experts (SMEs). Our specific objectives were to (1) describe similarities and differences in CDS and eCQM implementation and use, (2) describe potential benefits of the integration of CDS and eCQM, (3) describe technical and cultural barriers to integrating CDS and eCQM, and (4) formulate 18 recommendations for CDS-eCQM integration. 2.3 Materials and Methods 2.3.1 Study Design A qualitative study was conducted using in-depth semistructured interviews with subject matter experts (SMEs). The critical incident technique was used during the interview process to identify components related to the key challenges in CDS and eCQM integration. 2.3.2 Research Team The study was conducted by a multidisciplinary research team with experience in CDS, eCQM, clinical and public health informatics, standards-based interoperability, qualitative methods, cognitive task analysis, biostatistics, and information technology. 2.3.3 Subjects SMEs were enrolled through an open invitation for participation made via email to relevant email listservs sponsored by the American Medical Informatics Association (AMIA) CDS and Implementation work groups, the Health Level 7 (HL7) Clinical Quality Information and CDS work groups, and the Clinical Quality Framework (CQF) initiative.13 In order to maximize the representation of relevant expert insights, invitations were also sent to individuals identified as being key SMEs based on literature review and by the study personnel. Participation was open to all interested professionals in the field who had both of the following qualifications: 19  Experience developing or using a quality measurement system, and  Experience developing or using CDS interventions. Thirty individuals responded initially and 15 individuals decided to proceed with the interview. It has been previously shown that 12-13 interviews could be sufficient to gather a majority of insights.22,23 Thus, we did not send any new invitations after conducting the 15 interviews. At the beginning of each interview, a verbal consent was obtained for participating in the study, recording and transcribing the interview, and including participants' names in publications. A financial incentive ($40) was offered to the participants for their time, but some participants declined. The study was approved by the University of Utah institutional review board (IRB) (Protocol # 00077948). 2.3.4 Interviews One-hour in-depth semistructured interviews included 3 parts: (1) questions about the participants' background and experience with CDS and eCQM, (2) critical incident questions, and (3) general questions about integration of CDS and eCQM. We did not include a prespecified and constrained definition of the "integration" construct in the interview script in order to provide the respondent with flexibility to discuss any aspects of potential integration that they felt were important. Questions about the participants' background and experience concerned their current organizational role, the type of organization, whether they had encountered CDS or eCQM first in their career, whether they had more experience with CDS or eCQM, and 20 the degree of integration between CDS and eCQM in their organization on a 1 to 10 scale. The critical incident technique allows collecting rich data from the respondents' perspective and in their own words without forcing them into any given framework. The critical incident technique allows identifying even rare events that might be missed by other methods that only focus on common and everyday events.24 The critical incident methodology was adapted from cognitive work analysis methods described by Crandall et al. where a 4-phase format was used: (1) incident identification, (2) timeline verification, (3) deepening, and (4) ‘what-if' queries. First, we asked the interviewee to identify a specific project where he/she used both CDS and eCQM. Second, we asked the participant to provide a time-based description of the sequence of tasks in order to create an explicit timeline. Third, we asked a set of more specific questions to identify and verify project goals, social context, organizational issues, challenges, and decision points. Finally, a few "what-if" questions were posed to explore what could have been done differently under critical relevant conditions. General questions about integration of CDS and eCQM included questions about similarities and differences between CDS and eCQM implementation and use, technical and nontechnical barriers to the integration of CDS and eCQM, and recommendations to reach a higher degree of integration. Interviews were recorded and transcribed. All responses were used for the analysis. 21 2.3.5 Data Analysis The interviews, including answers to critical incident questions and general questions about integration of CDS and eCQM, were analyzed using content analysis techniques described by Patton and Graneheim et al.25,26 Transcript analysis began with one author (PK) identifying responses as relevant or not relevant using 5 predefined areas of interest as general categories. The following taxonomy was chosen by study personnel for its pragmatic utility for understanding why CDS-eCQM integration is desirable, why such integration is still quite limited, and how integration could be achieved:  similarities in CDS and eCQM implementation and use,  differences in CDS and eCQM implementation and use,  benefits to the integration of CDS and eCQM,  technical and nontechnical barriers to the integration of CDS and eCQM, and  recommendations for the integration of CDS and eCQM. Relevant responses were reviewed at the paragraph level by a 3 person multidisciplinary research team with qualitative research experience. The team converted responses to condensed descriptions that preserved the meaning of the response. Then, related condensed descriptions and corresponding responses were summarized into constructs. The research team discussion was iterative, with condensed descriptions discussed, reviewed, and then reviewed again until no new constructs emerged within each area. Constructs were then aggregated into thematic statements within each area in order to elucidate the "gist" of the content. The thematic statements were then reviewed by the research team. Insights were not tied to any specific individual study participant. 22 Internal participant identification number is shown in parentheses after each quote. 2.4 Results 2.4.1 Participants Fifteen SMEs with diverse backgrounds and organizational experience participated, including executives and other leaders from academia, government, healthcare provider organizations, consulting companies, and CDS and electronic health record (EHR) system vendors (Table 2.1). Eleven SMEs first encountered CDS in their career. Among these 11 participants who encountered CDS first, 5 remained currently more experienced in CDS, 1 is now more experienced in QM, and 5 reported being equally experienced in both domains. In contrast, among the 4 participants who encountered QM first in their career, 3 remained currently more experienced with QM than CDS, while 1 of the 4 is now more experienced with CDS than QM. When asked to report on current level of integration between CDS and eCQM on a scale of 1 to 10, SMEs varied widely in their responses. SMEs reported that optimal level of integration between CDS and eCQM has not been reached yet, even in most advanced healthcare systems. Three participants refrained from answering this question. One participant felt this question was only applicable to provider organizations. Critical incident stories covered a wide range of use cases in different settings, including inpatient, outpatient, and emergency departments. Stories were told from different perspectives, including large healthcare systems and small practices, as well as consulting and vendor companies. The majority of responders described quality 23 improvement projects where both CDS and eCQM were used. For example, some projects were aimed at improving previsit planning reports for pediatric patients, or creating checklists to reduce cancellation rates at a cardiac surgery service. Goals of other projects included reducing hypoglycemic episodes, reducing catheter associated urinary tract infections, improving blood pressure control and diabetes management, improving timeliness of thromboembolism prophylaxis, prescribing warfarin for atrial fibrillation for patients that were high risk of stroke, and improving pneumonia management in emergency department. Some projects focused on implementation of eCQMs, such as for depression management, while other projects aimed to improve Medicare-related quality measures or the quality of clinical problem lists. Some of these projects succeeded and some failed according to the respondents' perceptions. Over 250 quotations were extracted from the interview transcripts. The comments were summarized into condensed descriptions that describe similarities (n = 6 descriptions), differences (n = 21), benefits (n = 13), and barriers (n = 55). Additionally, comments related to potential solutions were summarized into 71 condensed descriptions, and then further summarized into 10 actionable recommendations. Sample responses, condensed descriptions, resulting constructs, and summarized thematic statements are presented in table format. 2.4.2 Similarities and Differences in CDS and eCQM Practice All SMEs noted that CDS and eCQM are similar but also different in important aspects. Key similarities included the common goal of clinical quality improvement, the use of similar patient data for calculation, scalability requirements, and the need for logic 24 (Table 2.2a). Key differences in CDS and eCQM implementation and use, included differences in the level of analysis (ie, patient vs. population), whether eligible patients are defined strictly or loosely, and the culture and motivation of implementing teams (Table 2.2b). 2.4.3 Potential Benefits of Integration of CDS and eCQM SMEs identified many potential benefits of integrating CDS and eCQM, including more effective quality improvement, better prioritization, and higher consistency of quality improvement interventions, reduced cost of implementation and financial benefits for the healthcare organization (Table 2.3). However, some SMEs were more optimistic than others about the potential to achieve those benefits. One participant also pointed that costs of integration of CDS and eCQM might outweigh benefits. 2.4.4 Barriers for Integrating CDS and eCQM SMEs identified many technical and nontechnical barriers to implementing CDS and eCQM (Table 2.4). Five themes concerning barriers to integration were identified, including limited availability and adoption of standards and technological solutions, problems with authoring guidelines, different organizational cultural and structural barriers, and financial barriers. 2.4.5 Recommendations for Integration of CDS and eCQM The SMEs noted that integration of CDS and eCQM will require contributions from many stakeholders, including standards developers, EHR vendors, CDS vendors, 25 eCQM vendors, CDS and eCQM implementers, healthcare executives, healthcare providers, guideline and quality measure authoring agencies, and the payer community. To accelerate integration of CDS and eCQM, 10 actionable recommendations were generated based on the insights of SMEs. The recommendations are grouped by stakeholder type.  Standards Developers o Develop and improve harmonized standards, including standard terminologies, to represent executable logic, clinical data, and metadata that address both CDS and eCQM use cases.  EHR Vendors o Develop EHR capabilities to coimplement CDS and eCQM. Provide ways to expose the data in a standard and secure way that can be used across both CDS and eCQM in a common manner.  Technology Developers and Implementers o Use existing and emerging harmonized standards and technical approaches (eg, libraries of reusable elements and modules, data access standards) to implement and share CDS and eCQM knowledge across institutions. o Use a sustainable and robust maintenance strategy that includes versioning, documentation, validation, and updates to account for asynchronous changes in both CDS and eCQM specifications. o Use strategies for selecting evidence-based interventions and reconciling differences between CDS and eCQM definitions and requirements to account for multiple competing recommendations. 26 o Engage all relevant stakeholders and iteratively develop common, streamlined solutions to account for the multidisciplinary nature of CDSeCQM projects.  Healthcare Executives and Organizational Leadership o Cross-train individuals who can serve as liaisons, develop coordinated governance, and create a culture of collaboration instead of competition to improve communication between CDS and eCQM groups.  Guideline/Specification Authoring Groups o Specify corresponding CDS when developing eCQMs, and vice versa. o For eCQMs, use data elements already available in the EHR at the time of the encounter (eg, clinical data collected as a part of routine workflow) rather than depending on new documentation or billing data captured after the encounter.  Payers and Government Agencies o Use financial incentives that promote CDS and eCQM, such as a "pay for value" reimbursement model.27 2.5 Discussion Based on the insights from SMEs, CDS and eCQM integration could promote clinical improvement, increase consistency of quality improvement interventions, and reduce cost of implementation. However, they also described challenges that must be overcome before integration and subsequent efficiencies can be realized. These challenges include divergent standards and technical approaches, uncoordinated 27 specification authoring, lack of cultural and structural integration between CDS and eCQM teams in healthcare organizations, and misaligned financial incentives. CDS and eCQM have historically belonged to 2 different worlds within the healthcare enterprise. According to SMEs, CDS and eCQM professionals have different professional cultures and, oftentimes, have limited communication between each other. While many of these challenges are already currently being addressed, others remain outstanding and likely not fully appreciated by the stakeholders involved. To accelerate integration of CDS and eCQM, 10 actionable recommendations were synthesized based on the insights of SMEs in the fields of both CDS and quality measurement. In particular, the experts suggested improving availability and adoption of standards, changing the approach to CDS guidelines and eCQM specification development, addressing cultural and structural differences between CDS and eCQM teams, and aligning financial reimbursement models with quality of care. Our study is different in scope and purpose from previously published manuscripts related to complementarities between CDS and eCQM. This study not only confirms and expands on the findings from previous studies with regard to similarities and differences between CDS and eCQM, our paper also describes challenges and provides recommendations for the integration.5-7 Goldstein et al. described similarities and differences between CDS and eCQM with regard to cohort definitions, knowledge modeling, workflow integration, use of data, and output structures,5 while Brown et al. compared CDS and eCQM in terms of data sources, analytic methods, units of analysis, delivery timing, intended users, and recommendations.7 Haggstrom et al. focused on how the relationship between CDS and eCQM is perceived by relevant stakeholders. As in 28 these prior studies, this study found that CDS and eCQM are similar with regard to data sources but different in terms of analytical methods, units of analysis, delivery timing, cohort definitions, and intended users. The current study additionally found that CDS and eCQM differ in the professional cultures of the teams that implement these capabilities. The current study provides more detail compared to prior studies, with the inclusion of direct quotes from experts in the field to illustrate the many nuances of the complex relationship between CDS and eCQM. Furthermore, use of the critical incident technique allowed us to identify rare events such as conflicts and difficulties that may have not been reported otherwise. The SMEs identified many potential benefits to integrate CDS and eCQM, including reducing costs, increasing alignment between CDS and eCQM implementations, and avoiding inefficient, duplicative efforts in each area. Other potential benefits identified include the coupling of CDS with automated performance feedback, improved quality, enhanced organizational efficiency, and financial benefits. Taken together, the integration of CDS and eCQM can help transform healthcare organizations into Learning Healthcare Systems with effective feedback loops for quality improvement.11 However, as indicated by the wide variations in the provider responses about the degree of integration in their own organizations, there are large differences in the progress of organizations towards this goal. Furthermore, this variation could be partially explained by differences in participants' beliefs about what an ideal integration may entail. Several efforts are underway to address the challenges to the integration of CDS and eCQM identified by the SMEs. For example, the CQF initiative sponsored by the 29 ONC and CMS is developing harmonized standards for data representation, metadata, and executable logic to facilitate coimplementation of CDS and eCQM.13 Moreover, while many EHRs currently have limited native capabilities for coimplementation of CDS and eCQM, an evolving app marketplace may enable external vendors to produce standards-based solutions that could be used for both CDS and eCQM.28 As for differences in professional culture, several promising projects are ongoing, including the development of knowledge centers in academic health systems that integrate CDS and quality measurement, such as the New York-Presbyterian Hospital's Value Institute and the Johns Hopkins Armstrong Institute for Patient Safety and Quality.29,30 While many challenges are already being addressed, others still need to be resolved. First, limited native EHR capabilities continue to be a problem, and EHR vendors are not necessarily prioritizing coimplementation of CDS and eCQM. Second, quality measure specifications generally do not include CDS guidance. More will need to be done with regard to the authoring of clinical guidelines and quality measures to facilitate the integration of CDS and eCQM. Third, cultural differences between teams and lack of coordinated governance, structure, and processes largely remain to be addressed. Indeed, SMEs mentioned that nontechnical barriers to CDS-eCQM integration are probably more important than the technical ones. Integration will need to be achieved at different levels, including for standards integration, IT infrastructure integration, specification authoring integration, and organizational and cultural integration. If the CDS and eCQM stakeholders are able to address the described challenges, the vision of integrated and efficient quality improvement framework may be accomplished. We believe that the 10 recommendations provided in this paper can 30 facilitate this integration. Additionally, there may be ‘game-changers' that facilitate this transition, including a focus on payment for value and the sponsorship of integration efforts by CMS, which can drive healthcare policy in the United States. Our study has several potential limitations. First, as a qualitative study, the results may be influenced by the researchers' personal biases or by the phrasing of the interview questions. However, we used robust content analysis methodologies to help ensure the reliability of our findings.25,26 We also include the interview script in the manuscript to make the questions available to the readers. Second, the inclusion and analysis of only 15 interviews may limit generalizability. Even though it has been previously shown that 1213 interviews could be sufficient to gather a majority of insights,22,23 more interviews may have provided more insights in this particular study. Third, the self-selection recruitment strategy may have biased the included SMEs to those who strongly agree or disagree that CDS and eCQM should be integrated. However, the resulting sample included SMEs representing a broad spectrum of healthcare professionals from many geographical regions and with different past experiences, enhancing our ability to describe the breadth of issues. Forth, our study does not include estimates of costs of CDS-eCQM integration. However, we felt that the qualitative nature of our research would not allow us to estimate whether benefits of integration outweigh cost. Thus, we decided to leave this topic out of scope. We therefore believe that our conclusions remain generalizable. This study identified several areas where further research is needed to overcome remaining barriers to CDS and eCQM integration. In particular, there is a need to investigate strategies for mitigating the cultural differences and improving 31 communication between CDS and eCQM teams. In addition, there is a need to track progress and to evaluate the benefits and costs of enhanced CDS and eCQM integration through shared governance, infrastructure, and technical approaches. 2.6 Conclusion This study improves our understanding of the challenges and opportunities for integrating CDS and eCQM. The findings could serve as a useful guide for ongoing activities in CDS-eCQM integration. Integration efforts will need to address many challenges, including those related to standards, technology, specification authoring, organization culture and structure, and financial incentives. While all the experts in the study agreed that integration of CDS and eCQM is important, the SMEs differed in their viewpoints on the feasibility of the integration in the near future. Integration of CDS and eCQM will likely require substantial effort for developing the necessary technical and organizational capabilities. 32 Table 2.1. Participants Name of participant Howard Bregman, MD, MS Nathaniel Weiner, MS Clement J. McDonald, MD Role/title Name of Organization Epic, WI Type of organization EHR vendor Co-Founder, Chief Operating Officer Scientific Director Avhana Health, MD CDS vendor US National Library of Medicine, MD Samson Tu, MS Senior Research Scientist Stanford University, CA Art Wallace, MD, PhD Chief, Anesthesia Service Adam Wright, PhD Associate Professor of Medicine San Francisco Veterans Affair Medical Center, CA Harvard Medical School, MA Keith Marsolo, PhD Associate Professor Keith F. Woeltje, MD, PhD Vice president, Chief medical informatics officer Hojjat Salmasian, MD, MPH, PhD Program Director of Research Science Government, academia*, healthcare provider* Government, healthcare provider Government, healthcare provider Academia, healthcare provider Academia, healthcare provider Academia, healthcare provider Healthcare provider Joseph Kunisch PhD, RN-BC, CPHQ Benjamin Brown, MRCGP, MSc Jerome A. Osheroff, MD Michelle Currie, RN, MSN, CPHQ, CPHIMS William Salomon, MD, MS MPH Enterprise Director for Clinical Quality Informatics- Regulatory Performance General Practitioner and Research Training Fellow Founder/Principal James McCormack, PhD Instructor - Health IT Project Management Director, Clinical Informatics Founder and Healthcare Solution Architect Senior Medical Informatician * - previous employment Cincinnati Children's Hospital Medical Center, OH BJC HealthCare, MO Value Institute, NewYorkPresbyterian Hospital, NY Memorial Hermann Hospital System, TX University of Manchester, UK TMIT Consulting, LLC, TX Savant Solutions4HIT, LLC, CA Clinical Metrics, Limited liability company, ME Oregon Health & Science University, OR Healthcare provider Healthcare provider Consultant Consultant Consultant Consultant 33 Table 2.2. Similarities and differences between CDS and eCQM Construct Condensed Description Sample Responses 2a. Similarities between CDS and eCQM Theme: CDS and eCQM aim to improve healthcare quality. "The clinical purpose is generally similar, in both cases. My goal, building an eCQM or building CDS, is to improve care …" (14) Theme: CDS and eCQM are based on similar patient data. Common goal CDS and eCQM have the same purpose of healthcare quality improvement. "They're measuring the same thing, they're working on the same datasets, they are using electronic records, most of the time, or administrative data ..." (15) Dependence on Results of CDS and eCQM are only "They are both predicated on the data quality as good as the quality of the quality of the electronic data, so underlying data. they're only as good as the electronic data." (1) Theme: CDS and eCQM are automated approaches following similar logic and applied to large patient populations. Executable CDS and eCQM are defined by a "There's a little clinical reminder logic combination of logical expressions that says, 'Hey, please do X, Y, Z,' and value sets (eg, denominator and then you check to see how criteria, numerator criteria); often people did X, Y, Z. And you CDS and eCQM follow similar logic can bug people to the point where they'll actually change their behavior." (5) Machine CDS and eCQM require automation "The similarity between them is automation to be used at scale. that they're both obviously using technology to automate information." (11) Reliance on patient data CDS and eCQM rely on similar patient data. 34 Table 2.2 Continued Construct Condensed Description Sample Responses 2b. Differences between CDS and eCQM Theme: eCQM is more retrospective and population based, with more conservative population definitions compared to CDS. Being retrospective, eCQM could use claims data. CDS is prospective, individual-centered, relies on current data, and has inclusive population definitions. Focus of CDS is generally more prospectively "There's also a difference in the analysis oriented, presented in real-time during temporality of it. Decision support the patient visit, and focused on typically occurs in real time or changing clinician behavior and near real time, whereas quality collecting data; measurement is usually after the eCQM tends to be more retrospective fact, retrospective, looking back and is usually related to evaluation, over a lot larger periods of time in monitoring, and developing a strategy the clinical data." (1) to improve clinical quality. Data elements CDS usually relies on the EHR data; "… So how do you then run eCQM could rely both on EHR data decision support, when you're and on claims data, or even on essentially required to consider manually abstracted data; data that hasn't even been eCQM can rely on ‘future' data which recorded yet, right? … of course are not available when CDS is firing, the coded diagnosis is not going to such as lab results, procedures be generated for days to weeks completed, etc. after the clinical scenario that you're faced with. " (9) Level of CDS is usually calculated at the "I mean eventually when you're evaluation patient level; doing CDS, you're basically taking eCQM could be aggregated at the EMR and doing this at the different levels; patient level. Okay, if you're doing CDS is often triggered by a change in CQM you're doing this at the the patient data, such as a new population level." (4) problem; or by an action from the provider, such as opening the order entry dialog eCQM is usually run at periodic intervals, or on demand. Population CDS may have more loose definitions "Decision support is somewhat definitions since it is expected to cover all more crude in terms of how it's patients to whom the proposed applied. … There's a lot of effort definition might apply; that goes into defining the eCQM may have more strict population so that you're truly population definitions (with more measuring what's important. I exclusion criteria defined), to ensure don't know if that same level of appropriate comparisons over time or rigor yet exists on the decision between organizations and support side." (3) benchmarks, especially if it is related to financial incentives. 35 Table 2.2 Continued Construct Condensed Description Sample Responses 2b. Differences between CDS and eCQM Theme: CDS is context aware and should be integrated within clinical workflow, while eCQM is context independent. Context CDS is context aware; "I think in CDS, what's dependence eCQM is context independent; important is the context. … This CDS requires workflow integration, person has this role. This is which could be associated with higher when the alert should appear. implementation effort; Clinical quality measurement CDS might require clinician judgment. only looks from a logical perspective." (10) Visualization CDS and eCQM are usually presented "Hypothetically, in an ideal and differently given their different world, you just have to define presentation audiences and purposes; them once, and program all of CDS is often presented in textual form, the things once, and then just eg, as alerts, reminders, or smart have two different visualizations forms; for the data: one which happens eCQM is often presented in a table, at the point of care on a casegraph or dashboard. by-case basis and you want to send an alert out, and one which happens at population level on demand." (12) Theme: CDS and eCQM are implemented by different teams having different professional cultures and motivational factors. Professional CDS tends to be implemented by IT and "I suppose just different culture informatics teams; cultures, the people who do the eCQM tends to be implemented by quality measurement tend to be quality department specialists with more from a public health analytics, public health, or nursing background or a nursing backgrounds. background whereas the people who do the CDS tend to be from an informatics or IT background, and so they don't always know exactly how they will work together." (14) Motivators CDS efforts are often initiated from "… a lot of our quality within the healthcare institutions and measures for better or worse based on internal quality goals that can right now come from the federal be locally defined; government or from an eCQM requirements are often externally insurance company …" (14) regulated and incentivized, and evolve more slowly. EMR - Electronic Medical Record 36 Table 2.3. Benefits of an integrated approach to CDS and eCQM Construct Condensed Description Sample Responses Theme: Integration of CDS and eCQM will likely result in more effective quality improvement, better prioritization, and higher consistency of quality improvement interventions. Clinical More effective clinical improvement "It takes, on average, about 17 improvement and adoption of evidence-based care; years to get a doctor to implement a Facilitated implementation of the Level 1 standard of care. … quality improvement cycle, including Clinical decision support can be through baseline performance used to educate people about what measurement and continuous to do and speed up this very tracking; prolonged timeframe. The clinical Improved prioritization of quality decision support speeds up the improvement interventions; implementation of quality Improved documentation of improvement and then you can use contraindications, therapy, or the system to see how well people discussion with the patient. are doing with it." (5) Improved Improved consistency of quality "It seems unfair that we would have consistency improvement interventions and inconsistencies between our CDS recommendations. and our quality measures. I think like, we owe it to our users to harmonize those approaches." (14) Theme: Integration of CDS and eCQM will likely result in reduced cost of implementation and financial benefits for the healthcare organization. Improved Reduced implementation burden and "It makes the production time efficiency by shorter production time within and incredibly shorter since you're reducing time across healthcare systems; working from a common set of and cost of Reusing approaches between CDS concepts. You're basically creating implementation and eCQM; your clinical content with an aim of Improved data flow and data sharing doing CDS and quality within and between organizations, measurement. Same set of concepts; including commercial CDS and therefore you're not having to eCQM vendors; basically worry about compatibility More robust system, where it is of different sets of content - easier to fix errors. meaning the CDS and the quality measurement being based on different things." (4) Financial Eligibility for government incentives "The performance of an benefits and avoiding penalties; organization on eCQM, either an Opportunity to redirect eCQM organization or an individual level, funding to CDS development: there is probably tied to reimbursement is currently significant funding from some way. Or, if it's not tied today, the federal government for eCQM it's going to be tied in the future. So related efforts, and this funding organization would see a benefit to could be used to improve CDS as improving their scores in quality well. measurement. So therefore, they would want their CDS to be at least somewhat aligned with the quality measure performance." (9) 37 Table 2.4. Barriers to the integration of CDS and eCQM Construct Condensed Description Sample Quotes Theme: Poor standards availability and adoption complicate development of advanced systems. Poor data quality complicates data transformation and utilization. Incomplete Incomplete standards, leading to "… the standards don't support all terminology inconsistent implementations; of the use cases we would need…" and modeling Not all clinical use cases supported (6) standards by current standards. Poor standards Multiple unharmonized standards; "We've got automated processes to adoption Low standards adoption. compute the measures, but all of those activities are not yet standards-based." (6) Poor data Poor data quality inhibits integration "We don't have all the data that we quality between the systems. need in one system, it hasn't been validated …" (2) Theme: Currently existing technological solutions are suboptimal. Limited native Limited native EHR capabilities for "There's only so much you can do EHR coimplementation of CDS and with clinical decision support capabilities eCQM, especially the cases with without custom programming. … complex logic; There are things we can think of but Limited flexibility in EHR the EHR does not have the ability customization. …" (3) Performance Challenging optimization of "So, the performance issue of … issues algorithms, developed for individual how to efficiently convert it for patients, for thousands of patients at applying these inclusion/exclusion a time. criteria on the large cohort of the patients" (8) Diversity of Different EHRs and databases "Terminology changes; concepts platforms and implemented in different health change; standards for measurement unstable systems; change." environment Different software for CDS and "The issues of system integration in eCQM; so far as performance measures Fast pace of change in terminologies, and CDS are often built using standards, and EHR vendors. different infrastructures" (4) Workflow Invasive interventions; "We are not that good yet at issues Interruptive data collection for knowing when to show the CDS eCQM; workflow or even less good about Not user friendly interfaces; knowing when to show quality Over-alerting clinicians; measures in the workflow. Right CDS not optimized for population now, once a quarter, we send the management. quality report to your department chair and then they might meet with you and tell you what we're doing …" (14) Documentation, Different expectations for provision "… you really have to maintain maintenance of adequate documentation, CDS and I think that's one of the and versioning maintenance and versioning; harder problems with it. Medical Higher expectations for CDS for care changes ..." (5) timely roll-out and tracking of updates. 38 Table 2.4 Continued Construct Condensed Description Theme: eCQM and CDS content can diverge. eCQM eCQMs designed without thinking of guidelines do CDS; not always eCQMs lacking a clear CDS support CDS counterpart; While CDS has to rely on currently available data, eCQM might need to use data that become available later. Conservative eCQMs, particularly those used for nature of many compensation, may be more quality conservative than the care guidelines measures upon which CDS is based. Sample Quotes "When CQMs are developed by the committees, the expert panels that do them, and the stewarding organizations, they're not thinking in terms of CDS. …" (9) "… But pay for performance measurement uses a target of 150 over 90, because the target of 140 over 90 is too difficult to reach: they don't want to interfere with payment." (15) Uncoordinated Uncoordinated updating of quality "... measures are defined by updating of measures and CDS guidelines, Meaningful Use, by National CDS and conducted on different timeframes; Quality Forum … . And the CDS eCQM Prevalence of locally defined CDS may be based on recommendations specifications interventions, as opposed to quality from professional societies … ." measures defined at a national level. (8) Theme: Organizational and cultural factors inhibit integration of CDS and eCQM. Perception as Perception of CDS and eCQM as two "People just don't view these separate different approaches. things as the same" (6) domains Cultural Cultural differences between CDS and "So you have two ways of viewing differences eCQM teams; the world, different terminology, between teams Difficulty communicating between IT, and just different ways of talking quality and other stakeholders with and things like that." (6) different worldviews. Lack of Independent CDS and eCQM teams "The CDS developers are creating coordinated with limited processes for tools that organizations use for governance, coordination; CDS purposes. And they basically structure, and No organizational structure and work to refine those tools and add processes governance for unified CDS and new functionality. ... Whereas the eCQM; CQM team is essentially trying to Hard to get right people at the same keep up with the regulatory table; requirements. And the end result is EHR vendors have separate teams they don't have a lot of working on CDS and eCQM. intersection …" (9) Competing Independent groups with competing "I see a lot where a certain group interests interests, each with desire to be the wants to be the group that solves primary stakeholder in terms of the problem, so that they can decision making, resourcing, and either get the recognition, or recognition; substantiate their position … ." Competing priorities. (11) 39 Table 2.4 Continued Construct Not seeing a rational for integration Condensed Description Preference for the tools people are most familiar with; Not seeing a benefit of integrating CDS and eCQM. Sample Quotes "… tendency to believe in your tool. If you do CDS, that's because you think CDS is better, if you focus on quality measures, probably you think quality measurements are more population focused ..." (14) Inadequate Lack of informatics training of "… it may actually be that those resources and personnel; standards exist and we just training Limited IT resources. weren't aware of them, didn't know how to leverage them." (6) Theme: There is often no clear financial rationale to co-implement CDS and eCQM. No clear Limited financial or clinical incentive "… the perception is there's no financial for many providers to adopt eCQMdirect link between physicians incentive for based CDS, coupled with potentially clicking another thing and more providers extra work money coming into the practice or better outcomes for the patient." (13) Need for Limited funding for innovation "… That's the hardest: in any funding project it's getting the money." (5) 40 2.7 References 1. Shojania KG, McDonald KM, Owens DK. Evidence-Based Review Methodology for the Closing the Quality Gap Series (Vol. 1: Series Overview and Methodology). Rockville (MD): Agency for Healthcare Research and Quality (US); 2004. 2. Osheroff JA, Teich JM, Middleton B, Steen EB, Wright A, Detmer DE. A roadmap for national action on clinical decision support. J Am Med Inform Assoc. 2007;14(2):141-145. doi:10.1197/jamia.M2334. 3. Lobach D, Sanders GD, Bright TJ, et al. Enabling health care decisionmaking through clinical decision support and knowledge management. Evid Rep Technol Assess (Full Rep). 2012;(203):1-784. 4. Anderson KM, Marsh CA, Flemming AC, Isenstein H RJ. Quality Measurement Enabled by Health IT: Overview, Challenges, and Possibilities: An Environmental Snapshot. Rockville (MD): Agency for Healthcare Research and Quality (US); 2012. 5. Goldstein MK, Tu SW, Martins S, et al. Automating performance measures and clinical practice guidelines: differences and complementarities. AMIA Annu Symp Proc. 2014:37-38. 6. Haggstrom DA, Saleem JJ, Militello LG, Arbuckle N, Flanagan M, Doebbeling BN. Examining the relationship between clinical decision support and performance measurement. AMIA Annu Symp Proc. 2009;2009:223-227. 7. Brown B, Peek N, Buchan I. The case for conceptual and computable crossfertilization between audit and feedback and clinical decision support. Stud Health Technol Inform. 2015;216:419-423. 8. Cleveringa FGW, Gorter KJ, van den Donk M, van Gijsel J, Rutten GEHM. Computerized decision support systems in primary care for type 2 diabetes patients only improve patients' outcomes when combined with feedback on performance and case management: a systematic review. Diabetes Technol Ther. 2013;15(2):180-192. doi:10.1089/dia.2012.0201. 9. Weber V, Bloom F, Pierdon S, Wood C. Employing the electronic health record to improve diabetes care: a multifaceted intervention in an integrated delivery system. J Gen Intern Med. 2008;23(4):379-382. doi:10.1007/s11606-007-0439-2. 10. Persell SD, Kaiser D, Dolan NC, et al. Changes in performance after implementation of a multifaceted electronic-health-record-based quality improvement system. Med Care. 2011;49(2):117-125. doi:10.1097/MLR.0b013e318202913d. 11. Institute of Medicine. Leadership Commitments to Improve Value in Healthcare: Finding Common Ground. Washington, DC: The National Academies Press; 2009. 12. Blumenthal D, Tavenner M. The "meaningful use" regulation for electronic health 41 records. N Engl J Med. 2010;363(6):501-504. doi:10.1056/NEJMp1006114. 13. Kawamoto K, Hadley MJ, Oniki T, Skapik J. The clinical quality framework initiative to harmonize decision support and quality measurement standards: defined standards, pilot results, and moving beyond quality improvement. AMIA Annu Symp Proc. 2015:202-204. 14. Pathak J, Bailey KR, Beebe CE, et al. Normalization and standardization of electronic health records for high-throughput phenotyping: the SHARPn consortium. J Am Med Inform Assoc. 2013;20(e2):e341-8. doi:10.1136/amiajnl-2013-001939. 15. Kukhareva P V, Kawamoto K, Shields DE, et al. Clinical decision support-based quality measurement (CDS-QM) framework: prototype implementation, evaluation, and future directions. AMIA Annu Symp Proc. 2014;2014:825-834. 16. Li D, Endle CM, Murthy S, et al. Modeling and executing electronic health records driven phenotyping algorithms using the NQF Quality Data Model and JBoss® Drools Engine. AMIA Annu Symp Proc. 2012;2012:532-541. 17. Thompson WK, Rasmussen L, Pacheco JA, et al. An evaluation of the NQF quality data model for representing electronic health record driven phenotyping algorithms. AMIA Annu Symp Proc. 2012;2012:911-920. 18. Konrad R, Tulu B, Lawley M. Monitoring adherence to evidence-based practices: a method to utilize HL7 messages from hospital information systems. Appl Clin Inform. 2013;4(1):126-143. doi:10.4338/ACI-2012-06-RA-0026. 19. Raja AS, Gupta A, Ip IK, Mills AM, Khorasani R. The use of decision support to measure documented adherence to a national imaging quality measure. Acad Radiol. 2014;21(3):378-383. doi:10.1016/j.acra.2013.10.017. 20. Forrest CB, Fiks AG, Bailey LC, et al. Improving adherence to otitis media guidelines with clinical decision support and physician feedback. Pediatrics. 2013;131(4):e1071-81. doi:10.1542/peds.2012-1988. 21. Ivers N, Barnsley J, Upshur R, et al. "My approach to this job is...one person at a time": Perceived discordance between population-level quality targets and patient-centred care. Can Fam physician Médecin Fam Can. 2014;60(3):258-266. http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3952764&tool=pmcentrez&r endertype=abstract. Accessed February 25, 2016. 22. Guest G. How many interviews are enough?: An experiment with data saturation and variability. Field methods. 2006;18(1):59-82. doi:10.1177/1525822X05279903. 23. Francis JJ, Johnston M, Robertson C, et al. What is an adequate sample size? Operationalising data saturation for theory-based interview studies. Psychol Health. 2010;25(10):1229-1245. doi:10.1080/08870440903194015. 42 24. Crandall B, Klein GA, Hoffman RR. Working Minds. A Practitioner's Guide to Cognitive Task Analysis. 1st ed. MIT Press: MIT Press; 2006. 25. Patton MQ. Qualitative Research & Evaluation Methods. 3rd ed. Thousand Oaks (CA): SAGE Publications; 2002. 26. Graneheim UH, Lundman B. Qualitative content analysis in nursing research: concepts, procedures and measures to achieve trustworthiness. Nurse Educ Today. 2004;24(2):105-112. doi:10.1016/j.nedt.2003.10.001. 27. Burwell SM. Setting value-based payment goals--HHS efforts to improve U.S. health care. N Engl J Med. 2015;372(10):897-899. doi:10.1056/NEJMp1500445. 28. Mandel JC, Kreda DA, Mandl KD, Kohane IS, Ramoni RB. SMART on FHIR: A standards-based, interoperable apps platform for electronic health records. J Am Med Informatics Assoc. 2016;23(5):1-10. doi:10.1093/jamia/ocv189. 29. NewYork-Presbyterian Hospital's Value Institute. http://www.nyp.org/valueinstitute/. Accessed June 14, 2016. 30. Armstrong Institute for Patient Safety and Quality. http://www.hopkinsmedicine.org/armstrong_institute/. Accessed June 14, 2016. CHAPTER 3 CLINICAL DECISION SUPPORT-BASED QUALITY MEASUREMENT (CDS-QM) FRAMEWORK: PROTOTYPE IMPLEMENTATION, EVALUATION, AND FUTURE DIRECTIONS Reprinted from American Medical Informatics Association Annual Symposium proceedings, P. V Kukhareva, K. Kawamoto, D. E. Shields, D. T. Barfuss, A. M. Halley, T. J. Tippetts, P. B. Warner, B. E. Bray, and C. J. Staes, "Clinical Decision Supportbased Quality Measurement (CDS-QM) Framework: Prototype Implementation, Evaluation, and Future Directions," pp. 825-34, 2014, Copyright (2014), with permission from American Medical Informatics Association. 44 45 46 47 48 49 50 51 52 53 CHAPTER 4 SINGLE-REVIEWER ELECTRONIC PHENOTYPING VALIDATION IN OPERATIONAL SETTINGS: COMPARISON OF STRATEGIES AND RECOMMENDATIONS Reprinted from Journal of Biomedical Informatics, P. Kukhareva, C. Staes, K. W. Noonan, H. L. Mueller, P. Warner, D. E. Shields, H. Weeks, and K. Kawamoto, "Singlereviewer electronic phenotyping validation in operational settings: Comparison of strategies and recommendations," vol. 66, pp. Feb. 2017, Copyright (2017), with permission from Elsevier. 55 56 57 58 59 60 61 62 63 64 CHAPTER 5 DISCUSSION This dissertation consists of 3 interrelated research studies aimed at advancing computer-facilitated clinical quality improvement. Chapter 2 describes a qualitative study in which domain experts were interviewed to identify opportunities and challenges in advancing clinical quality improvement through the coordinated integration of CDS and eCQM. This study established the need for better integration of CDS and eCQM, identified benefits and challenges to integration of CDS and eCQM, and proposed approaches to addressing these challenges. Chapter 3 addresses one of the main challenges described in the first study - the lack of a standard-based framework that would allow implementation of CDS and eCQM in the same fashion.1 A CDS framework called OpenCDS2 was successfully used to support eCQM. However, a capability to implement both CDS and eCQM using the same framework did not guarantee high accuracy in the generated electronic phenotypes. Indeed, low accuracy of electronic phenotyping was one of the key problems identified by the domain experts in the first study. The last study in this dissertation, described in Chapter 4, investigated how to most effectively improve the accuracy of electronic phenotypes in operational settings.3 Taken together, these studies have advanced the science of the use of informatics in healthcare. 66 5.1 Concurrent Efforts by Others As a testament to the importance of this topic, several other groups were actively engaged in related efforts during the timeframe of this dissertation research. In particular, there were significant ongoing standards development and validation efforts in the areas of CDS, eCQM, and CDS-eCQM harmonization. These relevant standards development efforts are described below. One of the most notable standards development efforts was the Clinical Quality Framework (CQF) initiative, a public-private partnership sponsored by the Centers for Medicare & Medicaid Services (CMS) and the Office of the National Coordinator for Health Information Technology (ONC) to identify, develop, and harmonize standards for CDS and eCQM.4 The CQF work group developed and tested the HL7 Clinical Quality Language (CQL) standard to enable representing computable expression logic for both CDS and eCQM.5 The Clinical Quality Language Specification, Release 1 was published in May 2015 as an HL7 Standard for Trial Use. CQF also worked on the Quality Improvement and Clinical Knowledge (QUICK) data model to represent patient data for CDS and eCQM, as well as a variety of standards based on the HL7 Fast Healthcare Interoperability Resources (FHIR) standard.6 These FHIR-based standards include the FHIR Clinical Reasoning module and the FHIR QICore Implementation Guide.7 In another highly relevant initiative, the HL7 Clinical Information Modeling Initiative (CIMI) Work Group is developing detailed clinical models that can serve as the foundation of other standards, including FHIR profiles.8 The HL7 CDS and Clinical Quality Improvement (CQI) Work Groups are working with the HL7 CIMI Work Group to enable a rigorous foundation of data interoperability to support CDS and eCQM. 67 5.2 Context Within Continuous Clinical Quality Improvement CDS and eCQM are a component of the larger context of continuous quality improvement. According to the Institute of Medicine, healthcare organizations should transform into learning healthcare systems (LHS) through such continuous and systematic efforts to measure and improve care quality.9 The Institute of Medicine suggests that the patient care experience should be systematically captured, assessed, and translated into reliable care. The LHS is based on accountability and feedback which allow virtuous cycles. Due to the "imperfectability of men,"10 perfect healthcare cannot be achieved without relying on computers. Integration of CDS and eCQM and improved validation strategies can simplify the automation required to support a LHS. 5.3 Significance This dissertation contributes significantly to the field of computer-facilitated clinical quality improvement. Advancing CDS and eCQM is essential to improving care quality and bending the cost curve. Integration of CDS and eCQM has the potential to improve medical care because it allows the closing of the feedback loop for the quality improvement cycles and simplifies the development, implementation, and maintenance of machine-executable knowledge for both CDS and eCQM. Reduced duplication of effort could help to enable greater progress in quality improvement in the face of limited available resources. Furthermore, CDS could help improve the accuracy of eCQMs by enabling the point-of-care collection of data points relevant for eCQMs, such as exclusion conditions for care interventions. In summary, a unified and validated CDSQM framework could facilitate the provision of higher quality care within the larger 68 context of continuous quality improvement and the LHS. 5.4 Innovation The work presented in this dissertation is innovative because it provides a new vision and a new framework for quality improvement in healthcare. Even though the number of publications about CDS is large, associated quality measurement efforts rarely use the same underlying technical approach.11 To the best of our knowledge, there is only a limited number of papers in the peer-reviewed literature which describe the software architecture, implementation issues, and cultural challenges associated with simultaneous implementation of performance measures and corresponding CDS interventions in a broad spectrum of healthcare related organizations.12,13 Moreover, existing manuscripts describe experiences within specific organization which may not be directly generalizable,12,13 whereas our qualitative study interviewed domain experts from numerous organizations to gather more generalizable insights. The double independent human expert review approach, with adjudication performed for interreviewer discrepancies, is generally considered the gold standard for electronic phenotyping validation in research settings.14-16 However, such double review is generally not feasible in operational settings, and we overcame this challenge by proposing and validating an innovative pragmatic single reviewer validation framework which could be used in routine operational settings. Finally, while there were a handful of prior studies that used the same underlying technology for both CDS and eCQM,12,17,18 we were one of the only ones to accomplish this integration using a standards-based, open-source approach. 69 5.5 Limitations This research has some limitations. First, we were unable to address all the challenges in computer-facilitated quality improvement. However, the field is so immense that no single body of work can adequately address all the current challenges. Second, Chapters 2 and 3 are based on research carried out in a single academic hospital. However, University of Utah Health Care is representative of many other academic hospitals and we believe that the study findings should be generalizable to other care settings. 5.6 Future Directions There are many outstanding issues remaining for improving care through CDS and eCQM, and the recommendations synthesized from domain experts could be used to guide future work. In particular, the integration of CDS and eCQM is still in its early stages, requiring significant continued work to impact care broadly. In particular, as was noted by the domain experts in Chapter 2, there is still significant heterogeneity in data representation across health IT systems and healthcare institutions. Such heterogeneity must be addressed if CDS and eCQM are to be truly interoperable. Currently, the most promising approach for addressing this long-standing issue appears to be the use of detailed FHIR profiles based on CIMI models, so that a widely adopted data interoperability approach (FHIR) can be coupled with the level of detailed semantics required for true interoperability. While the definition of such detailed FHIR profiles and underlying CIMI models still will not fully address issues of different clinical workflows and associated data collection methodologies, as well as differences in data already 70 collected in different means (if they cannot be mapped 1:1 to these detailed models), the first step must be the definition of such detailed models. With regard to the CDS-eCQM framework, a natural progression would be to update the data model from the vMR to FHIR. Also, the CDS service framework could be updated to use the CDS Hooks19 specification rather than the Decision Support Service specification, given the increasing adoption of CDS Hooks by EHR vendors. Indeed, active efforts are currently underway at the University of Utah to make this transition in the CDS-eCQM framework. In the area of electronic phenotype validations, a potential future direction is to develop cross-institutional applications for enabling electronic validation of phenotypes in operational settings. Underlying these validations will need to be accurate phenotyping that can be scaled, which potentially could be accomplished through the use of detailed FHIR profiles as well as scalable CDS-eCQM evaluation approaches as described in Chapter 3. Using these phenotyping results, a Substitutable Medical Applications and Reusable Technologies (SMART) application could be developed for enabling a validation framework fully integrated with the EHR, thereby facilitating the necessary human chart reviews.20,21 In addition, moving forward, the work presented in this dissertation should be validated in other institutions to ensure generalizability and broad applicability. Once validated, the hope would be that this work will be able to influence care widely across various healthcare settings. 71 5.7 References 1. Kukhareva P V, Kawamoto K, Shields DE, et al. Clinical decision support-based quality measurement (CDS-QM) framework: prototype implementation, evaluation, and future directions. AMIA Annu Symp Proc. 2014;2014:825-834. 2. OpenCDS Home. http://www.opencds.org/. Accessed January 30, 2014. 3. Kukhareva P, Staes C, Noonan KW, et al. Single-reviewer electronic phenotyping validation in operational settings: Comparison of strategies and recommendations. J Biomed Inform. 2017;66:1-10. doi:10.1016/j.jbi.2016.12.004. 4. Kawamoto K, Hadley MJ, Oniki T, Skapik J. The clinical quality framework initiative to harmonize decision support and quality measurement standards: defined standards, pilot results, and moving beyond quality improvement. AMIA Annu Symp Proc. 2015:202-204. 5. HL7 Standards Product Brief - HL7 Cross-Paradigm Specification: Clinical Quality Language, Release 1. http://www.hl7.org/implement/standards/product_brief.cfm?product_id=400. Accessed July 25, 2017. 6. HL7 Standards Product Brief - HL7 Fast Healthcare Interoperability Resources Specification (FHIR®), DSTU Release 2. http://www.hl7.org/implement/standards/product_brief.cfm?product_id=448. Accessed July 25, 2017. 7. Quality Improvement Core (QI-Core) Implementation Guide - FHIR v1.6.0. http://hl7.org/FHIR/us/qicore/2016Sep/index.html. Accessed July 25, 2017. 8. Goossen WTF. Detailed clinical models: representing knowledge, data and semantics in healthcare information technology. Healthc Inform Res. 2014;20(3):163172. doi:10.4258/hir.2014.20.3.163. 9. Institute of Medicine. Leadership Commitments to Improve Value in Healthcare: Finding Common Ground. Washington (DC): The National Academies Press; 2009. 10. McDonald CJ. Protocol-based computer reminders, the quality of care and the non-perfectability of man. N Engl J Med. 1976;295(24):1351-1355. doi:10.1056/NEJM197612092952405. 11. Lobach D, Sanders GD, Bright TJ, et al. Enabling health care decisionmaking through clinical decision support and knowledge management. Evid Rep Technol Assess (Full Rep). 2012;(203):1-784. 12. Goldstein MK, Tu SW, Martins S, et al. Automating performance measures and clinical practice guidelines: differences and complementarities. AMIA Annu Symp Proc. 2014:37-38. 72 13. Haggstrom DA, Saleem JJ, Militello LG, Arbuckle N, Flanagan M, Doebbeling BN. Examining the relationship between clinical decision support and performance measurement. AMIA Annu Symp Proc. 2009;2009:223-227. 14. Richesson RL, Smerek M. Electronic health records-based phenotyping. NIH Health Care Systems Research Collaboratory. http://sites.duke.edu/rethinkingclinicaltrials/ehr-phenotyping/. Published 2014. Accessed July 22, 2015. 15. Brown SH, Elkin PL, Rosenbloom ST, Fielstein E, Speroff T. eQuality for all: extending automated quality measurement of free text clinical narratives. AMIA Annu Symp Proc. January 2008:71-75. 16. Linder JA, Kaleba EO, Kmetik KS. Using electronic health records to measure physician performance for acute conditions in primary care: empirical evaluation of the community-acquired pneumonia clinical quality measure set. Med Care. 2009;47(2):208216. doi:10.1097/MLR.0b013e318189375f. 17. Raja AS, Gupta A, Ip IK, Mills AM, Khorasani R. The use of decision support to measure documented adherence to a national imaging quality measure. Acad Radiol. 2014;21(3):378-383. doi:10.1016/j.acra.2013.10.017. 18. Lakshminarayan K, Rostambeigi N, Fuller CC, Peacock JM, Tsai AW. Impact of an electronic medical record-based clinical decision support tool for dysphagia screening on care quality. Stroke. 2012;43(12):3399-3401. doi:10.1161/STROKEAHA.112.662536. 19. CDS Hooks. http://cds-hooks.org/. Accessed September 14, 2017. 20. Mandel JC, Kreda DA, Mandl KD, Kohane IS, Ramoni RB. SMART on FHIR: A standards-based, interoperable apps platform for electronic health records. J Am Med Informatics Assoc. 2016;23(5):1-10. doi:10.1093/jamia/ocv189. 21. SMART Health IT - Connecting health system data to innovators' apps. https://smarthealthit.org/. Accessed September 14, 2017. CHAPTER 6 CONCLUSION The overarching goal of this research was to advance computer-facilitated clinical quality improvement. Within this larger goal, this work aimed to address the lack of integration of CDS and eCQM and the inadequate accuracy of electronic phenotyping. The aims of this dissertation were achieved by (1) conducting a qualitative study of domain experts which explored beliefs and perceptions regarding the integration of CDS and eCQM functionality and activities, (2) demonstrating the feasibility of implementing eCQM using a CDS infrastructure, and (3) evaluating pragmatic strategies for single human validation of electronic phenotype evaluation results in operational settings. This research succeeded in exploratory analysis of issues related to CDS-eCQM integration; proposed and evaluated a standard-based, open-source CDS-eCQM framework; and evaluated 2 approaches to single-reviewer validation of electronic phenotyping results. This dissertation represents a significant step towards understanding and addressing barriers to the integration and validation of CDS and eCQM. Computer-facilitated quality improvement is an active, growing, and constantly changing field. While many challenges remain in the use of computer-facilitated quality improvement, this dissertation suggests solutions and approaches that could be followed 74 to improve the quality of healthcare using informatics. It is hoped that results from this dissertation, along with other projects currently ongoing in this field, including FHIR and CIMI, will inform new strategies for enhancing the efficiency and accuracy of computerfacilitated quality improvement, thereby ultimately leading to improvements in care quality in the United States and beyond. Reproduced with permission of copyright owner. Further reproduction prohibited without permission.