An in silico method to compare and evaluate computer-based clinical protocols

The task of comparing and evaluating the performance of different computer-based clinical protocols is difficult and expensive to accomplish. This dissertation explores methods to compare and evaluate computer-based insulin infusion protocols based on an in silico analytical framework iteratively developed for this study, using data from the intensive care unit (ICU). In Methods for Aim 1, we used a pairwise comparative technique to evaluate two computer-based insulin infusion protocols. Our result showed that the pairwise method can rapidly identify a promising computer-based clinical protocol but with limitations. In Methods for Aim 2, we used a ranking strategy to evaluate six computer-based insulin infusion protocols. The ranking method enabled us to overcome a key limitation in Methods for Aim 1, making it possible to compare multiple computer-based clinical protocols simultaneously. In Methods for Aim 3, we developed a more comprehensive in silico method based on multiple-criteria decision analysis that included user-defined performance evaluation criteria examining different facets of the computer-based insulin infusion protocols. The in silico method appears to be an efficient way for identifying promising computer-based clinical protocols suitable for clinical evaluation. We discuss the advantages and disadvantages for each of the presented methods. We also discuss future research work and the generalizability of the framework to other potential clinical areas.

AN IN SILICO METHOD TO COMPARE AND EVALUATE COMPUTER-BASED CLINICAL PROTOCOLS by Anthony Fook Leong Wong A dissertation submitted to the faculty of The University of Utah in partial fulfillment of the requirements for the degree of Doctor of Philosophy Department of Biomedical Informatics The University of Utah December 2017 Copyright © Anthony Fook Leong Wong 2017 All Rights Reserved The University of Utah Graduate School STATEMENT OF DISSERTATION APPROVAL The dissertation of Anthony Fook Leong Wong has been approved by the following supervisory committee members: , Chair Peter J. Haug 4/14/2017 Date Approved , Member R. Scott Evans 4/14/2017 Date Approved , Member Lewis Frey Date Approved , Member Katherine Sward 4/14/2017 Date Approved , Member Alan H. Morris 4/14/2017 Date Approved and by Wendy Chapman the Department/College/School of , Chair/Dean of Biomedical Informatics and by David B. Kieda, Dean of The Graduate School. ABSTRACT The task of comparing and evaluating the performance of different computer-based clinical protocols is difficult and expensive to accomplish. This dissertation explores methods to compare and evaluate computer-based insulin infusion protocols based on an in silico analytical framework iteratively developed for this study, using data from the intensive care unit (ICU). In Methods for Aim 1, we used a pairwise comparative technique to evaluate two computer-based insulin infusion protocols. Our result showed that the pairwise method can rapidly identify a promising computer-based clinical protocol but with limitations. In Methods for Aim 2, we used a ranking strategy to evaluate six computerbased insulin infusion protocols. The ranking method enabled us to overcome a key limitation in Methods for Aim 1, making it possible to compare multiple computer-based clinical protocols simultaneously. In Methods for Aim 3, we developed a more comprehensive in silico method based on multiple-criteria decision analysis that included user-defined performance evaluation criteria examining different facets of the computerbased insulin infusion protocols. The in silico method appears to be an efficient way for identifying promising computer-based clinical protocols suitable for clinical evaluation. We discuss the advantages and disadvantages for each of the presented methods. We also discuss future research work and the generalizability of the framework to other potential clinical areas. TABLE OF CONTENTS ABSTRACT ....................................................................................................................... iii LIST OF FIGURES .......................................................................................................... vii LIST OF TABLES ............................................................................................................. ix DEFINITION OF TERMS ................................................................................................ xi ACKNOWLEDGMENTS ................................................................................................ xii 1 INTRODUCTION ...................................................................................................... 1 1.1 Statement of the Problem .............................................................................. 1 1.2 Purpose of the Study ..................................................................................... 2 1.3 Aims .............................................................................................................. 3 1.3.1 Aim 1 ..................................................................................................... 3 1.3.2 Aim 2 ..................................................................................................... 3 1.3.3 Aim 3 ..................................................................................................... 3 1.4 Importance of the Study ................................................................................ 4 1.5 Scope of the Study ........................................................................................ 5 1.6 Future Work .................................................................................................. 5 2 REVIEW OF LITERATURE ..................................................................................... 6 2.1 2.2 2.3 2.4 Computer-based Clinical Protocols .............................................................. 6 Reasons for Comparing Computer-based Clinical Protocols ....................... 7 Current Comparison Strategy for Computer-based Clinical Protocols......... 9 Stressed-induced Hyperglycemia and Insulin Infusion Protocols for Blood Glucose Management in ICU Patients ............................................ 10 2.5 Computer-based Insulin Infusion Protocols................................................ 11 2.5.1 eProtocol-insulin .................................................................................. 11 2.5.2 HWCIR Glucose Protocol ................................................................... 12 2.5.3 Glucosafe ............................................................................................. 13 2.5.4 Atlanta Medical Center Protocol ......................................................... 13 2.5.5 Thomas Jefferson Insulin Infusion Protocol ........................................ 14 2.5.6 Nice-Sugar Protocol ............................................................................ 14 2.6 Computer Simulations for Comparing Computer-based Clinical Protocols ..................................................................................................... 15 2.6.1 2.6.2 2.6.3 Computational Models ........................................................................ 15 Virtual Patients .................................................................................... 16 Using Computer Simulation for Computer-based Insulin Infusion Protocols Evaluation ............................................................. 17 2.7 Decision-making and Multi-attribute Utility Theory .................................. 18 2.8 Social Choice Theory and Voting System .................................................. 20 2.8.1 Single-winner Methods........................................................................ 21 2.8.1.1 Single Voting .................................................................................. 21 2.8.1.2 Ranked Voting ................................................................................ 22 2.8.2 Multiple-winner Methods .................................................................... 24 2.8.2.1 Proportional Method ....................................................................... 24 3 METHODS ............................................................................................................... 25 3.1 Data Sources ............................................................................................... 25 3.1.1 Data Collection .................................................................................... 25 3.1.2 Inclusion and Exclusion Criteria ......................................................... 25 3.2 Batch Comparison with Computer-based Clinical Protocol Candidates .................................................................................................. 27 3.2.1 HWCIR Glucose Protocol ................................................................... 27 3.2.2 Glucosafe ............................................................................................. 27 3.2.3 Atlanta Medical Center Protocol ......................................................... 27 3.2.4 Thomas Jefferson Insulin Infusion Protocol ........................................ 28 3.2.5 NICE-SUGAR Protocol ...................................................................... 28 3.3 In silico Framework .................................................................................... 29 3.3.1 Conceptual Design ............................................................................... 29 3.3.2 Features ................................................................................................ 30 3.4 Favorability Scoring.................................................................................... 32 3.5 Methods for Aim 1 ...................................................................................... 33 3.5.1 Performance Comparison and Evaluation ........................................... 39 3.5.2 Statistical Analysis .............................................................................. 40 3.5.2.1 One Sample Z-test for Proportion (for Distributions of IV Insulin Infusion Rates) ................................................................... 40 3.5.2.2 Wilcoxon Signed Rank Test (for favorability scores) .................... 40 3.6 Methods for Aim 2 ...................................................................................... 40 3.7 Methods for Aim 3 ...................................................................................... 43 3.7.1 Estimating the Subsequent Blood Glucose Level................................ 43 3.7.2 Multiple Criteria Decision Analysis .................................................... 49 3.7.2.1 Ranked Favorability Scoring .......................................................... 49 3.7.2.2 Estimation of Hypoglycemia Rate .................................................. 50 3.7.2.3 Estimation of Hyperglycemia Rate ................................................. 51 3.7.2.4 Estimation of Cases Within the Target Range ................................ 51 3.7.2.5 Mean of Recommended Continuous IV Insulin Infusion Rates ............................................................................................... 51 3.7.2.6 Median of Recommended Continuous IV Insulin Infusion Rates ............................................................................................... 51 3.7.2.7 Distributed Favorability Scoring ..................................................... 52 v 3.7.3 4 Aggregating Decision Scores .............................................................. 52 RESULTS AND DISCUSSIONS ............................................................................. 54 4.1 Aim 1 .......................................................................................................... 54 4.1.1 Results for Aim 1 ................................................................................. 54 4.1.2 Discussion for Aim 1 ........................................................................... 67 4.2 Aim 2 .......................................................................................................... 69 4.2.1 Results for Aim 2 ................................................................................. 69 4.2.2 Discussion for Aim 2 ........................................................................... 77 4.3 Aim 3 .......................................................................................................... 78 4.3.1 Results for Aim 3 ................................................................................. 78 4.3.1.1 Ranked Favorability Scoring .......................................................... 78 4.3.1.2 Estimation of Hypoglycemia Rate .................................................. 79 4.3.1.3 Estimation of Hyperglycemia Rate ................................................. 81 4.3.1.4 Estimation of Cases Within the Target Range ................................ 82 4.3.1.5 Mean of Recommended Continuous IV Insulin Infusion Rates ............................................................................................... 85 4.3.1.6 Median of Recommended Continuous IV Insulin Infusion Rates ............................................................................................... 88 4.3.1.7 Distributed Favorability Scoring ..................................................... 90 4.3.2 Aggregating Decision Scores .............................................................. 91 4.3.3 Discussion for Aim 3 ........................................................................... 94 5 CONCLUSION ....................................................................................................... 101 5.1 5.2 5.3 Strengths and Limitations ......................................................................... 102 Generalizability ......................................................................................... 104 Suggestions for Future Research .............................................................. 104 REFERENCES ............................................................................................................... 107 vi LIST OF FIGURES 3.1 Conceptual framework for in silico comparison and evaluation of computerbased clinical protocols ............................................................................................ 30 3.2 Physiological process simulation interacting with computer-based clinical protocols................................................................................................................... 31 3.3 The comparison and evaluation module supporting multiple platforms ................. 32 3.4 Proof of concept software to compare Glucosafe with eProtocol-insulin ............... 34 3.5 Input data and output response of Glucosafe ........................................................... 34 3.6 Temporal characteristics of eProtocol-insulin and Glucosafe. Times of data used for continuous IV insulin infusion rate recommendation and time of blood glucose used for assessment of the appropriateness of the continuous IV insulin infusion rate recommendation. ............................................................... 36 3.7 An example of favorability scoring when the observed blood glucose at time=ti+1 is high........................................................................................................ 38 3.8 An example of favorability scoring when the observed blood glucose at time=ti+1 is in the target range. ................................................................................. 38 3.9 An example of association between blood glucose and insulin in one patient over time .................................................................................................................. 45 3.10 An example of estimating the subsequent blood glucose to determine the favorability ............................................................................................................... 47 3.11 The estimated blood glucose at time=t2 due to candidate's recommended insulin....................................................................................................................... 50 3.12 Assignment of scores for distributed favorability scoring ....................................... 53 4.1 CONSORT diagram of considered patients............................................................. 58 4.2 Distributions of recommended continuous IV insulin infusion rate by eProtocol-insulin and Glucosafe for all blood glucose measurements .................... 62 4.3 Distributions of recommended continuous IV insulin infusion rate by eProtocol-insulin and Glucosafe stratified by blood glucose measurements at time=ti ...................................................................................................................... 64 4.4 Distributions of recommended continuous IV insulin infusion rate by six computer-based insulin infusion protocol candidates for all blood glucose measurements........................................................................................................... 72 4.5 Ranked favorability scores for six computer-based insulin infusion protocol candidates................................................................................................................. 74 4.6 Ranked favorability scores stratified by blood glucose range ................................. 75 4.7 Estimated cases of hypoglycemia (* represents actual cases). ................................ 80 4.8 Estimated cases of hyperglycemia (* represents actual cases). ............................... 83 4.9 Estimated cases of blood glucose within the target range (* represents actual cases). ....................................................................................................................... 84 4.10 Mean of recommended continuous IV insulin infusion rates .................................. 86 4.11 Median of recommended continuous IV insulin infusion rates ............................... 89 4.12 Distributed favorability scoring ............................................................................... 91 4.13 Aggregated score using multiple-criteria decision analysis (*higher score is better). ...................................................................................................................... 94 viii LIST OF TABLES 2.1 An example of Borda count with five candidates .................................................... 23 2.2 An example of Borda count with N-i points ............................................................ 23 3.1 An example input data for estimating the subsequent blood glucose level for a computer-based insulin infusion protocol candidate............................................. 48 4.1 Number of patients in different versions of eProtocol-insulin................................. 55 4.2 Completeness of patient data ................................................................................... 56 4.3 Statistical description of patients ............................................................................. 59 4.4 Primary discharge diagnostic categories for eProtocol-insulin patients in the data sample............................................................................................................... 60 4.5 Statistical results of recommended continuous IV insulin infusion rates between eProtocol-insulin and Glucosafe ................................................................ 62 4.6 Counts of more favorable continuous IV insulin infusion rate recommendations at time=ti from eProtocol-insulin or Glucosafe, based on three blood glucose categories at time=ti+1. ............................................................. 65 4.7 Continuous IV insulin infusion rate recommendations and difference in recommended IV insulin infusion rate ..................................................................... 66 4.8 Statistical results of recommended continuous IV insulin infusion rates by six computer-based insulin infusion protocols. ............................................................. 71 4.9 Pairwise Wilcoxon signed rank test for six computer-based insulin infusion protocol candidates .................................................................................................. 73 4.10 Continuous IV insulin infusion rate recommendations by six computer-based insulin infusion protocol candidates. ....................................................................... 76 4.11 Ranking of computer-based insulin infusion protocol candidates using criterion 1 (ranked favorability scoring) .................................................................. 79 4.12 Ranking of computer-based insulin infusion protocol candidates using criterion 2 (estimated cases of hypoglycemia)......................................................... 80 4.13 Number of cases estimated to have risk of hypoglycemia stratified according to favorability scoring in Methods for Aim 1 .......................................................... 81 4.14 Ranking of computer-based insulin infusion protocol candidates using criterion 3 (estimated cases of hyperglycemia) ....................................................... 83 4.15 Ranking of computer-based insulin infusion protocol candidates using criterion 4 (estimated cases of blood glucose within the target range) .................... 84 4.16 Ranking of computer-based insulin infusion protocol candidates using criterion 5 (mean recommended continuous IV insulin infusion rate) .................... 87 4.17 Ranking of computer-based insulin infusion protocol candidates using criterion 5 (median recommended continuous IV insulin infusion rate) ................. 90 4.18 Ranking of computer-based insulin infusion protocol candidates using criterion 7 (distributed favorability scoring) ............................................................ 92 4.19 Scores using multiple-criteria decision analysis with seven criteria to measure computer-based insulin infusion protocols (*higher score is better). ...................... 93 x DEFINITION OF TERMS COMPUTER-BASED CLINICAL PROTOCOLS Computer version of specialized clinical guidelines to help healthcare practitioners make decision about the diagnosis, management, and treatment in medicine HWCIR Homer Warner Center for Informatics Research FAVORABILITY SCORE A numerical measure to indicate the level of user-defined performance for a computer-based clinical protocol based on a set rules ICU Intensive Care Unit IN SILICO A scientific research performed via computer modeling or computer simulation IQR Interquartile Range IV Intravenous NICE-SUGAR Normoglycemia in Intensive Care Evaluation and Surviving Using Glucose Algorithm Regulation PBPK Physiologically-based pharmacokinetic Time=ti-1 Previous time Time=ti Current time Time=ti+1 Subsequent time U/h Units per hour ACKNOWLEDGMENTS The journey to complete this dissertation has been long and ardous. This work would not have been possible without the financial support of the Homer Warner Center for Informatics at Intermountain Healthcare. At the forefront, I am especially indebted to my advisor and chairman of my committee, Dr. Peter Haug, for his excellent guidance, patience, and support throughout my research. He has taught me more than I could ever give him credit for. I would like to thank Dr. Alan Morris, who initiated this project, for his continuous support and mentorship. I am grateful to Dr. Kathy Sward, who has provided her extensive personal time to develop and corrected my writing. Without her help and encouragement, completing this dissertation would not have been possible. I would also like to thank Drs. R. Scott Evans and Lewis Frey for guiding my research for the past several years. Special thanks go to Drs. Steen Andreassen and Ulrike Pielmeier for supporting our research effort involving Glucosafe. I am grateful to all of those with whom I have had the pleasure to work during this and other related projects, especially the physicians, informaticists, nurses, statisticians, and database administrators at Intermountain Healthcare. My research would not have been possible without their help. I would like to thank Drs. Senthil Nachimuthu and Vikrant Deshmukh, who, as good friends and colleagues, were always willing to help and give invaluable suggestions. My family has been the source of motivation while I pursue this dissertation. I would like to thank my parents, whose love and guidance are with me in whatever I pursue in life. Most importantly, I would like to thank my loving and supportive wife, Jessie, and my two wonderful children, Cayden and Christian, who provide unending inspiration. xiii 1 INTRODUCTION This dissertation addressed some of the issues that healthcare practitioners face when comparing and evaluating computer-based clinical protocols for bedside use and research purposes. With the advent of high-performance computing, computer-based clinical protocols can be evaluated in a computer-simulated environment. Computer simulation has the ability to reproduce the behavior of a system or analyze complex processes safely. Healthcare practitioners can use computer simulations to explore issues, gain new insights, and analyze the performance of a system without directly affecting their patients. This dissertation proposes a framework to develop an in silico method to identify promising computer-based protocols for clinical study. 1.1 Statement of the Problem The rising cost of healthcare, fueled by increased demand for better care, has urged many healthcare practitioners to seek more effective tools for improving clinical care practice [1-4]. Computer-based clinical protocols offer the possibilities of a wider dissemination and effective use of evidence-based guidelines and clinical protocols while reducing variation in the treatment of patients [5-8]. Extensive developments in multiple healthcare institutions have resulted in an increase in the number of computer-based clinical protocols available to healthcare practitioners. For example, computer-based insulin infusion protocol described in the literature has many variations [9-13]. When 2 healthcare practitioners attempt to select a computer-based clinical protocol that will be suitable for their practice, they may have concerns such as which computer-based clinical protocol performs better or safer. Current strategy for comparing computer-based clinical protocols is to implement these protocols in clinical trials [14,15]. However, this approach is expensive, time-consuming, and requires an extensive amount of clinical care resources [16-18]. An efficient method for comparing and assessing the performance of computerbased clinical protocols prior to evaluation in the clinical setting would be valuable to address those concerns. Any computer-based clinical protocol must be evaluated extensively by healthcare practitioners before it can be utilized at the bedside. Flawed scientific evidence due to lacking or misinterpretation of results can lead to suboptimal development of computerbased clinical protocols [19]. Unfortunately, investigating this issue in clinical trials is probably not be the best option because almost every clinical trial exposes patients to risk and some patients may be harmed during a trial. This makes it more challenging to compare alternative computer-based clinical protocols. By rethinking our approach to include preliminary evaluation of candidate computer-based clinical protocols, we can find a more effective in silico method that can help us identify computer-based clinical protocols worthy for clinical trials. 1.2 Purpose of the Study The purpose of this study was to develop an informatics-based approach for comparing and evaluating computer-based clinical protocols. The framework proposed in this dissertation is an in silico method, with emphasis on computer simulations of physiological processes and systematically critiquing the alternative computer-based clinical protocols. 3 1.3 1.3.1 Aims Aim 1 The first aim of this study was to investigate the feasibility of developing an in silico method for identifying promising computer-based clinical protocols for clinical study. Research question 1.1: Can we develop an in silico method for comparing and evaluating alternative computer-based clinical protocols? Research question 1.2: How do we critique the clinical significance of the comparison? Aim 1 was addressed by developing an in silico method that could compare two computer-based clinical protocols using data linked to the use of one of the protocols. 1.3.2 Aim 2 The second aim of the study was to expand the in silico method to analyze and evaluate multiple computer-based clinical protocol candidates. Research question 2.1: How do we expand the favorability scoring algorithm to evaluate multiple computer-based clinical protocols? Research question 2.2: What are the advantages and disadvantages of the new in silico method? When addressing Aim 2, we designed a different approach to favorability scoring algorithm that could compare multiple computer-based clinical protocol candidates. 1.3.3 Aim 3 The third aim of the study was to design a method that can help healthcare practitioners choose between alternative computer-based clinical protocols. 4 Research question 3.1: What are the appropriate criteria for estimating the performance of computer-based clinical protocols? Research question 3.2: How do we measure and critique potential risks when comparing the outcome of the scoring models? Research question 3.3: How do we facilitate the decision-making process and present the results to the healthcare practitioners? To address Aim 3, we designed a method based on multiple-criteria decision analysis that can assign weights to alternative computer-based clinical protocols. 1.4 Importance of the Study The in silico method plays an important role in the translation of research results to clinical practice. While the in silico method cannot replace actual clinical trials, many discoveries can be made in computer simulations before we invest in clinical evaluations. The performance of each computer-based clinical protocol and its expected outcome can be examined thoroughly for different clinical and patient scenarios. Therefore, healthcare practitioners can make a more informed decision about which computer-based clinical protocol is best suited for further evaluation. Our in silico method offers advantages in cost, time, and safety. The huge expenses related to a full-scale clinical trial can be reduced by conducting a pre-trial using in silico experiments. Since human subjects are not directly involved in the in silico method, clinical care resources are not required during this phase. Computer simulations using retrospective data derived from real patients clinically treated with a similar protocol allow healthcare practitioners to study how future patients may respond to the new treatment protocols. Conditions can be varied and probable outcomes can be investigated safely. Furthermore, 5 critical situations can be examined without posing any risk to actual patients. Finally, these evaluations can also enhance the development of computer-based clinical protocols because we can iteratively refine the protocol specifications and get an immediate estimation of clinical responses through the in silico method. 1.5 Scope of the Study This dissertation study was limited to an investigation of in silico methods for comparing computer-based insulin infusion protocols. The computer-based insulin infusion protocols were specifically designed to treat patients with stress hyperglycemia using intravenous (IV) insulin infusion. The researcher did not systematically compare and contrast these computer-based insulin infusion protocols with other types of computerbased clinical protocols, such as mechanical ventilator management or Coumadin dosing. 1.6 Future Work This single study of computer-based insulin infusion protocol provides a richness of data and can lead to deeper understanding of the in silico method. The working hypotheses derived from this study can be tested in subsequent research. In the future, better simulation models can be developed using the in silico framework to determine which patient will benefit from the computer-based clinical protocol intervention. We could also better target patients and select the most appropriate computer-based clinical protocol for personalized care solutions. 2 2.1 REVIEW OF LITERATURE Computer-based Clinical Protocols Clinical guidelines and protocols can help healthcare practitioners to make decisions about the diagnosis, management, and treatment in specific areas of medicine [20-22]. They are the most effective method for disseminating evidence-based healthcare practices. Clinical guidelines are typically general statements about clinical recommendations and best practices based on the examination of current literature and expert opinions [20-22]. Clinical protocols are specialized, more detailed versions of the guidelines, often containing locally specific details or algorithms such as drug dosage schedules. For this research, we focused only on clinical protocols. Development methods for clinical guidelines and protocols can vary widely. Healthcare practitioners and researchers use a variety of heuristics or modeling techniques that may not reflect standardized clinician decision making [9,23-25]. Different methods may lead to different patient outcomes. Unaided healthcare practitioners and inconsistent use of clinical guidelines can affect their clinical care practice and health outcomes [26]. Compliance with guidelines and protocols among healthcare practitioners can vary widely even when they are based on reputable evidences [27-29]. As a result, patients can be harmed when clinicians do not comply with best evidence [30]. Rapid development of information technology has continue to change how clinical 7 protocols are implemented and disseminated [7,31]. Many of the challenges highlighted above can be resolved by implementing the clinical protocols in computers. Computerbased clinical protocols can produce standardized clinical decisions while retaining the ability to adapt to contextual changes, thus personalizing patient care [7,31,32]. Computerbased clinical protocols can offer a wider dissemination and effective use of clinical protocols while reducing variation in the treatment of patients [5-8]. Use of computerbased clinical protocols has produced favorable clinical outcomes [2,5,6,33-35]. Studies have shown improved protocol adherence when healthcare practitioners use computerbased clinical protocols [6,36,37]. Most importantly, computer-based clinical protocols have been reported to improve clinician performance, healthcare processes, and patient outcomes [4-6,38]. 2.2 Reasons for Comparing Computer-based Clinical Protocols It is anticipated that an increasing number of computer-based clinical protocols will be developed to meet the needs of healthcare [1,4,6,9,11,12,39]. However, extensive development of computer-based clinical protocols can also lead to variations in their implementation because they were developed in local institutions. The computer-based insulin infusion protocol for treating stress hyperglycemia is one such example [9-13]. Medical researchers who developed these computer-based clinical protocols used different heuristics, algorithms, expert opinions, and clinical evidence in their local implementation. Different methods, even though rigorous, can lead to different health outcomes. As expected, healthcare practitioners would prefer to use the most suitable computer-based clinical protocol that will have a positive impact in their clinical practice [40]. Development of computer-based clinical protocols is usually expensive. Knowledge 8 engineers must transform clinical evidences, rules, and expert opinions into computable form so that healthcare practitioners can make appropriate clinical decisions with the aid of a computer [31]. The computer-based clinical protocol has to be tested extensively before it can be used in the clinical setting. Software testing ensures that the computerbased clinical protocol meets the requirement, responds correctly to the inputs, and performs its functions accurately within an acceptable time. However, part of the knowledge engineering development lifecycle of a computer-based clinical protocol is to understand how the software can impact future patients. Therefore, software testing is inadequate in this respect because the test will only inform us whether the software components perform according to their specifications. While it is important to assess how the computer-based clinical protocol performs in an actual clinical environment, incorporating patient assessment in a clinical trial during the software testing phase poses a high risk for patients. It would be better to test the computer-based clinical protocol in a computer simulation and compare the performance with an existing protocol. Hence, an effective in silico method can help software developers and healthcare practitioners better understand the impact of the computer-based clinical protocol in a real-world setting. Since the computer-based clinical protocol is based on scientific data and expert opinion available at the time the protocol is adopted, the protocol must be constantly reevaluated and updated when new data and information become available [22]. Healthcare practitioners are usually involved in the adoption and evaluation of computer-based clinical protocols so that they can understand, accept, and use them effectively [26,31]. Therefore, the evaluation strategy should also include investigating patient safety issues 9 and patient benefits before and after the new evidence is incorporated into the computerbased clinical protocol. 2.3 Current Comparison Strategy for Computer-based Clinical Protocols Clinical evaluation for different strategies of care involving computer-based clinical protocols is challenging because of: (i) the complexity of clinical environment, (ii) the expense of clinical trials, (iii) the time necessary for these trials, (iv) the large consumption of clinical research and care resources during these trials, and (v) regulatory barriers [1,41]. A method of comparing different computer-based clinical protocols to determine those with sufficient merit to warrant evaluation in a clinical trial would be valuable to ease these challenges. It is in the interest of healthcare practitioners to evaluate the computer-based clinical protocols rigorously before they are implemented as a routine clinical intervention. Current evidence still suggests clinical trials as the best method to compare computer-based clinical protocols [14,15]. For example, Blaha et al. compared three insulin infusion protocols, including a computer-based insulin infusion protocol called enhanced model predictive control (eMPC) algorithm, in a randomized controlled trial [42]. However, as indicated above, this approach is expensive, utilizes large amount of clinical resources, and is timeconsuming [16-18]. Many comparisons typically involve a computer-based clinical protocol compared with a paper-based protocol or routine treatment management [9,12-14,42-46]. Morris et al. investigated two different versions of computer-based insulin infusion protocols used in clinical practices and compared their performance retrospectively [47]. Several comparison of clinical protocols, including computer-based protocols, were reported in 10 systematic reviews [12,40,48,49]. These reviews yielded useful information, but unfortunately, are not adequate for healthcare practitioners to make a decision about selecting the best computer-based clinical protocol for further clinical study. The literature lacked in-depth analysis of the problem and contained limited critiquing of the protocols, mainly due to the heterogeneity of the case studies. Allart et al. outlined a software architecture to compare computer-based clinical protocols but did not address specific methods to evaluate them [50]. Lee et al. [51] described an in silico evaluation of insulin infusion protocols using virtual populations developed by Hovorka et al. [52]. However, the study did not use any real patients in the ICU or a fully developed computer-based insulin infusion protocol. It is plausible that these preliminary efforts could be extended to create a more effective strategy to compare and evaluate computer-based clinical protocols. 2.4 Stressed-induced Hyperglycemia and Insulin Infusion Protocols for Blood Glucose Management in ICU Patients Critically ill patients often develop hyperglycemia (increased blood glucose above the normal range), insulin resistance, and glucose intolerance due to hypermetabolic stress [53]. Drugs such as steroids, beta blockers, diuretics, and niacin can cause clinically significant elevated blood glucose concentrations. Hyperglycemia is also seen as a response to stress due to elevated levels of cortisol and catecholamines. Hyperglycemia can also be seen in critically ill patients with no prior history of diabetes. Reports that intensive intravenous (IV) insulin therapy could decrease morbidity and mortality of patients in the intensive care unit (ICU) changed thinking about management 11 of stress-induced hyperglycemia, and about the proper target for blood glucose [54,55]. As a result, many experts developed guidelines to manage blood glucose in these patients. Titration of IV insulin infusion is one of the most common clinical guidelines for critically ill patients. The goal is to maintain near-normal glycemic levels through continuous IV insulin infusion. However, this is not an easy task. Insulin sensitivity varies from person to person and at different time points in the same person. Human bodies have different physiologic responses that affect how insulin regulates blood glucose. Risk factors can include age, obesity, diet, genetics, infections, and medication. Because of the complexities in this process, unaided healthcare practitioner decisions about insulin dosing, or decisions using a general guideline, vary widely. A computer-based clinical protocol can help standardize the clinical decision-making process of adjusting appropriate IV insulin infusion for individual patients and increase compliance among healthcare practitioners [7,31,47]. 2.5 Computer-based Insulin Infusion Protocols 2.5.1 eProtocol-insulin Intermountain Healthcare implemented a detailed, adequately explicit, computerbased protocol (eProtocol-insulin) for management of stress hyperglycemia in the intensive care unit (ICU) from 2004 to 2010 [37,47,56]. eProtocol-insulin is an open-loop, heuristic, rule-based system. It is an empiric protocol that recommends continuous IV insulin infusion rate, based on the difference between the most recent blood glucose and the blood glucose target, the rate of change of blood glucose, the current continuous IV insulin infusion rate, previous concentrated IV glucose doses (if any, for treatment of hypoglycemia), and time [37,47,56]. Blood glucose is measured every two hours and 12 entered into eProtocol-insulin to get the next continuous IV insulin infusion rate recommendation. Bedside healthcare practitioners review each eProtocol-insulin recommendation before adjusting the continuous IV insulin infusion rate. If the healthcare practitioner declines the recommendation, the healthcare practitioner will set their preferred continuous IV insulin infusion rate according to their clinical judgment. Bedside healthcare practitioners accepted 95% of eProtocol-insulin recommendations [37,47,56]. This is an iterative method that produces intermediate outcome results (blood glucose value changes) that are either successful (fall within target range or move closer to the target range) or unsuccessful. The resulting clinical database with its sequence of successful and unsuccessful changes is unique because it is a robust reflection of the interaction between the eProtocol-insulin and ICU patients with stress hyperglycemia. This database provides a resource for comparison of eProtocol-insulin with other replicable methods for managing stress hyperglycemia. At any time point, the patient's physiologic state can be estimated based on their response to the administered insulin. Comparison of the successful and unsuccessful response rates for eProtocol-insulin recommendations, and the probable responses produced by an alternative strategy, allows evaluation of their comparative clinical suitability. 2.5.2 HWCIR Glucose Protocol The HWCIR Glucose Protocol is a derivation of eProtocol-insulin implemented as a .NET application in Intermountain Healthcare. This was done so that the application conformed to the enterprise standards for use at the bedside in Intermountain Healthcare. Several rules in the computer-based protocol were updated by healthcare practitioners to reflect their current practice. Notable changes included discontinuing the use of IV insulin 13 infusion when the blood glucose measurement falls below 60 mg/dL. 2.5.3 Glucosafe Glucosafe is a complex, multi-organ physiologic model, decision support system for blood glucose management developed by University of Aalborg, Denmark [45,57,58]. Glucosafe recommended a continuous IV insulin infusion rate and an IV insulin bolus. The IV insulin bolus was only recommended when blood glucose exceeds 180 mg/dL. Glucosafe calculated insulin sensitivity based on blood glucose measurements, amount of insulin previously given, and various nutritional inputs. The model also considers insulin saturation effects and the glucose absorption rate as a function of carbohydrate content in the gastrointestinal tract (based on the rate and type of enteral feeding) [58]. Internally, Glucosafe had four penalty functions to optimize the amount of the continuous IV insulin infusion rate for every recommendation; blood glucose penalty, insulin penalty, caloric input penalty, and nutrition penalty [57,58]. Small-scale prospective studies were conducted in Europe to examine safety and performance issues [45,57]. 2.5.4 Atlanta Medical Center Protocol The Atlanta Medical Center (AMC) protocol is a columnar insulin dosing chart developed by the Diabetes Special Interest Group (Georgia Hospital Association) in Georgia, USA to standardize the management of hyperglycemia [59]. The target blood glucose range was 80-110 mg/dL. A pilot study involving twenty patients was conducted at the Atlanta Medical Center in 2006 [59]. The average time to reach the target blood glucose range was 12.8 hours and hypoglycemia (below 60 mg/dL) was found in 0.9% of the measured blood glucoses [59]. 14 2.5.5 Thomas Jefferson Insulin Infusion Protocol The Thomas Jefferson Insulin Infusion Protocol (TJIIP) is a nurse-managed protocol for delivering continuous IV insulin infusion to hyperglycemic patients at the ICU; it was developed by Thomas Jefferson Hospital, Pennsylvania, USA [60]. The target blood glucose range was 100-140 mg/dL. Murphy et al. conducted a retrospective study on 108 patients in a surgical cardiac care unit where their blood glucose were managed by TJIIP [60]. Median blood glucose was 154 mg/dL. Two hypoglycemic (less than 60 mg/dL) episodes were reported affecting two patients. However, it was not clear how many total blood glucose measurements were made. The article concluded that an intense use of insulin infusion protocol may not necessarily provide better glycemic control [60]. 2.5.6 Nice-Sugar Protocol The Normoglycemia in Intensive Care Evaluation and Surviving Using Glucose Algorithm Regulation (NICE-SUGAR) study was a collaboration between the Australian and New Zealand Intensive Care Society Clinical Trials Group, the George Institute for International Health (University of Sydney), the Canadian Critical Care Trials Group, and the Vancouver Coastal Health Research Institute (University of British Columbia) on intensive glycemic control [61]. The NICE-SUGAR study investigators conducted a multicenter randomized trial involving 6,104 patients admitted to the surgical and medical ICUs in 42 hospitals during the period from December 2004 to November 2008 [61]. The patients were randomly assigned to intensive glycemic control with a target blood glucose of 81108 mg/dL and conventional glycemic control with target blood glucose ≤ 180 mg/dL. The study found that the 90-day mortality rate for the intensive glycemic control group was higher than the conventional group. This result contradicted with the study by Van den 15 Berghe et al. which had demonstrated a significantly lower ICU mortality in the intensive glycemic control arm [54]. Severe hypoglycemia (blood glucose ≤ 40 mg/dL) was also much higher in the intensive glycemic control arm of the NICE-SUGAR study (6.8% vs 0.5%) [61]. 2.6 Computer Simulations for Comparing Computer-based Clinical Protocols Computer simulation continues to play an increasingly important role in medicine [62- 64]. For this study, we defined computer simulation as the act of imitating a real-world process or behavior over time. Computer simulations are often used in medical education, computer-based assessment, and physiological modeling research [65-68]. The advantage of using computer simulation includes the ability to analyze complex processes, examine critical issues, and mimic life-like situations using real patient data without impacting the health of patients [62]. Thus, medical researchers can investigate various scenarios of care including life-threatening situations without harming the actual patient [66]. 2.6.1 Computational Models Computer simulation in medicine often requires a computational model to represent a realistic situation, system, or process, and a set of virtual patients to act on the situation, system, or process [65,68-70]. For example, computational models for glucose regulation have been used to develop new insulin infusion algorithms [52,71,72]. Many of these simulation models and virtual patients were designed using mathematical models including physiologic based pharmacokinetic models or compartmental models [52,71-74]. 16 Physiologically-based pharmacokinetic (PBPK) modeling is a mathematical modeling technique for predicting the absorption, distribution, metabolism, and excretion (ADME) of synthetic or natural chemical substances in humans and animals [67,68,75]. This approach has been demonstrated to be useful in identifying drug targets by applying the knowledge of pharmacodynamics interactions between drugs and biochemistry during the modeling of drug interaction [76]. Schaller et al. developed a deeper understanding of the insulin-glucose regulatory system by using this PBPK modeling approach [75]. Another approach to mathematical modeling is through compartmental models where a collection of interconnected physiological compartments with specified inputs and outputs defined the system [70]. Hovorka et al. developed a complex system of five submodels (endogenous insulin secretion, insulin kinetics, enteral glucose absorption, insulin action, and glucose kinetics) for closed-loop glucose control [44,52]. Subsequently, Pielmeier adopted this model for the development of Glucosafe [58]. Increasingly large amounts of clinical data are being delivered and stored in electronic format. This can lead to the development of more data-driven computational models in medicine. Parameter fitting models are a type of mathematical model where a function or complex equation is optimized to best fit to the data points. Some of the common parameter estimation techniques used are ordinary least squares, linear regression, and Bayesian inference [77,78]. Parameter fitting models have been used to study glucose-insulin regulation system [79-84]. 2.6.2 Virtual Patients Virtual patients are reflection of actual patients engaged in healthcare during computer simulations. Virtual patients can take several different forms: (i) artificial patients where 17 biochemical or physiological processes are simulated, (ii) responses based on the data from real patients from electronic health records (EHR), (iii) physical simulators such as mannequins, and (iv) simulated patients where the patient is recreated to engage in patient acting or role-play. Hovorka et al. coined the term "experimental in silico cloning" as the process of transforming clinical data from real patients into virtual patients [52]. Virtual patients derived from real patient data have been used to assist in the development and refinement of computer-based insulin infusion protocols [85,86]. 2.6.3 Using Computer Simulation for Computer-based Insulin Infusion Protocols Evaluation Preclinical trials using in silico methods have been studied to evaluate insulin infusion algorithms [72,87,88]. These trials involved a closed-loop strategy with virtual patients as subjects in the computer simulation [72,87,88]. However, these studies did not compare one insulin dosing algorithm with another computer-based clinical protocol. Wilinska et al. investigated two versions of a computer-based insulin infusion protocol (model predictive control (MPC)-based glucose control algorithm) in silico using 10 virtual patients [71]. These two versions were compared quantitatively using measures such as mean glucose, time in target, time to target, hypoglycemic episodes, and subjects with hypoglycemia [71]. Lee et al. investigated two insulin infusion protocols, SPRINT and NICE-SUGAR, using in silico method with a virtual patient model developed by Hovorka et al. [51,52]. Lee et al. ran a three-day simulation and the results were compared through the hyperglycemia and hypoglycemia index, blood glucose concentrations, insulin doses, intravenous glucose infusion rates, and glucose feed rates [51]. Lee et al. concluded that the in silico method was useful for predicting hypoglycemic episodes [51]. Lonergan et al. 18 described a method of simulating blood glucose control and comparison of insulin protocols using 19 virtual patients derived from retrospective data [85]. However, the insulin protocols used by Lonergan et al. were not computer-based clinical protocols [85]. Similar studies by Wilinska et al. and Lee et al. suggested that their method also lacked a formal critiquing model for the insulin doses recommended by competing protocols and its clinical impact. Therefore, we were not able to adequately assess which protocols is better for future clinical trial evaluations. 2.7 Decision-making and Multi-attribute Utility Theory Humans have limited ability to process information and make an informed decision within a given period of time. According to Halford et al., the number of variables that a person can mentally handle while solving a problem is four, at most five [89]. Healthcare practitioners are constantly making clinical decisions that have important implications to their patient outcomes [90]. However, most healthcare practitioners have difficulties handling large amounts of information given the constraints in a stressful clinical environment [90-93]. Poor clinical decisions can lead to adverse events, medical errors, and even death [94,95]. We often make decisions in our daily lives by considering simple criteria implicitly. Sometimes we are comfortable with the consequences of such decisions. However, when the stakes are high such as clinical decisions, we need to evaluate the problem and their criteria explicitly. This can lead to a more informed and better decision. We evaluate our decisions by weighing the options available to us. We try to make the best decision based on some standard of what is good or bad. Decision theory, as proposed by researchers and philosophers across different disciplinary fields, attempts to guide us in 19 evaluating decisions [96]. Methods supported by decision theory, such as the various methods used by psychologists to study the behavior of decision-making, or the study of voting rules by political scientists, can be applied to similar problems in other fields. To understand decision theory, let us first consider an example of choosing a car. The simplest case is when there is only one attribute to choose from, e.g., the look of the car. This is an example where the attribute being subjective to the decision maker. Suppose that you like the look of car A better than car B, and you like car B better than car C. Clearly, you should buy car A. In reality, there are additional attributes to consider. Attributes for choosing a car may be related to safety, engine performance, fuel economy, number of passengers, and price. In most cases, these attributes are explicit and well defined. Not all of the attributes of making a decision are created equal. Some attributes may be considered more important. We can make a more informed decision by weighing these attributes accordingly. A simple mechanism to express the value of an attribute is to use relative terms such as "better than", "worse than", or "equally good" [97]. They are used to compare two alternatives. However, this is not adequate when we have multiple candidates to consider. Another method of expressing the value of an attribute is to assign numerical values. The advantage of using numerical value is that we can evaluate the attributes mathematically. Multi-attribute utility theory is a structured methodology for evaluating and comparing alternatives when making an important decision [98]. A utility is defined as a measure of references or value satisfying a set of attributes. The multi-attribute utility theory is designed to find the most optimal choice by quantifying the desirability of each of these alternatives through its attributes. Each attribute is measured through a utility using the 20 same numerical scale. This allows the comparison and evaluation of many diverse and disparate attributes such as cost, fuel economy, and safety. The end result is a rank ordered evaluation of alternatives that reflects the decision makers' preferences. Multi-attribute utility model allows decision makers to explore different ways of evaluating the alternatives by adjusting the weights assigned to the attributes. Since the criteria are known to the decision makers, the weights can be adjusted depending on the importance of the attributes to yield different results. The advantage of using a multiattribute utility model is that many points of view can be taken into consideration when making a group decision. The basis on which the alternatives are being compared and evaluated is made transparent to all parties involved. The multi-attribute utility model is most effective when the group of decision makers can come to a consensus on the attributes in the model. 2.8 Social Choice Theory and Voting System Social choice theory is a study of collective decision processes and procedures [99]. Kenneth Arrow, one of the main proponents of this theory, developed a theoretical framework to analyze how we can combine individual opinions and preferences to form a collective decision [99]. Individual preferences can be modeled as a utility function. It is assumed that the individuals have a preference over all the alternatives in a particular order. The social welfare function will then aggregate these individual preferences in such a way as to maximize the social utility through the sum of individual utilities [99]. In a democratic process, the voting mechanism is typically used to determine the decision for the group [99]. Essentially, voting facilitates social choice in a market place where individuals are considered capable of making an independent decision. A voting 21 system has a set of rules which must be adhered to for a vote to be considered valid, and a method on how votes are being processed to get the final result [100]. Common voting systems are majority rule and proportional representation. The majority rule or plurality system is a voting system where the candidate who received more votes than any other candidates will win the election [100]. This may be referred as winner-take-all system. In this system, an underrepresented candidate does not have a chance to win a mandate. This may inevitably lead to only major players remaining on the table. Major players could also use gerrymandering tactics to influence or manipulate the electoral results. The proportional representation system is a voting system that allows candidates to be represented proportionally according to the vote received [100,101]. This will create more competition and give more voices to minor candidates. If there are only two candidates, the winner can simply be determined by using the majority voting system. However, when there are multiple candidates, a single winner may not be an ideal solution. Different voting systems may give different results. 2.8.1 Single-winner Methods 2.8.1.1 Single Voting In a single voting method, each voter is allowed to pick only one candidate at a time. The most common single voting method is called plurality or winner-takes-all. The candidate with the most votes wins, regardless of whether the candidate receives a majority of the votes. Runoff methods are used when the winner needs to be elected by the majority. Multiple rounds of plurality voting are conducted for this purpose. 22 2.8.1.2 Ranked Voting In a ranked voting system, each voter ranks the candidates in the order of their preferences. A score is given to each candidate based on their rank position [102]. This method is also known as the positional voting method [102]. Any distribution of points to the rank positions is valid as long as the value of the higher rank is worth more than the lower rank. The scores corresponding to the voters' preferences are then aggregated for the final score. The candidate with the highest score is the winner. Although there is only one winner in this method, other candidates can still be considered as a substitute because of their rank positions. The standard positional voting method is called Borda count [102]. In a single-winner election with N candidates, the most preferred candidate will receive N points, followed by N-1 for the second preference, and so on. The point value can be defined as: 𝑣 = 𝑎 − (𝑟 − 1)𝑑 (2-1) where v is the point value a is the weighting of the first preference r is the rank position d is the common difference between the ranks The following (see Table 2.1) is an example where the weighting of the first preference, a, is equal to the number of candidates, N. Alternatively (see Table 2.2), the number of points each candidate receives can be the number of candidates ranked below them. The most preferred candidate will receive N - 1 points, followed by N-2 for second preferred candidate, and so on, with the last candidate 23 Table 2.1: An example of Borda count with five candidates Ranking Candidate Formula Points 1st Candidate A N 5 2nd Candidate B N-1 4 3rd Candidate C N-2 3 4th Candidate D N-3 2 5th Candidate E N-4 1 Table 2.2: An example of Borda count with N-i points Ranking Candidate Formula Points 1st Candidate A N-1 4 2nd Candidate B N-2 3 3rd Candidate C N-3 2 4th Candidate D N-4 1 5th Candidate E N-5 0 24 receiving zero points. Therefore, a candidate ranked in the ith place receives N-i points. 2.8.2 Multiple-winner Methods 2.8.2.1 Proportional Method The proportional method gives opportunity to all candidates to get some form of representation based on the votes they received [101]. In legislation, the most common proportional systems are based on party-list proportional representation. Voters vote for parties instead of individual candidates. Seats are then allocated according to the proportion of votes each party receives. There are different methods to determine the number of votes assigned to a seat, also known as quota. The methods of seat allocation can be grouped into highest averages methods and largest remainder methods [103]. 3 3.1 3.1.1 METHODS1 Data Sources Data Collection The data source for the in silico comparison was the electronic medical record (EMR) database from patients admitted into LDS Hospital and Intermountain Medical Center in Salt Lake City, Utah. We extracted the data from Intermountain Healthcare's HELP system, a health information system with an integrated clinical data repository. The HELP system stored chronological clinical data when eProtocol-insulin was used to manage patients with stress hyperglycemia. We included patients admitted into the ICU from 2004 to 2010 who were at least 14 years old, had stress hyperglycemia, and were managed by a version of eProtocol-insulin whose blood glucose target range was 80-110 mg/dL [56]. 3.1.2 Inclusion and Exclusion Criteria We included data for a group of patients who were supported by eProtocol-insulin in a single clinical encounter that contained more than five complete records of blood glucose and associated data. We extracted patient demographic records, blood glucose measurements, continuous IV insulin infusion rate, nutrition, IV propofol infusion rates 1 Figures 3.6, Equations 3.1 and 3.2 were reproduced and adapted with permission from Anthony F. Wong et al. An in silico method to identify computer-based protocols worthy of clinical study: An insulin infusion protocol use case. Journal of the American Medical Informatics Association (2016) 23 (2): 283288. Published by Oxford University Press on behalf of AMIA online at: https://academic.oup.com/jamia/article/23/2/283/2572377/An-in-silico-method-to-identify-computer-based 26 (because propofol's caloric value was used by Glucosafe), and presence and types of diabetes mellitus. Glucosafe uses quantified nutrition for computation of IV insulin infusion rate recommendation, whereas eProtocol-insulin does not. One of our computer-based insulin infusion protocol candidates, Glucosafe, requires more input data than eProtocol-insulin. To maximize the validity of results in our in silico comparison by using the same data for all candidates, we required a complete data set. We therefore eliminated a large number of records using the exclusion criteria decscribed below. We excluded patient cohorts who were supported by different versions of eProtocolinsulin when they were transferred between hospitals. Glucosafe requires nutritional input for its IV insulin infusion rate recommendation. Therefore, we excluded patients that were neither given enteral nor total parenteral nutrition so that we have a complete data set. We excluded patients who had incomplete blood glucose measurements and insulin therapy data. We excluded patients whose records had missing information about clinician's acceptance of eProtocol-insulin recommendations because we did not know their decisions at the bedside. We excluded patients who had missing eProtocol-insulin recommendations, or had five or less recorded observations because we needed this minimal number of sequential decisions for our evaluation algorithms. We excluded patients with recorded propofol infusion rates exceeding 200 mcg/kg/min. We believed this amount was simply too excessive and not reflective of the clinical decision made at the bedside. We excluded patients with two sequential measurements of blood glucose more than 12 hours apart, to ensure uninterrupted management of blood glucose with eProtocol-insulin. 27 3.2 Batch Comparison with Computer-based Clinical Protocol Candidates 3.2.1 HWCIR Glucose Protocol The HWCIR Glucose Protocol program was modified to accept batched sequential data input from our curated data set. The output of the HWCIR Glucose Protocol program was the recommended continuous IV infusion rate. 3.2.2 Glucosafe The Glucosafe program was modified to accept batched sequential data input. The output of Glucosafe was an insulin sensitivity estimate and recommendations for a continuous IV insulin infusion rate and an IV insulin bolus. 3.2.3 Atlanta Medical Center Protocol We adopted this table-based protocol and developed our own computer-based version as Computer-based Insulin Infusion Protocol for Atlanta Medical Center Protocol (CIIPAMC). The following is a general outline of the rules of the protocol:  Start the insulin infusion using the drip rate for current blood glucose range.  Subsequent insulin infusion rate is determined by comparing the current blood glucose range and previous blood glucose range.  If the current blood glucose range is lower than the previous blood glucose range, stay in the same column of insulin dosing.  If the current blood glucose range is the same or higher than the previous blood glucose range, move one column to the right of insulin dosing.  When blood glucose is in the target range (80-110 mg/dL), stay in the same column 28 to determine the new insulin infusion rate.  If blood glucose is less than 80 mg/dL, move one column to the left. We did not implement the rules for rechecking blood glucose measurements and administration of concentrated glucose since they were not within the scope of this study evaluation. 3.2.4 Thomas Jefferson Insulin Infusion Protocol We adopted this protocol and developed our own computer-based version as Computer-based Insulin Infusion Protocol for Thomas Jefferson Insulin Infusion Protocol (CIIP-TJIIP). The following is a general outline of the rules of the protocol:  The protocol limits the amount of insulin a patient can receive to a maximum of 20 U/h.  The insulin infusion rate is determined via a table lookup using the current blood glucose and rate of change from the previous blood glucose level. We did not implement the rules for rechecking blood glucose measurements and administration of oral or concentrated glucose since they were not necessary for this study evaluation. 3.2.5 NICE-SUGAR Protocol We adopted this protocol and developed our own computer-based version as Computer-based Insulin Infusion Protocol for Nice-Sugar Protocol (CIIP-NS). The following is a general outline of the rules of the protocol:  The calculation of the recommended IV insulin infusion rate depends on the previous IV insulin rate, current blood glucose level, and previous blood 29 glucose level.  The amount of IV insulin infusion rate to recommend depends on whether the patient was previously on insulin. We did not implement the rules for rechecking blood glucose measurements and bolus administration of 50% glucose since they were not within the scope of this study. We also did not implement the action of giving the patient a stat dose of insulin as it was optional and at the discretion of the attending physician. 3.3 3.3.1 In silico Framework Conceptual Design We developed a conceptual framework for our in silico method to compare and evaluate computer-based clinical protocols. The framework (see Figure 3.1) has three main components: (i) computer-based clinical protocols, (ii) data and physiological process simulation, and (iii) performance comparison and evaluation. The main purpose of the framework is to guide the design of a system for the comparison and evaluation of multiple computer-based clinical protocols. Our goal is to provide a framework for comparing any type of computer-based clinical protocols including those that are being used currently in the clinical setting or newly developed system. The framework was designed to support computer-based clinical protocols that were built on different computer platforms. This is important because many of these computer-based clinical protocols were developed by researchers based on the enterprise system used in their environment or familiarity with certain programming language. 30 Computer-based Clinical Protocols Performance Comparison and Evaluation Data and Physiological Process Simulation Figure 3.1: Conceptual framework for in silico comparison and evaluation of computerbased clinical protocols 3.3.2 Features The framework's basic approach is to provide computer-based clinical protocols with access to specific physiological processes simulated with retrospective data derived from real patients. Figure 3.2 describes the simulation of patients responding to treatment when managed by computer-based clinical protocol candidates. A baseline assumption was that the model for the specific physiological process has to be developed with data from patients who were treated with a similar protocol. The specific physiological process depends on the type of computer-based clinical protocol being investigated. For example, we simulated the dynamics of blood glucose-insulin in order to compare computer-based insulin infusion protocols. Simulation was achieved by using real patient data to create a model to represent the dynamics of blood glucose-insulin. We predicted the next blood glucose level based on this model after a new insulin dose 31 Figure 3.2: Physiological process simulation interacting with computer-based clinical protocols was recommended by a competing computer-based insulin infusion protocol. Results from the computer-based clinical protocols were collected by the performance comparison and evaluation module for further analysis. This module performed statistical and quantitative analysis. We anticipated healthcare practitioners and researchers would want to critique the performance of the computer-based clinical protocols before adopting them in their clinical setting. The module is customizable and will allow users to specify their own rules to critique the analysis. The in silico framework was designed to support computer-based clinical protocols implemented in different computer platforms (Figure 3.3). The performance comparison and evaluation module needed to be able to evaluate results from all these computer-based clinical protocol candidates. 32 Figure 3.3: The comparison and evaluation module supporting multiple platforms 3.4 Favorability Scoring There are many ways the performance of a computer-based clinical protocol could be evaluated. We could attempt to quantify the ability of computer-based clinical protocols to achieve their primary purpose. For example, we could compare the continuous IV insulin infusion rates recommended by competing computer-based insulin infusion protocols by critiquing the outcome of their subsequent blood glucose level. Alternatively, we could measure the rate of adverse events or undesirable outcomes such as estimating the number of hypoglycemic or hyperglycemic cases within a certain blood glucose range. Generally, we defined a favorability score as the performance measure of a computerbased clinical protocol based on a set of predefined rules. We tended to focus our rules on clinical outcomes. For example, at low blood glucose (<80 mg/dL) and within the target range (80-110mg/dL), we preferred a lower continuous IV insulin infusion rate or no insulin at all to prevent hypoglycemia. At high blood glucose (>110 mg/dL), we preferred a higher continuous IV insulin infusion rate to bring down the blood glucose level. 33 3.5 Methods for Aim 1 The first aim of our research study was to investigate the feasibility of developing an in silico method for identifying promising computer-based clinical protocol. We identified computer-based insulin infusion protocols as the focus of our study because of our experience in managing stress hyperglycemia with eProtocol-insulin [37,47,56]. The focus for Aim 1 was to compare the performance of Glucosafe against eProtocol-insulin. We developed a proof of concept software based on the in silico framework to compare and evaluate our computer-based insulin infusion protocols. The following diagram described the major components of the proof of concept software and the flow of information between them (see Figure 3.4). The software comprised two major components: the patient module, and the performance comparison and evaluation module. The purpose of the patient module was to extract appropriate patient data from the EMR database and provide the patient data to Glucosafe (see Figure 3.4, step 1). Glucosafe batch processed these data by feeding the input data iteratively in chronological order (see Figure 3.4, step 2) to get the corresponding recommendations for continuous IV insulin infusion rates (see Figure 3.4, step 3 and Figure 3.5). The performance comparison and evaluation module processed the results after Glucosafe completed the batch processing (see Figure 3.4, step 4). During evaluation, Glucosafe may suggest an insulin bolus in addition to the recommendation of continuous IV insulin infusion rate. We converted the Glucosafe IV insulin bolus into its continuous IV insulin infusion rate equivalent (Insulinbolus_iv_equivalent), and added it to the continuous IV insulin infusion rate (InsulinGlucosafe_IV) to produce a total continuous IV infusion rate (InsulinGlucosafe_final), according to Equation (3-1) and Equation (3-2). 34 Figure 3.4: Proof of concept software to compare Glucosafe with eProtocol-insulin Figure 3.5: Input data and output response of Glucosafe 35 𝐼𝑛𝑠𝑢𝑙𝑖𝑛𝑏𝑜𝑙𝑢𝑠_𝑖𝑣_𝑒𝑞𝑢𝑖𝑣𝑎𝑙𝑒𝑛𝑡 (𝑈/ℎ) = 𝐼𝑛𝑠𝑢𝑙𝑖𝑛𝑏𝑜𝑙𝑢𝑠 (𝑈) 𝑡𝑖𝑚𝑒 𝑑𝑖𝑓𝑓𝑒𝑟𝑒𝑛𝑐𝑒 𝑏𝑒𝑡𝑤𝑒𝑒𝑛 𝑡𝑤𝑜 𝑠𝑒𝑞𝑢𝑒𝑛𝑡𝑖𝑎𝑙 𝑏𝑙𝑜𝑜𝑑 𝑔𝑙𝑢𝑐𝑜𝑠𝑒 𝑚𝑒𝑎𝑠𝑢𝑟𝑒𝑚𝑒𝑛𝑡𝑠 (ℎ) (3-1) 𝐼𝑛𝑠𝑢𝑙𝑖𝑛𝐺𝑙𝑢𝑐𝑜𝑠𝑎𝑓𝑒_𝑓𝑖𝑛𝑎𝑙 = 𝐼𝑛𝑠𝑢𝑙𝑖𝑛𝐺𝑙𝑢𝑐𝑜𝑠𝑎𝑓𝑒_𝑖𝑣 + 𝐼𝑛𝑠𝑢𝑙𝑖𝑛𝑏𝑜𝑙𝑢𝑠_𝑖𝑣_𝑒𝑞𝑢𝑖𝑣𝑎𝑙𝑒𝑛𝑡 (3-2) eProtocol-insulin used blood glucose measurements and continuous IV insulin infusion rates at times ti and ti-1, to generate a new continuous IV insulin infusion rate recommendation at time ti (Figure 3.6). Glucosafe used blood glucose measurements and continuous IV insulin infusion rates at time ti and at all previous times to generate a new recommendation. For outcome evaluations, we used a moving window of width determined by two sequential times, ti and ti+1. The blood glucose at time=ti+1 measured during the intervention by eProtocol-insulin was the outcome of the continuous IV insulin infusion rate recommended and given at time=ti (see Analysis of Glucose at time=ti+1 in Figure 3.6). We used this blood glucose measurement at time=ti+1 to compare the potential outcome of the insulin recommendation by Glucosafe with eProtocol-insulin. Comparison was performed iteratively over each of the accepted eProtocol-insulin recommendations. We defined the following blood glucose ranges at time=ti+1 for the analysis:  Low: < 80 mg/dL  On target: 80 - 110 mg/dL  High: > 110 mg/dL We used these subsequent blood glucose measurement ranges at time=ti+1 to identify 36 Figure 3.6: Temporal characteristics of eProtocol-insulin and Glucosafe. Times of data used for continuous IV insulin infusion rate recommendation and time of blood glucose used for assessment of the appropriateness of the continuous IV insulin infusion rate recommendation. if the continuous IV insulin infusion rate recommendation at time=ti was too high, appropriate, or too low. The 80-110 mg/dL blood glucose target range was the target in the original eProtocol-insulin clinical application that provided the clinical data for our computer-based protocol comparison. We used the following evaluation strategy to identify which of the two continuous IV insulin infusion rate recommendations (from eProtocol-insulin or Glucosafe) was more favorable, because it was more likely to bring the blood glucose at time=ti+1 closer to the blood glucose target range of 80-110 mg/dL:  If the blood glucose measurement at time=ti+1 was low (<80 mg/dL), it is likely that the current continuous IV insulin infusion rate was higher than desired. The lower 37 of the two recommended continuous IV insulin infusion rates at time=ti was "more favorable" because it would likely have a lower risk of hypoglycemia.  If the blood glucose measurement at time=ti+1 was high (>110 mg/dL), it is likely that the continuous IV insulin infusion rate was lower than desired. The higher of the two recommended continuous IV insulin infusion rates at time=ti was "more favorable" because it may prevent hyperglycemia.  When blood glucose at time=ti+1 was within target (80-110 mg/dL), the lower of the two recommended continuous IV insulin infusion rates at time=ti was "more favorable" because it would likely have a lower risk of hypoglycemia. We used two methods to determine if the two recommended continuous IV insulin infusion rates were "equivalent": if they were equal (analysis "a") or if the higher infusion rate was within 10% of the lower infusion rate (analysis "b") (Figure 3.6, "a" and "b"). 10% was chosen arbitrary based on clinical heuristic. The following is an example of favorability scoring (see Figure 3.7) when the observed blood glucose level at time=ti+1 was deemed high (160 mg/dL) after the patient was given an insulin dose of 2.0 U/h as recommended by eProtocol-insulin at time=ti. If the continuous IV insulin infusion rate recommended by a computer-based insulin infusion protocol candidate at time=ti is lower than eProtocol-insulin, we think that the patient's blood glucose level will drop at a slower rate compared to eProtocol-insulin. This is less desirable. However, if the continuous IV insulin infusion rate recommended by the computer-based insulin infusion protocol candidate is higher, we think this could bring the blood glucose level down to the target range sooner. We consider this to be more favorable. In another example (see Figure 3.8), we considered a situation where the observed 38 Figure 3.7: An example of favorability scoring when the observed blood glucose at time=ti+1 is high. Figure 3.8: An example of favorability scoring when the observed blood glucose at time=ti+1 is in the target range. 39 blood glucose level at time=ti+1 was in the target range (90 mg/dL) after the patient was given a continuous IV insulin infusion rate of 1.0 U/h as recommended by eProtocolinsulin at time=ti. If the recommended continuous IV insulin infusion rate by a computerbased insulin infusion protocol candidate is higher than eProtocol-insulin, we thought this would cause the blood glucose level to drop even further. We preferred a lower insulin dose at this point in order to prevent a case of hypoglycemia. We defined continuous IV insulin rates as clinically equivalent when they are within 10% of each other (10% rule; (b) in Figure 3.6) in some analyses. If we applied the 10% rule, we allow the recommended continuous IV insulin infusion rate by a computer-based insulin infusion protocol candidate (Glucosafe) to fall within the 10% of the recommendation from eProtocol-insulin in order to determine their favorability as equivalent to eProtocol-insulin. 3.5.1 Performance Comparison and Evaluation We analyzed the results using two different methods: statistics and favorability scoring. First, we analyzed the recommended continuous IV insulin infusion rates at time=ti (a continuous variable) from eProtocol-insulin and Glucosafe by performing a simple descriptive statistics (mean and median). We also measure the statistical dispersion of the distribution by measuring the standard deviation and interquartile range (IQR). As for favorability scoring, a point is given to the computer-based insulin infusion protocol (eProtocol-insulin or Glucosafe) for every iteration of the data set that has a more favorable recommended continuous IV insulin infusion rate. The results were then stratified for comparison according to the blood glucose range at time=ti+1 (low, on target, high). 40 3.5.2 Statistical Analysis 3.5.2.1 One Sample Z-test for Proportion (for Distributions of IV Insulin Infusion Rates) We conducted one-sample z-tests for proportions to assess the proportion of eProtocolinsulin and Glucosafe pairs of recommended IV continuous insulin infusion rates that were not equivalent at time=ti+1. We assessed if the more favorable fractions for eProtocolinsulin or for Glucosafe (categorical variables) at time=ti+1 were significantly different from 0.5 expected from chance alone. We evaluated more favorable fractions for three categories of blood glucose measurement at time=ti+1: low (< 80 mg/dL), on target (80-110 mg/dL with equivalence analyses (a) and (b)), and high (>110 mg/dL). 3.5.2.2 Wilcoxon Signed Rank Test (for favorability scores) We used the Wilcoxon signed rank test to compare the distributions of recommended continuous IV insulin infusion rates at time=ti. 3.6 Methods for Aim 2 The second aim of our study was to expand the in silico method to analyze and evaluate multiple computer-based clinical protocol candidates. The methods as described in Aim 1 were limited to analyzing two computer-based insulin infusion protocols at a time. The methods were also limited to comparison with eProtocol-insulin because eProtocol-insulin became the source of data for the analysis. When addressing Aim 2, we used a ranking strategy as a favorability scoring algorithm to compare multiple candidates of computerbased insulin infusion protocols. In this study, we compared and analyzed six computer-based insulin infusion 41 protocols:  eProtocol-insulin  Glucosafe  HWCIR Glucose Protocol  Atlanta Medical Center Protocol  Thomas Jefferson Insulin Infusion Protocol  NICE-SUGAR Protocol We developed the ranking strategy for favorability scoring based on a ranked voting system using Borda count. We preferred this strategy as opposed to the single voting strategy because single voting strategies aggregate all but the "winner" into a single "losing" category and we wanted a way to compare relative merits of the computer-based insulin infusion protocols. This is useful because organizational or other factors may influence healthcare practitioners to decide on a different computer-based insulin infusion protocol other than the winner of our evaluation. We used a moving window of two sequential times, ti and ti+1, for our outcome evaluations. Each iteration of the candidates' recommended continuous IV insulin infusion rate was given a score by evaluating their rank in terms of favorability. We defined favorability as those more likely to bring the blood glucose at time = ti+1 to the blood glucose target range of 80-110 mg/dL. The strategy required us to retrospectively compare recommended continuous IV insulin infusion rate at time=ti relative to the subsequent blood glucose measurement at time=ti+1. The following describes how recommended continuous IV insulin infusion rates were ranked according to their favorability:  If the blood glucose measurement at time = ti+1 was low (< 80 mg/dL), the 42 continuous IV insulin infusion rate was higher than desired. A lower continuous IV insulin infusion rates at time = ti recommended by the candidates was deemed "more favorable" because it would likely have the lower danger of hypoglycemia.  If the blood glucose measurement at time=ti+1 was high (> 110 mg/dL), the continuous IV insulin infusion rate was lower than desired and a higher recommended continuous IV insulin infusion rates at time = ti was "more favorable."  When blood glucose at time = ti+1 was within target (80-110 mg/dL), the lower recommended continuous IV insulin infusion rates at time = ti was "more favorable" because it would likely have the lower danger of hypoglycemia. We used a standard competition ranking strategy to assign the rankings. The computerbased insulin infusion protocol candidate that was considered more favorable will receive a higher rank. Candidates that recommended exactly the same amount of continuous IV insulin infusion rates will receive the same ranking number. For example, if A ranks ahead of B and C (compare equal), followed by D, then A gets ranking number 1 (first), B and C get ranking number 2 (joint second), and D gets ranking number 4 (fourth). Finally, we converted the ranking into a score using the Borda count formula (see Equation (2-1)) with 𝑎 = 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑐𝑎𝑛𝑑𝑖𝑑𝑎𝑡𝑒𝑠 − 1 Therefore 𝑆𝑐𝑜𝑟𝑒𝑐𝑎𝑛𝑑𝑖𝑑𝑎𝑡𝑒 = (𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑐𝑎𝑛𝑑𝑖𝑑𝑎𝑡𝑒𝑠 − 1) − (𝑅𝑎𝑛𝑘𝑐𝑎𝑛𝑑𝑖𝑑𝑎𝑡𝑒 − 1) 43 𝑆𝑐𝑜𝑟𝑒𝑐𝑎𝑛𝑑𝑖𝑑𝑎𝑡𝑒 = 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑐𝑎𝑛𝑑𝑖𝑑𝑎𝑡𝑒𝑠 − 𝑅𝑎𝑛𝑘𝑐𝑎𝑛𝑑𝑖𝑑𝑎𝑡𝑒 3.7 (3-3) Methods for Aim 3 The third aim of our study was to design a method of aiding our healthcare practitioners to make the right decision when choosing a computer-based clinical protocol for an upcoming clinical trial. The key to this design was to develop a multiple-criteria decision analysis with input from healthcare practitioners. In this study, we also compared and analyzed six computer-based insulin infusion protocols:  eProtocol-insulin  Glucosafe  HWCIR Glucose Protocol  Atlanta Medical Center Protocol  Thomas Jefferson Insulin Infusion Protocol  NICE-SUGAR Protocol 3.7.1 Estimating the Subsequent Blood Glucose Level There was a limitation with the methods described in Methods for Aim 1 and Methods for Aim 2. We operated under the assumption that the subsequent blood glucose (at time=ti+1) remain the same during the analysis for those methods mentioned above. We were not able to determine if the amount of continuous IV insulin infusion rate recommended by the computer-based insulin infusion protocol candidates was considered too little or excessive. This may put the patient at a higher risk for developing 44 hypoglycemia or hyperglycemia depending on the amount of continuous IV insulin infusion rate. We can improve the favorability scoring by predicting the subsequent blood glucose level when a different continuous IV insulin infusion rate is given. In previous studies, the prediction of subsequent blood glucose level was calculated by using insulin sensitivity profiles [85,86]. These insulin sensitivity profiles, a dimensionless factor, were estimated from parameters (e.g., nutrition, insulin dose, blood glucose measurement) obtained from patients in a clinical trial. The insulin sensitivity estimate varies between blood glucose measurements. The relationship between insulin doses and change in blood glucose was not clear either (see Figure 3.9). Many complex mathematical models have been theorized to describe the dynamics of blood glucose and insulin including ordinary differential equations (ODEs), partial differential equations (PDEs), and stochastic models [73,83,84]. We chose a simple linear model to describe the relationship between insulin and blood glucose in our first prototype. In this study, we used the eProtocol-insulin's observed rate of change of blood glucose per unit of insulin at time=ti to estimate the subsequent blood glucose level for other computer-based infusion protocol candidates at time=ti+1. First, we compute the rate of change of blood glucose for every unit of insulin recommended by eProtocol-insulin. 𝛿𝐵𝐺𝑒𝑃𝑟𝑜𝑡𝑜𝑐𝑜𝑙−𝑖𝑛𝑠𝑢𝑙𝑖𝑛 𝑏𝑙𝑜𝑜𝑑 𝑔𝑙𝑢𝑐𝑜𝑠𝑒𝑡+1 − 𝑏𝑙𝑜𝑜𝑑 𝑔𝑙𝑢𝑐𝑜𝑠𝑒𝑡 = 𝛿𝑡𝑖 (𝑡𝑖𝑚𝑒 = 𝑡𝑖+1 ) − (𝑡𝑖𝑚𝑒 = 𝑡𝑖 ) Hence, the rate of change for blood glucose per unit of insulin is: (3-4) 45 Figure 3.9: An example of association between blood glucose and insulin in one patient over time 46 𝑑𝐵𝐺_𝐼𝑒𝑃𝑟𝑜𝑡𝑜𝑐𝑜𝑙−𝑖𝑛𝑠𝑢𝑙𝑖𝑛 𝛿𝐵𝐺𝑒𝑃𝑟𝑜𝑡𝑜𝑐𝑜𝑙−𝑖𝑛𝑠𝑢𝑙𝑖𝑛 𝛿𝑡𝑖 = 𝑖𝑛𝑠𝑢𝑙𝑖𝑛 𝑖𝑛𝑓𝑢𝑠𝑖𝑜𝑛 𝑟𝑎𝑡𝑒 (𝑒𝑃𝑟𝑜𝑡𝑜𝑐𝑜𝑙 − 𝑖𝑛𝑠𝑢𝑙𝑖𝑛) 𝑡 (3-5) We assumed the rate of change for blood glucose per unit of insulin dose remains the same for this patient when treated with another computer-based insulin infusion protocol candidate. 𝑑𝐵𝐺_𝐼𝑐𝑎𝑛𝑑𝑖𝑑𝑎𝑡𝑒 = 𝑑𝐵𝐺_𝐼𝑒𝑃𝑟𝑜𝑡𝑜𝑐𝑜𝑙−𝑖𝑛𝑠𝑢𝑙𝑖𝑛 (3-6) Then, we calculated the new rate of change based on a new dose recommendation by the computer-based insulin infusion protocol candidate. Finally, the blood glucose level at time=ti+1 was estimated to be: 𝑒𝑠𝑡_𝐵𝐺𝑐𝑎𝑛𝑑𝑖𝑑𝑎𝑡𝑒 = 𝑏𝑙𝑜𝑜𝑑 𝑔𝑙𝑢𝑐𝑜𝑠𝑒𝑡 + (𝑑𝐵𝐺_𝐼𝑒𝑃𝑟𝑜𝑡𝑜𝑐𝑜𝑙−𝑖𝑛𝑠𝑢𝑙𝑖𝑛 (3-7) × 𝑖𝑛𝑠𝑢𝑙𝑖𝑛𝑐𝑎𝑛𝑑𝑖𝑑𝑎𝑡𝑒 × 𝛿𝑡𝑖 ) In the following example (see Figure 3.10), the patient's blood glucose had reduced from 180 mg/dL to 120 mg/dL after she was given 2.0 U/h of continuous IV insulin infusion, as observed in the eProtocol-insulin clinical trial. A computer-based insulin infusion protocol candidate recommended a higher insulin dose. The favorability scoring in Methods of Aim 1 would have judged the candidate more favorably because higher continuous IV insulin infusion rate is preferred. However, this was not necessarily true if 47 Figure 3.10: An example of estimating the subsequent blood glucose to determine the favorability the recommended continuous IV insulin infusion rate is too high for the same time period. The descent of blood glucose level may be too steep. Again, if we assume that healthcare practitioners will only return at time=ti+1 to check on their patient, the blood glucose level now may be too low or possibly hypoglycemic. We estimated the subsequent blood glucose level at time=ti+1 by calculating the rate of change of blood glucose in that patient data due to insulin. To illustrate this, we created a simple input data for estimating the subsequent blood glucose level in Table 3.1. First, we calculated the rate of change of blood glucose during the time window of ti and ti+1 according to Equation (3-4). 48 Table 3.1: An example input data for estimating the subsequent blood glucose level for a computer-based insulin infusion protocol candidate Candidate insulin eProtocol- Time t1 t2 Blood glucose (mg/dL) 220 115 Continuous IV insulin infusion rate given (U/h) 1.6 Elapsed time (h) Continuous 1.5 IV insulin infusion rate 2.2 recommended by candidate (U/h) 𝛿𝐵𝐺𝑒𝑃𝑟𝑜𝑡𝑜𝑐𝑜𝑙−𝑖𝑛𝑠𝑢𝑙𝑖𝑛 𝑏𝑙𝑜𝑜𝑑 𝑔𝑙𝑢𝑐𝑜𝑠𝑒𝑡+1 − 𝑏𝑙𝑜𝑜𝑑 𝑔𝑙𝑢𝑐𝑜𝑠𝑒𝑡 = 𝛿𝑡𝑖 (𝑡𝑖𝑚𝑒 = 𝑡𝑖+1 ) − (𝑡𝑖𝑚𝑒 = 𝑡𝑖 ) = 115 − 220 1.5 = −70 𝑚𝑔 /ℎ 𝑑𝐿 Then, we calculated the rate of change for blood glucose per unit of insulin according to Equation (3-5): 𝑑𝐵𝐺𝐼 𝑒𝑃𝑟𝑜𝑡𝑜𝑐𝑜𝑙−𝑖𝑛𝑠𝑢𝑙𝑖𝑛 𝛿𝐵𝐺𝑒𝑃𝑟𝑜𝑡𝑜𝑐𝑜𝑙−𝑖𝑛𝑠𝑢𝑙𝑖𝑛 𝛿𝑡𝑖 = 𝑖𝑛𝑠𝑢𝑙𝑖𝑛 𝑖𝑛𝑓𝑢𝑠𝑖𝑜𝑛 𝑟𝑎𝑡𝑒 (𝑒𝑃𝑟𝑜𝑡𝑜𝑐𝑜𝑙 − 𝑖𝑛𝑠𝑢𝑙𝑖𝑛) 𝑡 = −70 1.6 = −43.75 49 If we assume the computer-based insulin infusion protocol candidate recommended is 2.2 U/h at time=t1, the estimated blood glucose level at time=t2 for the computer-based insulin infusion protocol candidate according to Equation (3-7) would be: 𝑒𝑠𝑡𝐵𝐺 𝑐𝑎𝑛𝑑𝑖𝑑𝑎𝑡𝑒 = 𝑏𝑙𝑜𝑜𝑑 𝑔𝑙𝑢𝑐𝑜𝑠𝑒𝑡 + (𝑑𝐵𝐺𝐼 𝑒𝑃𝑟𝑜𝑡𝑜𝑐𝑜𝑙−𝑖𝑛𝑠𝑢𝑙𝑖𝑛 × 𝑖𝑛𝑠𝑢𝑙𝑖𝑛𝑐𝑎𝑛𝑑𝑖𝑑𝑎𝑡𝑒 × 𝛿𝑡𝑖 ) = 220 + (−43.75 × 2.2 × 1.5) = 𝟕𝟔 𝒎𝒈/𝒅𝑳 The patient is now fast approaching the hypoglycemic range (see Figure 3.11). Although this result may have been more favorable using the previous favorability analysis (higher insulin infusion rate was preferred), the estimate of subsequent blood glucose demonstrated that the observation may be fast approaching hypoglycemia. This is not desirable. 3.7.2 Multiple Criteria Decision Analysis We developed several criteria to help healthcare practitioners decide which of the computer-based insulin infusion protocol candidates is suitable for clinical trial. The criteria were also designed to probe the behavior and performance of the computer-based insulin infusion protocol candidates. 3.7.2.1 Ranked Favorability Scoring The objective of the ranked favorability scoring was to analyze the perceived favorability for each computer-based insulin infusion candidate by comparing the appropriateness of the recommended continuous IV insulin infusion rate at time=ti relative 50 Figure 3.11: The estimated blood glucose at time=t2 due to candidate's recommended insulin to the subsequent blood glucose measurement at time=ti+1. We used the ranked favorability scoring as described in Methods for Aim 2. We selected the total aggregate score for the computer-based insulin infusion protocol candidates as our scoring criterion in this decision analysis. 3.7.2.2 Estimation of Hypoglycemia Rate The objective of this criterion was to anticipate the risk of hypoglycemia when intervening with the computer-based insulin infusion protocol candidate. We used this method to identify individual cases where the estimated subsequent blood glucose < 60 mg/dL. 51 3.7.2.3 Estimation of Hyperglycemia Rate The objective of this criterion was to anticipate the risk of hyperglycemia when intervening with the computer-based insulin infusion protocol candidate. We used this method to identify individual cases where the estimated subsequent blood glucose > 180 mg/dL. 3.7.2.4 Estimation of Cases Within the Target Range The objective of this criterion was to determine the percentage of cases where the blood glucose will fall within the target range when intervening with the computer-based insulin infusion protocol candidate. We used this method to identify individual cases where the estimated subsequent blood glucose is within the target range (80-110 mg/dL). 3.7.2.5 Mean of Recommended Continuous IV Insulin Infusion Rates The objective of this criterion was to measure the central tendency of the output from computer-based insulin infusion protocol candidates. We stratified the output into different blood glucose range categories and calculated the mean of the recommended continuous IV insulin infusion rates at time=ti for each candidate. 3.7.2.6 Median of Recommended Continuous IV Insulin Infusion Rates The objective of this criterion was to provide an alternative measure for the central tendency of the output from computer-based insulin infusion protocol candidates. The advantage of median is that it will not be influenced by outliers (extremely large or small values). The measure of median is one of the ways of summarizing the typical values associated with the output of the computer-based insulin infusion protocol candidates. 52 Similarly, we stratified the output into different blood glucose range categories and calculated the median of the recommended continuous IV insulin infusion rates at time=ti for each candidate. 3.7.2.7 Distributed Favorability Scoring The objective of this criterion was to measure the variability of the candidates in terms of how well the computer-based insulin infusion protocol candidate maintains the patient within the target range. Scores were assigned to each estimated subsequent blood glucose (method for subsequent blood glucose estimation was discussed in section 3.7.1) for all computer-based insulin infusion protocol candidates. The candidate received a higher score if their estimated blood glucose was closer to the blood glucose target range. We defined the our distributed favorability scoring as seen in Figure 3.12. 3.7.3 Aggregating Decision Scores The scores from all the criteria were totaled before presenting the final results to the healthcare practitioners. We calculated these scores based on ranked voting and Borda count method. We assigned ranks to each of the candidates for every criterion to get a uniform score during aggregation. Weights were then assigned to the criteria with input from healthcare practitioners. These weights represent the strength and importance of the criteria, from the perspective of the local clinical experts, when judging the performance of the computer-based clinical protocol. In some environments, for example, efficiency in reaching protocol targets may be the most important concern, whereas in other environments, the over-riding concern might be avoiding a specific adverse event (like hypoglycemia). The concept for these weight assignments was adapted from the multiple- 53 Figure 3.12: Assignment of scores for distributed favorability scoring winner proportional methods as described in the social choice theory (section 2.8). We preferred the multiple-winner methods as oppose to single-winner methods because we believe each of the computer-based clinical protocols has tangible value to healthcare practitioners. This method allowed us to judge each of the computer-based clinical protocol fairly with the input from healthcare practitioners who will be using them eventually at the bedside. 4 RESULTS AND DISCUSSIONS2 4.1 4.1.1 Aim 1 Results for Aim 1 We found 2,560 patients managed by eProtocol-insulin at the LDS Hospital and Intermountain Medical Center from 2004 to 2010. These patients were managed by five different versions of eProtocol-insulin (see Table 4.1). We found 38 patients enrolled in two versions of eProtocol-insulin. We decided to exclude patients with multiple versions of eProtocol-insulin. We focused our patient data on eProtocol-insulin version "DA 95(80110)Jjrt4cMR" because the target range is 80 to 110 mg/dL and has the most patients in the database. We extracted deidentified patient demographic records, blood glucose measurements, continuous IV insulin infusion rate, nutrition, IV propofol infusion rates, presence and types of diabetes mellitus, and nutritional data (enteral and total parenteral nutrition). Glucosafe requires more data input than eProtocol-insulin to make a continuous IV insulin infusion rate recommendation. The input data include IV propofol infusion rate, types of diabetes mellitus, and nutritional data. We found several data fields with incomplete data (see Table 4.2). They may be incomplete due to missing data, patients who were not 2 Figures 4.1, Tables 4.3, 4.4, 4.6, and 4.7 were reproduced and adapted with permission from Anthony F. Wong et al. An in silico method to identify computer-based protocols worthy of clinical study: An insulin infusion protocol use case. Journal of the American Medical Informatics Association (2016) 23 (2): 283-288. Published by Oxford University Press on behalf of AMIA online at: https://academic.oup.com/jamia/article/23/2/283/2572377/An-in-silico-method-to-identify-computer-based 55 Table 4.1: Number of patients in different versions of eProtocol-insulin Number of patients eProtocol-insulin Number of patients in version name each eProtocol-insulin excluding those with multiple enrolment DA 115(90-140)Jjrt4cMR 449 440 DA 115(90-140)JjrtEq4bBSA 29 21 1,375 1,351 DA 95(80-110)JjrtEq4bBSA 486 466 DA 95(80-110)JjrtEq4bMR 259 244 DA 95(80-110)Jjrt4cMR 56 Table 4.2: Completeness of patient data Number of patients having Patient data field % Complete the required data Age 2,560 100.0 Weight 2,560 100.0 Height 2,558 99.9 Gender 2,560 100.0 Has Diabetes Mellitus Code 1,027 40.2 Enteral nutrition 1,541 60.2 Blood glucose 2,560 100.0 2,560 100.0 380 14.8 Enteral OR TPN 1,635 63.9 IV Propofol infusion rate 1,048 40.9 Continuous IV insulin infusion rate Total parenteral nutrition (TPN) 57 diagnosed with any types of diabetes, or patients not given any of the enteral nutrition or TPN during their hospitalization. The actual reasons were not documented. Our goal was to use data associated with eProtocol-insulin continuous IV insulin infusion rate recommendations accepted by the bedside healthcare practitioners. We examined 118,377 eProtocol-insulin recommendations from 2,560 patients. While the target for data acquisition interval was two hours, the realities of delivering care in a clinical setting results in some timing variability (median=2.05 hours, standard deviation=0.81 hour, mean=2.10 hours). We eliminated the incomplete data using the exclusion criteria described in the methods section (see Figure 4.1). We excluded 2,152 patients initially using the exclusion criteria, leaving 408 patients with 20,770 eProtocol-insulin recommendations. We removed the 3.7% of 20,770 eProtocol-insulin recommendations rejected by bedside healthcare practitioners and used only eProtocol-insulin recommendations accepted by bedside healthcare practitioners. This was done to ensure that the data set from the observations reflected the decisions made by healthcare practitioners at the bedside. We also removed another 1,021 records because blood glucose at time= ti+1 was not available or because the records followed an eProtocol-insulin recommendation rejected by the bedside healthcare practitioners. The final result was a sample data set containing 408 patients with 18,984 eProtocol-insulin recommendations and associated data. We analyzed our study sample of 408 patients and found 11 patients with type 1 diabetes and 113 patients with type 2 diabetes. There were 241 males and 167 females in the study sample. The statistical description of the patients can be found in Table 4.3. We also analyzed the primary discharge diagnostic codes associated with the patients 58 Figure 4.1: CONSORT diagram of considered patients. 59 Table 4.3: Statistical description of patients Standard Min Max Mean Deviation Age (years) 14 95 49.5 20.3 Weight (kg) 39.5 275.8 86.3 26.0 Height (cm) 139.7 208.3 173.1 10.3 in the study sample (see Table 4.4). The majority of the patients in the data sample suffered from trauma or sepsis infections (216 of 408 patients). We analyzed the results of the recommended continuous IV insulin infusion rates from eProtocol-insulin and Glucosafe when blood glucose is at time=ti. They were clinically similar (see Table 4.5 and Figure 4.2) although statistically significantly different (Wilcoxon signed rank test p=0.01). We plotted the distributions of the recommended continuous IV insulin infusion rates by eProtocol-insulin and Glucosafe (see Figure 4.2). eProtocol-insulin appeared to recommend insulin infusion rates between 2 to 5 U/h more frequently than Glucosafe. Glucosafe recommended insulin infusion rates between 6 to 9 U/h slightly more frequently than eProtocol-insulin. When we stratified the time=ti continuous IV insulin infusion rates by the three blood glucose measurement categories at time=ti (<80, 80-110, and >110 mg/dL), the 60 Table 4.4: Primary discharge diagnostic categories for eProtocol-insulin patients in the data sample Primary Discharge Diagnostic Categories Number of patients Sepsis/Infection 90 Trauma 126 Pneumonitis 9 Respiratory, other 23 Cardiovascular 41 Abdominal 18 Liver 24 Gall Bladder/Pancreas 5 Malignancy 20 Diabetes Mellitus 2 Other Endocrine 1 Renal 7 Central Nervous System 2 Drug Overdose 21 Peripartum 4 Vasculitis 2 61 (Table 4.4: Continued) Primary Discharge Diagnostic Categories Other Number of patients 8 62 Table 4.5: Statistical results of recommended continuous IV insulin infusion rates between eProtocol-insulin and Glucosafe eProtocol-insulin Glucosafe Mean (U/h) 3.9 4.0 Median (U/h) 3.3 3.5 Standard deviation (U/h) 2.7 3.1 Minimum (U/h) 0 0 Maximum (U/h) 21.5 21.8 Figure 4.2: Distributions of recommended continuous IV insulin infusion rate by eProtocol-insulin and Glucosafe for all blood glucose measurements 63 recommended time=ti IV continuous insulin infusion rates from eProtocol-insulin and from Glucosafe were not only statistically significantly different (Wilcoxon signed rank test, p<0.001), but also appeared clinically different (see Figure 4.3). Finally, we analyzed the favorability using the evaluation strategy as outlined in the methods section. In analysis [a] (see Table 4.6), the continuous IV insulin infusion rate recommendations of eProtocol-insulin and Glucosafe are equivalent only when they are equal. In analysis [b] (see Table 4.6), the continuous IV insulin infusion rate recommendations of eProtocol-insulin and Glucosafe are equivalent only when they are within 10% of the lower continuous IV insulin infusion rate recommendation. Glucosafe's continuous IV insulin infusion rate recommendations were found to be more favorable than those of eProtocol-insulin in all three time=ti+1 blood glucose categories. Analysis [b] has a higher portion of continuous IV insulin infusion rate recommendations that were considered to be equivalent compared to analysis [a] (14% vs 5%). We further analyzed the continuous IV insulin infusion rate recommendations from eProtocol-insulin and Glucosafe by studying their statistical dispersion. We reported the medians and interquartile range (IQR) of the continuous IV insulin infusion rate recommendations because their distributions appeared skewed. We reported the mean and standard deviation of the pairwise differences (Glucosafe minus eProtocol-insulin) because their differences appeared normally distributed. For blood glucose < 80 mg/dL, Glucosafe recommended lower median rates of continuous IV insulin infusion (1.5 U/h) than eProtocol-insulin (3.4 U/h) and Glucosafe's recommendations were more favorable 80% of the time (see Table 4.6 and Table 4.7). This is consistent with the notion that lower continuous IV insulin infusion rate is preferable at 64 Figure 4.3: Distributions of recommended continuous IV insulin infusion rate by eProtocol-insulin and Glucosafe stratified by blood glucose measurements at time=ti 65 Table 4.6: Counts of more favorable continuous IV insulin infusion rate recommendations at time=ti from eProtocol-insulin or Glucosafe, based on three blood glucose categories at time=ti+1. Recommended Favorability Frequency Blood Glucose continuous IV category (mg/dL) eProtocolinsulin infusion rate at time=ti+1 Glucosafe Equivalent 273 (15%) 1470 (80%) 102 (5%) 2919 (31%) 5984 (64%) 453 (5%) 2483 (26%) 5573 (60%) 1300 (14%) 2045 (26%) 5473 (70%) 265 (4%) insulin at time=ti < 80 [a] (continuous IV insulin infusion rate recommendations exactly equal) 80-110 [b] (continuous IV insulin infusion rate recommendations ± 10% of the lower infusion rate) > 110 66 Table 4.7: Continuous IV insulin infusion rate recommendations and difference in recommended IV insulin infusion rate Pairwise continuous IV Continuous IV insulin infusion insulin infusion rate recommendation (U/h) at rate time=ti recommendation difference (U/h) at time=ti eProtocolBlood glucose category at Glucosafe (Glucosafe - Median eProtocol-insulin) (IQR) Mean (SD) insulin Count time=ti+1 Median (IQR) Low (< 80 mg/dL) 1845 3.4 (3.5) 1.5 (3.2) -1.8 (2.4) On target (80 - 110 mg/dL) 9356 3.3 (3.2) 2.8 (3.6) -0.8 (2.2) High (> 110 mg/dL) 7783 3.3 (2.8) 5.4 (4.7) 1.7 (2.9) 18,984 3.3 (3.1) 3.5 (4.6) 0.1 (2.8) Total 67 low blood glucose range. For blood glucose > 110 mg/dL, Glucosafe recommended higher median rates of continuous IV insulin infusion (5.4 U/h) than eProtocol-insulin (3.3 U/h) and Glucosafe recommendations were more favorable 70% of the time (see Table 4.6 and Table 4.7). This was consistent with the notion that higher continuous IV insulin infusion rate is preferable at higher blood glucose range. For blood glucose within the target range (80-110 mg/dL), Glucosafe recommended lower median rates of continuous IV insulin infusion (2.8 U/h) than eProtocol-insulin (3.3 U/h) and Glucosafe recommendations were more favorable 64% of the time for analysis (a) and 60% of the time for analysis (b) (see Table 4.6 and Table 4.7). Further, the proportion of continuous IV insulin infusion rate recommendations deemed more favorable (see Table 4.6) was significantly different from 0.5 for both Glucosafe and eProtocol-insulin in each of the three blood glucose categories (one-sample z-test, p<0.001). 4.1.2 Discussion for Aim 1 We have successfully performed an in silico comparison and evaluation of two computer-based insulin infusion protocols, eProtocol-insulin and Glucosafe, using a robust EMR database generated from previous use of eProtocol-insulin at the bedside. eProtocolinsulin has been proven to be effective in managing stress hyperglycemia in the ICU. This dependable clinician decision-making method allowed us to use the clinical EMR data to rigorously evaluate Glucosafe and assess its worthiness for expensive and resource consumptive evaluation in a clinical trial. The recommended continuous IV insulin infusion rates at time=ti for eProtocol-insulin and Glucosafe were found to be statistically significantly different even though they appeared to be clinically similar. However, when stratified by blood glucose categories 68 (low, on target, high) at time=ti+1, the continuous IV insulin infusion rate recommendations at time=ti were both statistically significantly different and the difference appeared clinically important (see Table 4.6 and Table 4.7). Glucosafe produced considerably more favorable continuous IV insulin infusion rate recommendations than eProtocol-insulin (see Table 4.6). In the low range, Glucosafe consistently recommended lower continuous IV insulin infusion rate than eProtocol-insulin and could potentially reduce the hypoglycemia rates. In the high range, Glucosafe recommended higher continuous IV insulin infusion rate more frequently than eProtocol-insulin. In other words, this could potentially lower the blood glucose level to the desired target range faster than eProtocol-insulin. These results suggested Glucosafe was the preferable computer-based insulin infusion protocol. The result in analysis [b], where we allowed the recommended continuous IV insulin infusion rate to stay within 10% of the other candidate in order to be considered equivalent, showed a significantly higher percentage compared to analysis [a]. We introduced this analysis as an experiment to see whether an interval band will affect the favorability scoring. The result suggested that, while the number of equivalents was higher in the 10% interval band, it did not affect the favorability scoring. Glucosafe still scored significantly better than eProtocol-insulin in this blood glucose category. The internal implementation of the computer-based insulin infusion protocols has played an important role in this outcome. Glucosafe was designed using a comprehensive physiologic algorithm that includes detailed nutritional information and effects of insulin sensitivity [57,58]. On the other hand, eProtocol-insulin was developed through years of protocol management in hypermetabolic stress. The insulin protocol used heuristic models and empiric rule sets with simple nutrition rules to manage blood glucose in ICU patients 69 [56]. We believed the detailed enteral and total parenteral nutritional information included in Glucosafe was one of the key factors why it had been able to recommend more favorable continuous IV insulin infusion rate compared to eProtocol-insulin. There were several limitations in the Methods for Aim 1. The pairwise comparative technique can only evaluate two computer-based insulin infusion protocols concurrently. In order to evaluate more than two computer-based insulin infusion protocols, we will need to compare them pairwise, repeatedly, for all combinations. This was certainly not the most efficient way to compare multiple computer-based insulin infusion protocols. Furthermore, it was not easy to judge their relative performance using this method. The version of Glucosafe in this evaluation only used the minimum amount of information that was required as we wanted to limit the scope of work. We think Glucosafe might perform even better if more clinical data were made available because the internal algorithm of Glucosafe allows more detailed computation. While we only compared two computer-based insulin infusion protocols for the ICU, the in silico method we developed seems generally applicable to other insulin infusion protocols. We demonstrated that the in silico method was an effective method to evaluate a computer-based insulin protocol candidate before committing the resources to conduct a clinical trial. 4.2 4.2.1 Aim 2 Results for Aim 2 The second aim of our study was to expand the in silico method to evaluate multiple computer-based clinical protocol candidates. We found several limitations to the methods described in Aim 1 when we were attempting to compare more than two candidates. 70 Previous methods required us to compare the candidates multiple times since only two candidates can be compared each time. To address this limitation, we used ranking strategy to rank and compare multiple computer-based clinical protocol candidates. In this study, we compared and analyzed six computer-based insulin infusion protocols:  eProtocol-insulin  Glucosafe  HWCIR Glucose Protocol  Atlanta Medical Center Protocol (CIIP-AMC)  Thomas Jefferson Insulin Infusion Protocol (CIIP-TJIIP)  NICE-SUGAR Protocol (CIIP-NS) We used the same data from our previous study in Aim 1 (18,984 eProtocol-insulin recommendations with its associated data from 408 patients). We reported the statistical results of the recommended continuous IV insulin infusion rates when blood glucose was measured at time=ti from all six computer-based insulin infusion protocol candidates (see Table 4.8). We plotted the distributions of the recommended continuous IV insulin infusion rates by all six computer-based insulin infusion protocol candidates (see Figure 4.4). They appeared clinically similar. We conducted the pairwise Wilcoxon signed rank test for all the computer-based insulin infusion protocol candidates (see Table 4.9). The majority of the pairwise comparison showed that the p-value was less than 0.0001. HWCIR Glucose Protocol and CIIP-AMC has p-value of 0.02. This p-value was still less than 5% (0.05 significance level). We concluded the distribution of these two computer-based insulin infusion 71 Table 4.8: Statistical results of recommended continuous IV insulin infusion rates by six computer-based insulin infusion protocols. eProtocol Glucosafe HWCIR CIIP- -insulin CIIP- Glucose CIIP-NS AMC TJIIP Protocol Mean (U/h) 3.9 4.0 3.6 5.0 3.0 3.9 Median (U/h) 3.3 3.5 3.1 3.2 2.4 3.3 2.7 3.1 2.8 5.9 2.6 2.6 0 0 0 0 0 0 21.5 21.8 24.6 96.0 20.0 22.7 Standard deviation (U/h) Minimum (U/h) Maximum (U/h) 72 Figure 4.4: Distributions of recommended continuous IV insulin infusion rate by six computer-based insulin infusion protocol candidates for all blood glucose measurements 73 Table 4.9: Pairwise Wilcoxon signed rank test for six computer-based insulin infusion protocol candidates HWCIR eProtocol- CIIPGlucosafe CIIP- Glucose insulin CIIP-NS AMC TJIIP Protocol eProtocol- NA < 0.0001 < 0.0001 < 0.0001 < 0.0001 < 0.0001 NA < 0.0001 < 0.0001 < 0.0001 < 0.0001 NA 0.02 < 0.0001 < 0.0001 NA < 0.0001 < 0.0001 NA < 0.0001 insulin Glucosafe HWCIR Glucose Protocol CIIPAMC CIIPTJIIP CIIP-NS NA protocols to be statistically different. We analyzed the favorability scoring using the ranking strategy as outlined in the methods section. We used a standard competition ranking strategy to assign the rankings to our computer-based insulin infusion protocol candidates. Ranking scores were converted to favorability scores using the Borda count formula. Scores were then calculated and depicted in a histogram graph for comparison (see Figure 4.5). Glucosafe has the highest 74 Figure 4.5: Ranked favorability scores for six computer-based insulin infusion protocol candidates overall score, followed by CIIP-AMC and CIIP-TJIIP, respectively. eProtocol-insulin scored the lowest in the ranked favorability scoring. We stratified the ranked favorability scores by three blood glucose measurement categories at time=ti+1 (<80, 80-110, and >110 mg/dL) (see Figure 4.6). Glucosafe has the highest scores in the low (< 80 mg/dL) and high (> 110 mg/dL) blood glucose range. CIIPTJIIP scored the highest in the on-target blood glucose range. eProtocol-insulin scored the lowest in the low blood glucose range category. CIIP-SN scored the lowest in the on-target blood glucose range. CIIP-TJIIP scored the lowest in the high blood glucose range category. We calculated the median and interquartile range (IQR) for all six computer-based insulin infusion protocol candidates (see Table 4.10). The continuous IV insulin infusion rate recommendations appeared clinically similar. 75 Figure 4.6: Ranked favorability scores stratified by blood glucose range 76 Table 4.10: Continuous IV insulin infusion rate recommendations by six computer-based insulin infusion protocol candidates. Continuous IV insulin infusion rate recommendation Blood (U/h) at time=ti, Median (IQR) glucose Count HWCIR range at eProtocol- time=ti+1 insulin Glucosafe CIIP- CIIP- CIIP- AMC TJIIP NS 1.7 1.8 3.0 (2.9) (2.5) (3.2) 2.7 2.2 3.2 (3.7) (2.7) (3.0) 4.8 2.8 3.5 (6.4) (3.0) (3.0) 3.2 2.4 3.3 (4.8) (2.7) (3.1) Glucose Protocol Low (< 80 1845 3.4 (3.5) 1.5 (3.2) 2.4 (4.6) mg/dL) On target (80-110 9356 3.3 (3.2) 2.8 (3.6) 3.1 (3.3) mg/dL) High (> 110 7783 3.3 (2.8) 5.4 (4.7) 3.3 (3.0) mg/dL) Total/Overall 18,984 3.3 (3.1) 3.5 (4.6) 3.1 (3.2) 77 4.2.2 Discussion for Aim 2 We identified a major issue with the paired comparison method described in Aim 1 where the comparison of multiple computer-based insulin infusion protocol candidates was burdensome especially when the number of candidates increases. Our solution was to implement a ranking strategy. We successfully compared six computer-based insulin infusion protocol candidates. Overall, Glucosafe was found to be the most favorable. The pairwise Wilcoxon signed rank test confirmed that the distribution of recommended continuous IV insulin infusion rate for all the computer-based insulin infusion protocol candidates was statistically different. The results indicated that the computer-based insulin infusion protocol candidates were intrinsically different. The internal system of the computer-based insulin infusion protocols reacted differently although they were subjected to the same input. The median for each of the computer-based insulin infusion protocol candidates at different blood glucose ranges was consistent with the ranked favorability scoring. At low blood glucose range, the highest ranked computer-based insulin infusion protocol candidate was Glucosafe with a median at 1.5 U/h. At on target blood glucose range, the highest ranked computer-based insulin infusion protocol candidate was CIIP-TJIIP with insulin infusion rate of 2.2 U/h. This was followed by Glucosafe with a median of 2.8 U/h. At high blood glucose range, the highest ranked computer-based insulin infusion protocol candidate was Glucosafe. Again, Glucosafe has the highest median value of 5.4 U/h. There were several advantages of using the ranking method over paired comparison. The ranking method was straightforward to deploy. It has helped to reduce the number of times we needed to do paired comparison among the computer-based insulin infusion 78 protocol candidates. Using the ranking scale, the computer-based insulin infusion protocol candidates were graded from best to worst based on the favorability of the continuous IV insulin infusion rate. Therefore, the results can be easily analyzed by healthcare practitioners. However, using just one set of evaluation criteria can limit the effort of evaluating the performances of computer-based insulin infusion protocol candidates. We often find multiple conflicting criteria when evaluating our options. Going forward, it was imperative that we take this into consideration when choosing the best available computerbased insulin infusion protocol for a clinical trial. 4.3 4.3.1 Aim 3 Results for Aim 3 The third aim of the study was to design a method of aiding our healthcare practitioners to choose the most appropriate computer-based clinical protocol for a clinical trial. We used multiple-criteria decision analysis to compare six of the computer-based insulin infusion protocol candidates to derive our decision. We selected seven criteria to analyze the performance of the computer-based insulin infusion protocol candidates. 4.3.1.1 Ranked Favorability Scoring We obtained the score from the ranked favorability scoring as described in the Methods for Aim 2. The computer-based insulin infusion protocol candidates were then assigned their rank according to the order of score (see Table 4.11). Glucosafe has the highest ranked favorability score. 79 Table 4.11: Ranking of computer-based insulin infusion protocol candidates using criterion 1 (ranked favorability scoring) Computer-based insulin infusion Ranked favorability protocol candidate score Rank eProtocol-insulin 46,524 6 Glucosafe 62,635 1 HWCIR Glucose Protocol 52,580 4 CIIP-AMC 54,429 2 CIIP-TJIIP 53,403 3 CIIP-NS 46,600 5 4.3.1.2 Estimation of Hypoglycemia Rate We used the eProtocol-insulin observed rate of change of blood glucose per unit of insulin at time ti to estimate the subsequent blood glucose level for other computer-based insulin infusion protocols at time ti+1. An observation was marked as hypoglycemia if the estimated blood glucose was less than 60 mg/dL. The number of cases of hypoglycemia in eProtocol-insulin was the actual observations recorded. The results showed that CIIP-AMC has the highest estimated cases of hypoglycemia at 5.5% (see Figure 4.7). eProtocol-insulin has the lowest incidence of hypoglycemia. The computer-based insulin infusion protocol candidates were then assigned their rank according to the order of score obtained from the estimated cases of hypoglycemia (see Table 4.12). Lower percentage of cases of hypoglycemia was considered better. In another analysis (see Table 4.13), we pooled the number of cases with risk of hypoglycemia according to the column where the computer-based insulin infusion protocol 80 Figure 4.7: Estimated cases of hypoglycemia (* represents actual cases). Table 4.12: Ranking of computer-based insulin infusion protocol candidates using criterion 2 (estimated cases of hypoglycemia) Computer-based insulin infusion Estimated cases of protocol candidate hypoglycemia (%) Rank eProtocol-insulin* 0.8 1 Glucosafe 3.3 5 HWCIR Glucose Protocol 1.3 3 CIIP-AMC 5.5 6 CIIP-TJIIP 1.0 2 CIIP-NS 1.6 4 81 Table 4.13: Number of cases estimated to have risk of hypoglycemia stratified according to favorability scoring in Methods for Aim 1 Blood Glucose More favorable computer-based insulin infusion protocol category (mg/dL) according to Favorability Scoring in Methods for Aim 1 but at time=ti+1 with risk of hypoglycemia eProtocol-insulin Glucosafe Equivalent < 80 137 9 12 80 - 110 231 0 38 > 110 0 139 203 Total 368 148 114 was considered to be more favorable (see Methods for Aim 1, a pairwise comparison of Glucosafe and eProtocol-insulin using favorability scoring). This was an attempt to compare the findings in Methods for Aim 1 and Methods for Aim 3. We found 148 estimated cases of hypoglycemia when Glucosafe was considered more favorable. However, there were 482 estimated cases of hypoglycemia when Glucosafe was not considered more favorable (see Table 4.13). 4.3.1.3 Estimation of Hyperglycemia Rate We used the eProtocol-insulin observed rate of change of blood glucose per unit of insulin at time ti to determine the hyperglycemia rate by estimating the subsequent blood glucose level for other computer-based insulin infusion protocols at time ti+1. Observation was marked as hyperglycemia if the estimated blood glucose was greater than 180 mg/dL. 82 The number of cases of hyperglycemia in eProtocol-insulin was the actual observations recorded. The results showed that CIIP-AMC has the highest estimated cases of hyperglycemia at 5.6% (see Figure 4.8). eProtocol-insulin has the lowest incidence of hyperglycemia at 3.2%. The computer-based insulin infusion protocol candidates were then assigned their rank according to the order of score obtained from the estimated cases of hyperglycemia (see Table 4.14). Lower percentage of cases of hyperglycemia was considered better. 4.3.1.4 Estimation of Cases Within the Target Range We used the eProtocol-insulin observed rate of change of blood glucose per unit of insulin at time ti to estimate the subsequent blood glucose level for other computer-based insulin infusion protocols at time ti+1. An observation was marked as within target range if the estimated blood glucose was between 80 mg/dL and 110 mg/dL. The number of cases of blood glucose within the target range in eProtocol-insulin was the actual observations recorded. The results showed that CIIP-TJIIP has the highest estimated cases of blood glucose within the target range at 52.3% (see Figure 4.9). HWCIR Glucose Protocol has the lowest incidence of cases where the blood glucose was within the target range. The computer-based insulin infusion protocol candidates were then assigned their rank according to the order of score obtained from the estimated cases of blood glucose within the target range (see Table 4.15). Higher percentage of cases within the target range was considered better. 83 Figure 4.8: Estimated cases of hyperglycemia (* represents actual cases). Table 4.14: Ranking of computer-based insulin infusion protocol candidates using criterion 3 (estimated cases of hyperglycemia) Computer-based insulin infusion Estimated cases of protocol candidate hyperglycemia (%) Rank eProtocol-insulin* 3.2 1 Glucosafe 4.6 5 HWCIR Glucose Protocol 3.3 2 CIIP-AMC 5.6 6 CIIP-TJIIP 3.6 3 CIIP-NS 3.7 4 84 Figure 4.9: Estimated cases of blood glucose within the target range (* represents actual cases). Table 4.15: Ranking of computer-based insulin infusion protocol candidates using criterion 4 (estimated cases of blood glucose within the target range) Estimated cases Computer-based insulin infusion within the target Rank protocol candidate range (%) eProtocol-insulin* 49.3 3 Glucosafe 51.9 2 HWCIR Glucose Protocol 45.9 6 CIIP-AMC 48.0 5 CIIP-TJIIP 52.3 1 CIIP-NS 48.6 4 85 4.3.1.5 Mean of Recommended Continuous IV Insulin Infusion Rates We measured the mean of recommended continuous IV insulin infusion rates for each of the computer-based insulin infusion protocol candidates (see Figure 4.10). We stratified the results according to five blood glucose range categories at time=ti:  Less than 60 mg/dL  Less than 80 mg/dL  80-110 mg/dL  Greater than 110 mg/dL  Greater than 180 mg/dL At very low blood glucose range (< 60 mg/dL), all of the computer-based insulin infusion protocols recommended none or very little insulin for patients. At low blood glucose range (< 80 mg/dL), the highest average recommended continuous IV insulin infusion rate was eProtocol-insulin (mean=2.7 U/h). HWCIR Glucose Protocol did not recommend any insulin when blood glucose was below 80 mg/dL. When blood glucose was within the target range, the highest average recommended continuous IV insulin infusion rates were eProtocol-insulin (mean=3.7 U/h), HWCIR Glucose Protocol (mean=3.7 U/h), and CIIP-NS (mean=3.7 U/h). The lowest average recommended continuous IV insulin infusion rate was CIIP-TJIIP (mean=2.5 U/h). At high blood glucose range (> 110 mg/dL), CIIP-AMC has the highest average recommended continuous IV insulin infusion rate (mean=7.9 U/h) followed by Glucosafe (mean=6.3 U/h). The lowest average recommended continuous IV insulin infusion rate in the high blood glucose range was CIIP-TJIIP (mean=4.0 U/h). At very high blood glucose range (> 180 mg/dL), CIIP-AMC has the highest average 86 Figure 4.10: Mean of recommended continuous IV insulin infusion rates recommended continuous IV insulin infusion rate (mean=11.0 U/h) followed by Glucosafe (mean=9.1 U/h). The lowest average recommended continuous IV insulin infusion rate in the high blood glucose range was HWCIR Glucose Protocol (mean=5.2 U/h). The computer-based insulin infusion protocol candidates were then assigned their rank according to the amount of recommended continuous IV insulin infusion rate at each blood glucose range category (see Table 4.16). At very low (< 40 mg/dL), low (< 60 mg/dL), and within target range (80-110 mg/dL) categories, we preferred lower measure of mean for recommended continuous IV insulin infusion rate. At high (> 110 mg/dL) and very high (> 180 mg/dL) blood glucose range categories, we preferred higher measure of mean for recommended continuous IV insulin infusion rate. The ranks were summed up and overall ranking was recalculated based on the total in ranks (see Table 4.16). The results indicated that CIIP-AMC has the highest rank for mean of recommended continuous IV insulin infusion rate. eProtocol-insulin and CIIP-TJIIP 87 Table 4.16: Ranking of computer-based insulin infusion protocol candidates using criterion 5 (mean recommended continuous IV insulin infusion rate) Rank eProtocol-insulin Glucosafe HWCIR Glucose Protocol CIIP-AMC CIIP-TJIIP CIIP-NS Blood < 60 1 6 1 1 5 1 < 80 6 2 1 3 4 5 80-110 5 2 6 3 1 4 > 110 5 2 4 1 6 3 > 180 4 2 6 1 5 3 21 14 18 9 21 16 5 2 4 1 5 3 glucose (mg/dL) Total in Ranks Overall ranking 88 shared the lowest rank for the mean measure. 4.3.1.6 Median of Recommended Continuous IV Insulin Infusion Rates We measured the median of recommended continuous IV insulin infusion rates for each of the computer-based insulin infusion protocol candidates (see Figure 4.11). We stratified the results according to five blood glucose range categories at time=ti:  Less than 60 mg/dL  Less than 80 mg/dL  80-110 mg/dL  Greater than 110 mg/dL  Greater than 180 mg/dL At very low blood glucose range (< 60 mg/dL), the median for all of the computerbased insulin infusion protocols was zero. At low blood glucose range (< 80 mg/dL), the highest median for recommended continuous IV insulin infusion rate was eProtocol-insulin (median=2.2 U/h). HWCIR Glucose Protocol has a median of zero when blood glucose was below 80 mg/dL. When blood glucose was within the target range, the highest median for recommended continuous IV insulin infusion rates were eProtocol-insulin (median=3.2 U/h) and HWCIR Glucose Protocol (median=3.2 U/h). The lowest median for recommended continuous IV insulin infusion rate was CIIP-TJIIP (median=2.0 U/h). At high blood glucose range (> 110 mg/dL), the highest median for recommended continuous IV insulin infusion rate was Glucosafe (median=6.2 U/h) followed by CIIPAMC (median=5.6 U/h). The lowest median for recommended continuous IV insulin 89 Figure 4.11: Median of recommended continuous IV insulin infusion rates infusion rate in the high blood glucose range was CIIP-TJIIP (median=3.4 U/h). At very high blood glucose range (> 180 mg/dL), the highest median for recommended continuous IV insulin infusion rate at this blood glucose range was Glucosafe (median=8.8 U/h) followed by CIIP-AMC (median=6.0 U/h). The lowest median recommended continuous IV insulin infusion rate was CIIP-TJIIP (median=4.1 U/h). The computer-based insulin infusion protocol candidates were then assigned their rank according to the median results at each blood glucose range category (see Table 4.17). At very low (< 40 mg/dL), low (< 60 mg/dL), and within target range (80-110 mg/dL) categories, we preferred lower median for recommended continuous IV insulin infusion rate. At high (> 110 mg/dL) and very high (> 180 mg/dL) blood glucose range categories, we preferred higher median for recommended continuous IV insulin infusion rate. The final results showed that Glucosafe was ranked first using the median criterion followed by CIIP-AMC, HWCIR Glucose Protocol and CIIP-NS (joint third), CIIP-TJIIP, 90 Table 4.17: Ranking of computer-based insulin infusion protocol candidates using criterion 5 (median recommended continuous IV insulin infusion rate) Rank eProtocolinsulin Glucosafe HWCIR Glucose Protocol CIIP-AMC CIIP-TJIIP CIIP-NS Blood < 60 1 1 1 1 1 1 < 80 6 2 1 3 4 5 80-110 5 2 5 2 1 4 > 110 4 1 4 2 6 3 > 180 4 1 5 2 6 3 20 7 16 10 18 16 6 1 3 2 5 3 glucose (mg/dL) Total in Ranks Overall ranking and eProtocol-insulin. 4.3.1.7 Distributed Favorability Scoring We estimated the subsequent blood glucose for all computer-based insulin infusion protocol candidates and assigned them the score accordingly. The scores were then summed up (see Figure 4.12). The computer-based insulin infusion protocol candidates were assigned their rank 91 Figure 4.12: Distributed favorability scoring according to the order of score obtained from the distributed favorability scoring (see Table 4.18). Higher score for distributed favorability was considered better. Our results showed that CIIP-TJIIP attained the highest rank in distributed favorability scoring, followed by eProtocol-insulin and CIIP-NS. 4.3.2 Aggregating Decision Scores We measured the quality of our computer-based insulin protocol candidates by considering the seven criteria described above (see Methods section 3.7.2) using multiplecriteria decision analysis. The criteria weights were arbitrarily assigned based on heuristic as an example. Scores for each of the criteria were calculated by multiplying the weight and ranks obtained earlier (see Table 4.19). Finally, the scores were aggregated and ready to be presented to the healthcare practitioner for evaluation (see Figure 4.13). CIIP-TJIIP has the highest score in this 92 Table 4.18: Ranking of computer-based insulin infusion protocol candidates using criterion 7 (distributed favorability scoring) Computer-based insulin infusion Distributed protocol candidate favorability scoring Rank eProtocol-insulin 1,489,580 2 Glucosafe 1,462,000 4 HWCIR Glucose Protocol 1,446,850 5 CIIP-AMC 1,399,350 6 CIIP-TJIIP 1,511,830 1 CIIP-NS 1,474,610 3 93 Table 4.19: Scores using multiple-criteria decision analysis with seven criteria to measure computer-based insulin infusion protocols (*higher score is better). HWCIR Glucose Protocol CIIP-AMC CIIP-TJIIP CIIP-NS Weight Glucosafe Criteria eProtocol-insulin Score (ranking x weight) 8 48 24 40 32 16 60 20 40 10 50 30 48 16 40 8 32 24 28 35 7 14 42 21 10 25 15 30 10 20 3 18 12 15 6 12 8 48 24 40 32 16 Ranked favorability 8 scoring Estimation of 10 hypoglycemia rate Estimation of 8 hyperglycemia rate Estimation of cases 7 within the target range Mean of recommended continuous IV insulin 5 infusion rates Median of recommended continuous IV insulin 3 infusion rates Distributed favorability 6 scoring 94 Figure 4.13: Aggregated score using multiple-criteria decision analysis (*higher score is better). example, followed by Glucosafe and eProtocol-insulin. We did not evaluate the impression of healthcare practitioners using the multiple-criteria decision analysis. 4.3.3 Discussion for Aim 3 We have successfully developed an in silico method with increasingly sophisticated and informative techniques to help our healthcare practitioners make an informed decision about choosing the most appropriate computer-based insulin infusion protocol for a clinical trial. We believed the seven criteria that we have developed for the multiple-criteria decision analysis were critical in evaluating the performance of these computer-based insulin infusion protocol candidates. The ranked favorability scoring (criterion 1) analyzed the perceived favorability of each candidate by comparing the relative goodness of the recommended insulin infusion rate when the subsequent blood glucose was known. Glucosafe was the top performer in 95 this category. Glucosafe consistently gave more favorable recommended insulin infusion rates compared to others. In some cases, the amount of recommended insulin infusion rate by a certain computer-based insulin infusion protocol candidate can cause a greater drop in blood glucose level even though their overall favorability gave us a good score. By estimating the hypoglycemia rate (criterion 2), we were able to calculate the risk of hypoglycemia that may occur with the use of that computer-based insulin infusion protocol. CIIP-AMC has the highest estimated rate of hypoglycemia. Healthcare practitioners may decide not to choose this computer-based insulin infusion protocol if the risk of hypoglycemia is a major concern solely based on this criterion. If the amount of recommended insulin infusion rate by a certain computer-based insulin infusion protocol candidate was less than optimal, the blood glucose level may not drop to the desired level or may continue to increase. We were able to estimate the risk of hyperglycemia (criterion 3) by calculating the subsequent blood glucose level for computer-based insulin infusion protocols at time ti+1 and flagged the blood glucose levels that were greater than 180 mg/dL. CIIP-AMC has the highest estimated rate of hyperglycemia at 5.6% followed by Glucosafe at 4.6%. The rates were still clinically low compared to the actual cases measured by eProtocol-insulin at 3.2%. However, healthcare practitioners might be less likely to choose these computer-based insulin infusion protocols if this was the only criterion available. The estimation of cases within the target range (criterion 4) showed healthcare practitioners how consistently the computer-based insulin infusion protocols were able to maintain the blood glucose level within the target range. The estimated cases within the 96 target range among the computer-based insulin infusion protocol candidates appeared clinically similar (45.9% to 52.3%, range difference was 6.4%). CIIP-TJIIP has the highest number of cases estimated to be within the target range of 80-110 mg/dL, followed by Glucosafe. The mean and median measurement gave us a sense of the typical value of a computerbased insulin infusion protocol candidate in question. By stratifying the results according to five blood glucose range categories, we were able to understand the performance of the computer-based insulin infusion protocol candidates during these key moments of low, ontarget, and high blood glucose range. The results showed that the mean and median values were fairly similar clinically. There were some notable differences at the high blood glucose range, in particular for Glucosafe and CIIP-AMC. CIIP-AMC has a higher mean but lower median when compared to Glucosafe. This indicated that CIIP-AMC tends to recommend higher insulin infusion rates more frequently than Glucosafe. At low blood glucose range, HWCIR performed exceptionally well because the protocol will not recommend any continuous IV insulin infusion when blood glucose level falls below 80 mg/dL. When blood glucose was within the target range, CIIP-TJIIP has the lowest mean and median values. Thus, CIIP-TJIIP was regarded as having a much better performance compared to other computer-based insulin infusion protocol candidates because not only was it able to maintain the blood glucose within the desirable level, lower insulin infusion rate may prevent cases of hypoglycemia. At high blood glucose range, we have mixed results due to the mean and median values, but overall, Glucosafe and CIIP-AMC did well in this regard. However, Glucosafe was more consistent when recommending higher continuous IV insulin infusion rates at high blood glucose range. 97 The goal of the distributed favorability scoring was to analyze how well the computerbased insulin infusion protocol candidate manages the patient's blood glucose. Higher scores were given if their estimated blood glucoses were closer to the desired target range. CIIP-TJIIP has the highest aggregated score among the computer-based insulin infusion protocol candidates. This also means that CIIP-TJIIP was able to tightly control the blood glucose levels. We found that CIIP-TJIIP was the overall winner of the multiple-criteria decision analysis. The CIIP-TJIIP has the highest rank for the estimated number of cases within the target range and distributed favorability scoring, and a fairly low estimated rate of hypoglycemia. This may have been largely contributed by the higher target blood glucose range (100-140 mg/dL) set by the rules of the protocol. Glucosafe was a close second in the multiple-criteria decision analysis despite having scored less for estimated hypoglycemia and hyperglycemia rate. Nevertheless, healthcare practitioners who prefer a computer-based insulin infusion protocol that matches their target range of 80-110 mg/dL may want to consider Glucosafe for their clinical trial instead. In a separate analysis, we compared the findings in Methods for Aim 1 (pairwise comparison of Glucosafe and eProtocol-insulin using favorability scoring) and Methods for Aim 3 (estimated cases of hypoglycemia). While some may expect that Glucosafe continues to have relatively low risk of hypoglycemia because of how well it performed in the favorability scoring in Methods for Aim 1, the result predicted more instances of Glucosafe recommending insulin doses that could have a high risk of hypoglycemia (482 instances when Glucosafe was not favorable versus 148 instances when Glucosafe was favorable). The analysis highlighted a major disadvantage in Methods for Aim 1 in which 98 one criterion (using relative recommended insulin dose to measure favorability) is simply not adequate to get an overview for the performance of a computer-based clinical protocol candidate. Protocols have complex influences, and effects are multifactorial; there are often trade-offs between meeting one goal (e.g., quickly getting blood glucose reduced to the target range), and other goals or risks (e.g., likelihood of blood glucose continuing to fall past the target range, causing hypoglycemia). While all of the criteria described above were different, each of them was able to probe different aspects of the computer-based insulin infusion protocol candidates and illuminate their strengths and weaknesses. This is the main difference between the multiple-criteria decision analysis (Methods for Aim 3) and single criteria analysis (Methods for Aim 1 and Methods for Aim 2). While results from the single criteria analysis may have suggested that Glucosafe was the better candidate, an in-depth analysis using multiple-criteria decision analysis revealed otherwise. A high favorability scoring (as suggested in Methods for Aim 1 and Methods for Aim 2) does not guarantee high scoring for performance criteria such as estimated rate of hypoglycemia. While Glucosafe recommended more favorable continuous IV insulin infusion rates, some of the unfavorable infusion rates may be harmful. The other probes suggested in Methods for Aim 3 can illuminate these issues more effectively. The aggressive approach with Glucosafe may affect patients in terms of extreme blood glucose management (hypoglycemia and hyperglycemia) if healthcare practitioners do not have the resources to monitor the bedside situation more closely. The final results may still vary depending on the weights of the evaluation criteria and blood glucose target selected by healthcare practitioners during the preclinical trial evaluation. Healthcare practitioners have the ability to change the weights according to the 99 importance they place based on their patient's needs or clinical situations. For example, pediatricians may decide that avoiding the risk of hypoglycemia for patients in the pediatric ICU is more important than how efficient the insulin brings down the high blood glucose level. In some practice, healthcare practitioners may decide that a different blood glucose target range, e.g. 90-140 mg/dL, is better for their patients. This can change the outcome because many of these criteria are affected by the choice of blood glucose target range. In this study, the weights were assigned arbitrarily as an example to showcase the multiple-criteria decision analysis. While this was not the scope of the study, we believe that more robust weights should be obtained through healthcare practitioners involved in the evaluation. We will need to measure the degree of agreement among the raters through a form of interrater reliability measure such as Cohen's kappa. Such measurement can be useful to examine for any variability in the ratings and lend crediability to the weights. Multiple-criteria decision analysis can be susceptible to changes due to the decisionmaking environment. Each step in the decision-making process involves some form of uncertainty. This includes the selection of analytical method, choice of criteria, assessment of the values of the criteria, and choice of weights. A sensitivity analysis can help us test the robustness of the results from the multiple-criteria decision analysis in the presense of all these uncertainties. We will need to determine the amount of changes in the criterion weight that would affect the rank outcome of the computer-based insulin infusion protocols in the multiple-criteria decision analysis. One may think that the larger weight may have a significant change in the analysis. However, it may be possible that criteria with small weights can be critical when influencing the final result of the analysis. Thus, a sensitivity analysis is an important component of the decision-making process and should be 100 investigated to ensure successful implementation of the evaluation strategy. Our physiological process simulation in the in silico framework was based on the notion that patients who had been intervened with the computer-based insulin infusion protocol (in this case eProtocol-insulin) creates the necessary simulation for comparison with other computer-based insulin infusion protocols. We reasoned that such patient data provide the best representation of a treatment plan and the patient response to the intervention. We therefore assumed that patients intervened with eProtocol-insulin can recreate all the necessary blood glucose-insulin interactions if we have sufficient data in the simulation. In the absence of data for these patients who had been intervened with similar computer-based insulin infusion protocols, we may be able to recreate the physiological process simulation using wild type data where patients were treated with paper protocols or healthcare practitioners heuristics. The multiple-criteria decision analysis was key in summarizing the performance of computer-based insulin infusion protocol candidates. The performance scores provided key insights and necessary clarity into the inner workings of the system. The analysis can also help healthcare practitioners set the right expectations and avoid potential issues when assessing the computer-based insulin infusion protocol in an actual clinical trial. 5 CONCLUSION Clinical protocols are often used to make patient care better linked to best evidence. The advancement of medical knowledge and computerization has brought many benefits to healthcare. However, rapid development of healthcare systems has also spurned many versions of computer-based clinical protocols that were developed using different standards of practice and medical knowledge. How do we evaluate these clinical protocols? How do we compare and identify computer-based clinical protocols that will meet our care standards and be suitable for our clinical trial? These were the questions that first motivated our research in this field. A healthcare practitioner can spend time going through the manuals, rule sets, or mathematical algorithms describing the physiology behind the computer-based clinical protocols. Yet they may barely understand how all of these will impact their patients. Our in silico method enabled us to create a framework for comparing and evaluating computer-based clinical protocols by simulating patient care with these protocols. We have successfully demonstrated our in silico method using EMR data. We used wide ranging techniques to evaluate six computer-based insulin infusion protocols and helped our healthcare practitioners to select the best possible candidate for a clinical trial. 102 5.1 Strengths and Limitations There were many advantages to our in silico approach to compare and evaluate computer-based insulin infusion protocols. First, the in silico method was an inexpensive approach to identify a computer-based insulin infusion protocol without incurring the full cost of a clinical trial. The comprehensive analysis can be performed in a computer simulation without any intervention by healthcare practitioners. This has the potential to reduce the cost of healthcare in general. Second, the in silico method performed various simulations in a safe environment without involving real patients. Patients were not exposed to any risk when we investigated critical scenarios with our in silico method. Third, the in silico method has encouraged healthcare practitioners to get more involved in the development and evaluation of computer-based insulin infusion protocols. It enhanced collaboration between technical developers and healthcare providers. Participation of healthcare practitioners in the evaluation process can lead to better satisfaction and acceptance towards the use of computer-based clinical protocols in practice. Fourth, results from the in silico method can be used to improve future iterations of the computer-based insulin infusion protocols. Researchers can use these results to address issues in the system and to tweak the performance of the computer-based insulin infusion protocol accordingly. We have several limitations to our study. Our population sample was limited to Utah adult patients who were primarily Caucasians of northern European descent. The lack of diversity may have hampered the generalizability of the results to populations of other ethnicity. Studies have shown that there may be relations between insulin resistance and sensitivities among racial groups [104,105]. Our research data were generated with patients managed by eProtocol-insulin and their imprints were carried over to the analysis. Future 103 comparison of computer-based insulin infusion protocol candidates using clinical data generated with computer-based insulin infusion protocols other than eProtocol-insulin might be useful. We were limited by the data that we used as inputs to our computer-based insulin infusion protocol candidates. While these data sets were adequate to generate reasonable continuous IV insulin infusion recommendations, additional data such as medications, hypertension, and types of infection may provide valuable insights. We eliminated a large number of records with certain imperfections because we wanted to maximize the validity of our comparative results. However, doing so may have affected the generalizability of our evaluation towards other computer-based insulin infusion protocol candidates. It is possible that the patient records that were excluded may reflect clinical conditions that would have altered our findings. However, the demographic characteristics of the patients in the retained records appear to be reflective of the demographic characteristics in the ICUs. We found apparently contradictory results from our different methods, with different protocols emerging as the winner depending on what comparison method we chose. In addition, the weights that were used in our final evaluation were chosen heuristically by the investigators and may not be reflective of the weights that clinicians may actually choose in practice - the results of that evaluation may be different had different criteria been used. However, the results demonstrated that the framework is capable of accommodating user-defined preference criteria. 104 5.2 Generalizability Results must be reproducible before they are accepted in any scientific domain. These results often require experimental methods that can be replicated with clinical decisionmaking methods. We believe consistent clinical decision-making methods can be enabled by using computer-based clinical protocols. It can standardize the process of care and eliminate unnecessary variations in clinical practice. We have demonstrated that our in silico method was applicable to different types of computer-based insulin infusion protocols. Even though they were developed using different knowledge base and scientific details (rule-based, physiologic mathematical modeling, statistical models, and column-based charts), the framework allowed multiple comparison and evaluation of these computer-based insulin infusion protocols possible. While we only focused our research on one type of computer-based clinical protocol, we believe the framework for our in silico method can be extended to compare and evaluate other types of computer-based clinical protocols such as mechanical ventilator protocols and warfarin management protocols. 5.3 Suggestions for Future Research Future research may involve exploring more subtle favorability scoring algorithms that could analyze delicately complex patients' outcome. This may include blood glucose trends over a period of time, interaction with medications, diet, impact on patient's blood glucose due to insulin resistance, and insulin sensitivity to different ethnicity groups. These algorithms can add more contrast to the existing ones and help evaluate performance criteria that have not been adequately examined previously. We acknowledged that our current research was limited to data generated by the use 105 of eProtocol-insulin. We only used a subset of EMR data such as nutrition and use of Propofol as input to our computer-based insulin infusion protocol candidates. Going forward, future computer-based insulin infusion protocol candidates may require more extensive clinical observations and historical data for their recommendations. These may include genetic factors, family history, hypertension, triglyceride levels, types of infection, and various medications (glucosamine, rifampicin, corticosteroids, glucocorticoids, methadone, antiretrovirals, etc.) We believe our next generation of in silico methods will have more sophisticated computer simulation to simulate realistic patients' response to insulin intervention. The in silico methods will need to be able to incorporate clinical usage patterns including time variation so that they will closely mimic the use of a computer-based insulin infusion protocol at the bedside. We can continue to enhance our glucose-insulin regulation models by incorporating more sophisticated techniques such as machine learning and probabilistic methods. Since our focus in this initial study was not about developing more advanced models, we chose a simple linear model to describe the relationship between insulin dosage and change in blood glucose. We can continue to incorporate more complex mathematical models in our computer simulation to investigate their usefulness. We will also need to validate these models in clinical studies to verify the accuracy and generalizability of these models. Going forward, we believe machine learning techniques such as deep learning will become particularly important when we are able to gather huge amounts of data. One critical enhancement to the in silico framework will be interoperability. There has to be a common semantics where the in silico framework can work seamlessly with 106 different types of computer-based insulin infusion protocols. This is important because many computer-based insulin infusion protocols are built on different platforms for specific users or systems in a particular healthcare institution. A patient model can help map the various inputs and outputs of the computer-based insulin infusion protocol with the structure of the computer simulation. The in silico framework also has to support a common structure for exchanging information between the computer simulation and the computerbased insulin infusion protocol candidates. REFERENCES 1. Smith M, Saunders R, Stuckhardt L, Mcginnis JM. Best care at lower cost: The path to continuously learning health care in America. Washington, DC: The National Academies Press. 2013. 2. Kawamoto K, Houlihan C a, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005 Apr 2;330(7494):765. 3. Lobach D, Sanders GD, Bright TJ, Wong A, Dhurjati R, Bristow E, et al. Enabling health care decisionmaking through clinical decision support and knowledge management. Evid Rep Technol Assess (Full Rep). 2012 Apr;(203):1-784. 4. Bright TJ, Wong A, Dhurjati R, Bristow E, Bastian L, Coeytaux RR, et al. Effect of clinical decision-support systems: A systematic review. Ann Intern Med. 2012 Jul 3;157(1):29-43. 5. Johnston ME, Langton KB, Haynes RB, Mathieu A. Effects of computer-based clinical decision support systems on clinician performance and patient outcome. A critical appraisal of research. Ann Intern Med. 1994 Jan 15;120(2):135-42. 6. Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA. American Medical Association; 1998 Oct 21;280(15):1339-46. 7. Morris AH. Developing and implementing computerized protocols for standardization of clinical decisions. Ann Intern Med. 2000 Mar 7;132(5):373-83. 8. Garg AX, Adhikari NKJ, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA. 2005 Mar;293(10):1223-38. 9. Vogelzang M, Zijlstra F, Nijsten MWN. Design and implementation of GRIP: a computerized glucose control system at a surgical intensive care unit. BMC Med Inform Decis Mak. 2005 Jan;5(1):38. 10. Juneja R, Roudebush C, Kumar N, Macy A, Golas A, Wall D, et al. Utilization of a computerized intravenous insulin infusion program to control blood glucose in the 108 intensive care unit. Diabetes Technol Ther. 2007 Jun;9(3):232-40. 11. Juneja R, Roudebush CP, Nasraway SA, Golas AA, Jacobi J, Carroll J, et al. Computerized intensive insulin dosing can mitigate hypoglycemia and achieve tight glycemic control when glucose measurement is performed frequently and on time. Crit Care. 2009 Jan;13(5):R163. 12. Krikorian A, Ismail-Beigi F, Moghissi ES. Comparisons of different insulin infusion protocols: a review of recent literature. Curr Opin Clin Nutr Metab Care. 2010 Mar;13(2):198-204. 13. Mann E a, Jones J a, Wolf SE, Wade CE. Computer decision support software safely improves glycemic control in the burn intensive care unit: a randomized controlled clinical study. J Burn Care Res. 2011 Jan;32(2):246-55. 14. Newton CA, Smiley D, Bode BW, Kitabchi AE, Davidson PC, Jacobs S, et al. A comparison study of continuous insulin infusion protocols in the medical intensive care unit: computer-guided vs. standard column-based algorithms. J Hosp Med. 2010 Oct;5(8):432-7. 15. Damiani G, Pinnarelli L, Colosimo SC, Almiento R, Sicuro L, Galasso R, et al. The effectiveness of computerized clinical guidelines in the process of care: a systematic review. BMC Health Serv Res. 2010 Jan;10(1):2. 16. Heathfield H, Pitty D, Hanka R. Evaluating information technology in health care: barriers and challenges. BMJ. 1998 Jun 27;316(7149):1959-61. 17. Collier R. Rapidly rising clinical trial costs worry researchers. CMAJ. 2009 Feb 3;180(3):277-8. 18. Holve E, Pittman P. A First Look at the Volume and Cost of Comparative Effectiveness Research in the United States [Internet]. Academy Health. 2009. 19. Woolf SH, Grol R, Hutchinson A, Eccles M, Grimshaw J. Clinical guidelines: potential benefits, limitations, and harms of clinical guidelines. BMJ. 1999 Mar 20;318(7182):527-30. 20. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn't. BMJ. 1996 Jan 13;312(7023):71-2. 21. Thomas L. Clinical practice guidelines. Evid Based Nurs. 1999 Apr 1;2(2):38-9. 22. Hewitt-Taylor J. Clinical guidelines and care protocols. Intensive Crit Care Nurs. 2004 Feb;20(1):45-52. 23. Oniki TA, Clemmer TP, Arthur LK, Linford LH. Using statistical quality control techniques to monitor blood glucose levels. Proc Annu Symp Comput Appl Med Care. 1995 Jan;586-90. 109 24. Finney SJ, Zekveld C, Elia A, Evans TW. Glucose control and mortality in critically ill patients. JAMA. American Medical Association; 2003 Oct 15;290(15):2041-7. 25. Krinsley JS. Effect of an intensive glucose management protocol on the mortality of critically ill adult patients. Mayo Clin Proc. 2004 Aug;79(8):992-1000. 26. Morris AH. Decision support and safety of clinical environments. Qual Saf Health Care. 2002 Mar;11(1):69-75. 27. Safran C, Rind DM, Davis RB, Ives D, Sands DZ, Currier J, et al. Effects of a knowledge-based electronic patient record in adherence to practice guidelines. MD Comput. Jan;13(1):55-63. 28. Ebben RHA, Vloet LCM, Verhofstad MHJ, Meijer S, Mintjes-de Groot JAJ, van Achterberg T. Adherence to guidelines and protocols in the prehospital and emergency care setting: a systematic review. Scand J Trauma Resusc Emerg Med. 2013 Jan;21:9. 29. Ebben RHA, Vloet LCM, van Grunsven PM, Breeman W, Goosselink B, Lichtveld RA, et al. Factors influencing ambulance nurses' adherence to a national protocol ambulance care: an implementation study in the Netherlands. Eur J Emerg Med. 2014 Mar 1; 30. Linda T. Kohn Janet M. Corrigan, Molla S. Donaldson Editors; Committee on Quality of Health Care in America I of M. To Err Is Human: Building a Safer Health System [Internet]. The National Academies Press; 2000. 31. Morris AH, Hirshberg E, Sward KA. Computer protocols: how to implement. Best Pract Res Clin Anaesthesiol. 2009 Mar;23(1):51-67. 32. Hammond JJ. Protocols and guidelines in critical care: development and implementation. Curr Opin Crit Care. 2001 Dec;7(6):464-8. 33. Wirtschafter DD, Scalise M, Henke C, Gams RA. Do information systems improve the quality of clinical research? Results of a randomized trial in a cooperative multiinstitutional cancer group. Comput Biomed Res. 1981;14(1):78-90. 34. Dawes RM, Faust D, Meehl PE. Clinical versus actuarial judgment. Science. 1989 Mar 31;243(4899):1668-74. 35. Fox J, Patkar V, Thomson R. Decision support for health care: the PROforma evidence base. Inform Prim Care. 2006 Jan;14(1):49-54. 36. Lobach DF, Hammond WE. Development and evaluation of a Computer-Assisted Management Protocol (CAMP): improved compliance with care guidelines for diabetes mellitus. Proc Annu Symp Comput Appl Med Care. 1994 Jan;787-91. 37. Thompson BT, Orme JF, Zheng H, Luckett PM, Truwit JD, Willson DF, et al. 110 Multicenter validation of a computer-based clinical decision support tool for glucose control in adult and pediatric intensive care units. J Diabetes Sci Technol. 2008 May;2(3):357-68. 38. Rosas-Arellano MP, Sam J, Adhikari NKJ, Beyene J, McDonald H, Garg AX, et al. Effects of Computerized Clinical Decision Support Systems on Practitioner Performance and Patient Outcomes: A Systematic Review. JAMA. 2005 Mar;293(10):1223-38. 39. Shulman R, Finney SJ, O'Sullivan C, Glynne PA, Greene R. Tight glycaemic control: a prospective observational study of a computerised decision-supported intensive insulin therapy protocol. Crit Care. 2007 Jan;11(4):R75. 40. Steil GM, Deiss D, Shih J, Buckingham B, Weinzimer S, Agus MSD. Intensive Care Unit Insulin Delivery Algorithms: Why So Many? How to Choose? J Diabetes Sci Technol. 2009 Jan;3(1):125-40. 41. Faden RR, Kass NE, Goodman SN, Pronovost P, Tunis S, Beauchamp TL. An ethics framework for a learning health care system: a departure from traditional research ethics and clinical ethics. Hastings Cent Rep. Jan;Spec No:S16-27. 42. Blaha J, Kopecky P, Matias M, Hovorka R, Kunstyr J, Kotulak T, et al. Comparison of three protocols for tight glycemic control in cardiac surgery patients. Diabetes Care. 2009 May;32(5):757-61. 43. Boord JB, Sharifi M, Greevy R a., Griffin MR, Lee VK, Webb T a., et al. Computerbased Insulin Infusion Protocol Improves Glycemia Control over Manual Protocol. J Am Med Informatics Assoc. 2007 Jan;14(3):278-87. 44. Hovorka R, Kremen J, Blaha J, Matias M, Anderlova K, Bosanska L, et al. Blood glucose control by a model predictive control algorithm with variable sampling rate versus a routine glucose management protocol in cardiac surgery patients: a randomized controlled trial. J Clin Endocrinol Metab. 2007 Aug;92(8):2960-4. 45. Pielmeier U, Rousing ML, Andreassen S, Nielsen BS, Haure P. Decision support for optimized blood glucose control and nutrition in a neurotrauma intensive care unit: preliminary results of clinical advice and prediction accuracy of the Glucosafe system. J Clin Monit Comput. 2012 Aug;26(4):319-28. 46. Plank J, Blaha J, Cordingley J, Wilinska ME, Chassin LJ, Morgan C, et al. Multicentric, randomized, controlled trial to evaluate blood glucose control by the model predictive control algorithm versus routine glucose management protocols in intensive care unit patients. Diabetes Care. 2006 Feb;29(2):271-6. 47. Morris AH, Orme J, Rocha BH, Holmen J, Clemmer T, Nelson N, et al. An electronic protocol for translation of research results to clinical practice: a preliminary report. J Diabetes Sci Technol. 2008 Sep;2(5):802-8. 111 48. Wilson M, Weinreb J, Hoo GWS. Intensive insulin therapy in critical care: a review of 12 protocols. Diabetes Care. 2007 Apr 1;30(4):1005-11. 49. Kansagara D, Fu R, Freeman M, Wolf F, Helfand M. Intensive insulin therapy in hospitalized patients: a systematic review. Ann Intern Med. 2011 Feb 15;154(4):268-82. 50. Allart L, Vilhelm C, Mehdaoui H, Hubert H, Sarrazin B, Zitouni D, et al. An architecture for online comparison and validation of processing methods and computerized guidelines in intensive care units. Comput Methods Programs Biomed. 2009 Jan;93(1):93-103. 51. Lee JC, Kim M, Choi KR, Oh TJ, Kim MY, Cho YM, et al. In silico evaluation of glucose control protocols for critically ill patients. IEEE Trans Biomed Eng. 2012 Jan;59(1):54-7. 52. Hovorka R, Chassin LJ, Ellmerer M, Plank J, Wilinska ME. A simulation model of glucose regulation in the critically ill. Physiol Meas. IOP Publishing; 2008 Aug 1;29(8):959-78. 53. Marik PE, Bellomo R. Stress hyperglycemia: an essential survival response! Crit Care. 2013;17(2):305. 54. van den Berghe G, Wouters P, Weekers F, Verwaest C, Bruyninckx F, Schetz M, et al. Intensive insulin therapy in critically ill patients. N Engl J Med. 2001 Nov 8;345(19):1359-67. 55. Toschlog E a, Newton C, Allen N, Newell M a, Goettler CE, Schenarts PJ, et al. Morbidity reduction in critically ill trauma patients through use of a computerized insulin infusion protocol: a preliminary study. J Trauma. 2007 Jun;62(6):1370-5. 56. Morris AH, Orme J, Truwit JD, Steingrub J, Grissom C, Lee KH, et al. A replicable method for blood glucose control in critically Ill patients. Crit Care Med. 2008 Jun;36(6):1787-95. 57. Pielmeier U, Andreassen S, Juliussen B, Chase GG, Nielsen BSS, Haure P. The Glucosafe system for tight glycemic control in critical care: a pilot evaluation study. J Crit Care. 2010 Mar;25(1):97-104. 58. Pielmeier U, Andreassen S, Nielsen BS, Chase JG, Haure P. A simulation model of insulin saturation and glucose balance for glycemic control in ICU patients. Comput Methods Programs Biomed. 2010 Mar;97(3):211-22. 59. Osburne RC, Cook CB, Stockton L, Baird M, Harmon V, Keddo A, et al. Improving hyperglycemia management in the intensive care unit: preliminary report of a nursedriven quality improvement project using a redesigned insulin infusion algorithm. Diabetes Educ. Jan;32(3):394-403. 112 60. Murphy MA, Whitman I, Campfield A, Moxey E, Haddad M, Whitman G. Intense implementation of a strict insulin infusion protocol does not guarantee postoperative glycemic control. J Am Coll Surg. 2010 Oct;211(4):465-469.e3. 61. Finfer S, Chittock DR, Su SY-S, Blair D, Foster D, Dhingra V, et al. Intensive versus Conventional Glucose Control in Critically Ill Patients. N Engl J Med. 2009 Mar 26;360(13):1283-97. 62. Bergeron BP, Greenes R a. Modeling and Simulation in Medicine: The State of the Art. Proceedings of the Annual Symposium on Computer Application in Medical Care. American Medical Informatics Association; 1988. p. 282-6. 63. Sander C. Genomic medicine and the future of health care. Science. 2000 Mar 17;287(5460):1977-8. 64. Fone D, Hollinghurst S, Temple M, Round A, Lester N, Weightman A, et al. Systematic review of the use and value of computer simulation modelling in population health and health care delivery. J Public Health (Bangkok). 2003 Dec 1;25(4):325-35. 65. Cook DA, Erwin PJ, Triola MM. Computerized virtual patients in health professions education: a systematic review and meta-analysis. Acad Med. 2010 Oct;85(10):1589-602. 66. Lateef F. Simulation-based learning: Just like the real thing. J Emerg Trauma Shock. 2010 Oct;3(4):348-52. 67. Foster DM, Boston RC. Using Computer Simulation Models of Physiological and Metabolic Processes in Laboratory Animals. ILAR J. 1997 Jan 1;38(2):58-68. 68. Beard DA, Bassingthwaighte JB, Greene AS. Computational modeling of physiological systems. Physiol Genomics. 2005 Sep 21;23(1):1-3; discussion 4. 69. Moule P, Pollard K, Armoogum J, Messer S. Virtual Patients: Development In Cancer Nursing Education. Nurse Educ Today. 2015 Feb 21; 70. Brown D, Namas RA, Almahmoud K, Zaaqoq A, Sarkar J, Barclay DA, et al. Trauma in silico: Individual-specific mathematical models and virtual clinical populations. Sci Transl Med. 2015 Apr 29;7(285):285ra61. 71. Wilinska ME, Chassin LJ, Hovorka R. In silico testing--impact on the progress of the closed loop insulin infusion for critically ill patients project. J Diabetes Sci Technol. 2008 May;2(3):417-23. 72. Wilinska ME, Nodale M. An evaluation of "I, Pancreas" algorithm performance in silico. J Diabetes Sci Technol. 2009 Jul;3(4):857-62. 73. Makroglou A, Li J, Kuang Y. Mathematical models and software tools for the 113 glucose-insulin regulatory system and diabetes: an overview. Appl Numer Math. 2006 Mar;56(3-4):559-73. 74. Chase JG, Le Compte AJ, Preiser J-C, Shaw GM, Penning S, Desaive T. Physiological modeling, tight glycemic control, and the ICU clinician: what are models and how can they affect practice? Ann Intensive Care. Springer; 2011 Jan 5;1(1):11. 75. Schaller S, Willmann S, Lippert J, Schaupp L, Pieber TR, Schuppert A, et al. A Generic Integrated Physiologically based Whole-body Model of the GlucoseInsulin-Glucagon Regulatory System. CPT pharmacometrics Syst Pharmacol. 2013 Jan;2:e65. 76. Bois FY. Physiologically based modelling and prediction of drug interactions. Basic Clin Pharmacol Toxicol. 2010 Mar;106(3):154-61. 77. Zhang Z. Parameter estimation techniques: a tutorial with application to conic fitting. Image Vis Comput. 1997 Jan;15(1):59-76. 78. Andersen KE, Højbjerre M. A population-based Bayesian approach to the minimal model of glucose and insulin homeostasis. Stat Med. 2005 Aug 15;24(15):2381- 400. 79. Dalla Man C, Rizza RA, Cobelli C. Meal simulation model of the glucose-insulin system. IEEE Trans Biomed Eng. 2007 Oct;54(10):1740-9. 80. Srinivasan A, Lee JB, Dassau E, Doyle FJ. Novel insulin delivery profiles for mixed meals for sensor-augmented pump and closed-loop artificial pancreas therapy for type 1 diabetes mellitus. J Diabetes Sci Technol. 2014 Sep;8(5):957-68. 81. Farmer TG, Edgar TF, Peppas NA. Parameter set uniqueness and confidence limits in model identification of insulin transport models from simulation data. Diabetes Technol Ther. 2008 Apr;10(2):128-41. 82. Palma RCI. Estimation and personalization of clinical insulin therapy parameters. 2013 Sep 27; 83. Shiang KD, Kandeel F. A computational model of the human glucose-insulin regulatory system. J Biomed Res. 2010 Sep;24(5):347-64. 84. Boutayeb A, Chetouani A. A critical review of mathematical models and data used in diabetology. Biomed Eng Online. 2006 Jun 29;5(1):43. 85. Lonergan T, Le Compte A, Willacy M, Chase JG, Shaw GM, Wong X-W, et al. A simple insulin-nutrition protocol for tight glycemic control in critical illness: development and protocol comparison. Diabetes Technol Ther. Mary Ann Liebert, Inc. 2 Madison Avenue Larchmont, NY 10538 USA; 2006 Apr 30;8(2):191-206. 114 86. Rousing ML, Pielmeier U, Andreassen S. Evaluating modifications to the Glucosafe decision support system for tight glycemic control in the ICU using virtual patients. Biomed Signal Process Control. 2014 Jan; 87. Kovatchev BP, Breton M, Man CD, Cobelli C. In silico preclinical trials: a proof of concept in closed-loop control of type 1 diabetes. J Diabetes Sci Technol. 2009 Jan;3(1):44-55. 88. Zavitsanou S, Mantalaris A, Georgiadis M, Pistikopoulos E. In-silico Closed Loop Control Validation Studies for Optimal Insulin Delivery in Type 1 Diabetes. IEEE Trans Biomed Eng. 2015 Apr 29; 89. Halford GS, Baker R, McCredden JE, Bain JD. How Many Variables Can Humans Process? Psychol Sci. 2005 Jan;16(1):70-6. 90. Thompson C, Cullum N, McCaughan D, Sheldon T, Raynor P. Nurses, information use, and clinical decision making--The real world potential for evidence-based decisions in nursing. Evid Based Nurs. 2004 Jul 1;7(3):68-72. 91. Smith R. What clinical information do doctors need? BMJ. 1996 Oct 26;313(7064):1062-8. 92. Smith R. Strategies for coping with information overload. BMJ. 2010 Jan 15;341(dec15_2):c7126. 93. Hall A, Walton G. Information overload within the health care system: a literature review. Health Info Libr J. 2004 Jun;21(2):102-8. 94. Neale G, Woloshynowych M, Vincent C. Exploring the causes of adverse events in NHS hospital practice. J R Soc Med. 2001 Jul;94(7):322-30. 95. Thomas EJ, Petersen LA. Measuring errors and adverse events in health care. J Gen Intern Med. 2003 Jan;18(1):61-7. 96. Edwards W. Behavioral decision theory. Annu Rev Psychol. Annual Reviews; 1961 Jan 28;12(1):473-98. 97. Prior AN. On the Logic of "Better". Sören Halldén. (Library of Theoria, No. 2.) (C. W. K. Gleerup, Lund, and Ejnar Munksgaard, Copenhagen, 1957. 111 pp.). 1960; 98. Keeney RL, Raiffa H. Decisions with Multiple Objectives: Preferences and Value Trade-Offs [Internet]. Cambridge University Press; 1993. 99. Kenneth J. Arrow ESM. Social Choice and Individual Values [Internet]. Yale University Press; 2012. 100. Amy DJ. Behind the Ballot Box: A Citizen's Guide to Voting Systems. Westport, Conn: Praeger; 2000. 115 101. Grofman, Bernard; Lijphart A. Electoral laws and their political consequences. New York: Agathon Press; 1986. 102. Saari D. Basic geometry of voting. Springer Berlin Heidelberg; 1995. 103. Gallagher M. Proportionality, disproportionality and electoral systems. Elect Stud. 1991 Mar;10(1):33-51. 104. Saad MF, Lillioja S, Nyomba BL, Castillo C, Ferraro R, De Gregorio M, et al. Racial Differences in the Relation between Blood Pressure and Insulin Resistance. N Engl J Med. Massachusetts Medical Society ; 1991 Mar 14;324(11):733-9. 105. Kodama K, Tojjar D, Yamada S, Toda K, Patel CJ, Butte AJ. Ethnic differences in the relationship between insulin sensitivity and insulin response: a systematic review and meta-analysis. Diabetes Care. 2013 Jun;36(6):1789-96.