Integrated information display to support ICU nurses at the bedside: ethnographic observation, design, and evaluation

Preventable adverse events are one of the leading causes of hospitalized patient deaths. Many of these adverse events occur in Intensive Care Units (ICUs) where nurses often work under cognitive, perceptual, and physical overloads. Contributing to these overloads are spatially separated devices which display treatment relevant information such as orders, monitoring information, and equipment status on numerous displays. If essential information of these separate devices was integrated into a single display at the bedside, nurses could potentially reduce their workload and improve their awareness of the patients' treatment plans and physiological status. We conducted a set of three studies for the purpose of designing an efficient and effective ICU display. We observed ICU nurses during their shifts and found that task-relevant information was often presented in the wrong format, unavailable at the point of care or laborious to obtain. Additionally, nurses were sometimes unaware of significant changes in their patient's status and equipment operation. Based on nurses' feedback, we designed an integrated information display that presents all of the information that nurses need at the patient bedside. Nurses selected a display based on the information organization of existing patient monitors, with added medication management and team communication features. The evaluation of paper-based prototypes of both the integrated display and existing ICU displays showed that nurses could answer questions about the patient's status and treatment faster (p<<0.05) and more accurately (p<<0.05) using the integrated display. The number of adverse events in the ICU could potentially be reduced by integrated displays, but to implement them into clinical practice will require significant engineering efforts.

INTEGRATED INFORMATION DISPLAY TO SUPPORT ICU NURSES AT THE BEDSIDE: ETHNOGRAPHIC OBSERVATION, DESIGN, AND EVALUATION by Sven Heinrich Koch A dissertation submitted to the faculty of The University of Utah in partial fulfillment of the requirements for the degree of Doctor of Philosophy Department of Biomedical Informatics The University of Utah December 2010Copyright © Sven Heinrich Koch 2010 All Rights Reserved STATEMENT OF DISSERTATION APPROVAL The dissertation of Sven Heinrich Koch has been approved by the following supervisory committee members: Dwayne Westenskow , Chair 8/2/2010 Date Approved Charlene Weir , Member 8/2/2010 Date Approved John Hurdle , Member 8/2/2010 Date Approved Jim Agutter , Member 8/29/2010 Date Approved Nancy Staggers , Member 8/31/2010 Date Approved and by Joyce Mitchell , Chair of the Department of Biomedical Informatics and by Charles A. Wight, Dean of The Graduate School. ABSTRACT Preventable adverse events are one of the leading causes of hospitalized patient deaths. Many of these adverse events occur in Intensive Care Units (ICUs) where nurses often work under cognitive, perceptual, and physical overloads. Contributing to these overloads are spatially separated devices which display treatment relevant information such as orders, monitoring information, and equipment status on numerous displays. If essential information of these separate devices was integrated into a single display at the bedside, nurses could potentially reduce their workload and improve their awareness of the patients' treatment plans and physiological status. We conducted a set of three studies for the purpose of designing an efficient and effective ICU display. We observed ICU nurses during their shifts and found that task-relevant information was often presented in the wrong format, unavailable at the point of care or laborious to obtain. Additionally, nurses were sometimes unaware of significant changes in their patient's status and equipment operation. Based on nurses' feedback, we designed an integrated information display that presents all of the information that nurses need at the patient bedside. Nurses selected a display based on the information organization of existing patient monitors, with added medication management and team communication features. The evaluation of paper-based prototypes of both the integrated display and existing ICU displays showed that nurses could answer questions about the patient's status and treatment faster (p<<0.05) and more accurately (p<<0.05) using the integrated display. The number of adverse events in the ICU could potentially be reduced by integrated displays, but to implement them into clinical practice will require significant engineering efforts. iv TABLE OF CONTENTS ABSTRACT .................................................................................................. iii LIST OF TABLES ......................................................................................... viii LIST OF FIGURES ......................................................................................... x CHAPTER 1. INTRODUCTION ..................................................................................... 1 Medical errors in ICUs ............................................................................. 2 Suggestions to decrease medical errors ................................................. 3 Missing information to develop integrated displays ................................. 4 Specific aims ........................................................................................... 4 2. BACKGROUND ....................................................................................... 6 ICU nurses patient monitoring ................................................................. 7 Situation awareness ................................................................................ 8 Human factors design principles............................................................ 13 User centered development .................................................................. 18 Graphical information displays for ICU nurses ...................................... 20 Evaluation methods of displays ............................................................. 22 3. IDENTIFICATION OF REQUIREMENTS FOR AN INTEGRATED INFORMATION DISPLAY FOR NUSES AT THE BEDSIDE ................. 27 Methods ................................................................................................. 28 Observation of nursing task frequency and missing information ..... 29 Identification of patient safety critical task categories ..................... 32 Implications and vision for an integrated display ............................. 34 Results .................................................................................................. 35 Observation of nursing task frequency and missing information ..... 35 Identification of patient safety critical task categories ..................... 41 Implications and vision for an integrated display ............................. 41 Discussion ............................................................................................. 46 4. DEVELOPMENT AND FINE TUNING OF AN INTEGRATED INFORMATION DISPLAY ..................................................................... 48 Rough prototypes ..............................Fehler! Textmarke nicht definiert. Overall goals ................................................................................... 49 Overall methods .............................................................................. 52 First iteration ................................................................................... 55 Second iteration .............................................................................. 57 Third iteration .................................................................................. 60 Discussion and implication .............................................................. 63 Refined prototypes ................................................................................ 64 Overall goals ................................................................................... 64 Overall methods .............................................................................. 64 Fourth iteration ................................................................................ 66 Fifth iteration ................................................................................... 68 Sixth iteration .................................................................................. 68 Seventh iteration ............................................................................. 70 Discussion and implications ............................................................ 76 5. EVALUATION OF AN INTEGRADED INFORMATION DISPLAY ......... 77 Methods ................................................................................................. 78 Setting............................................................................................. 78 Participants ..................................................................................... 80 Study design ................................................................................... 80 Instrumentation ............................................................................... 80 Measurements and data collection ................................................. 85 Scenarios ........................................................................................ 86 Procedures ..................................................................................... 90 Data analysis .................................................................................. 91 Results .................................................................................................. 93 Situation awareness ....................................................................... 93 Response time ................................................................................ 94 Workload ......................................................................................... 99 User satisfaction ........................................................................... 100 6. DISCUSSION ....................................................................................... 102 Interpretation of the findings ................................................................ 103 Identification of requirements ........................................................ 104 Development and fine tuning ........................................................ 109 Evaluation ..................................................................................... 112 Limitations ........................................................................................... 115 Identification of requirements ........................................................ 116 Development and fine tuning ........................................................ 117 vi Evaluation ..................................................................................... 119 Lessons learned ........................................................................... 121 Implications for practice ....................................................................... 123 Future work ......................................................................................... 127 Conclusion ........................................................................................... 131 APPENDICES A. QUESTIONS TO ASSESS PATIENT SAFETY TASKS ...................... 132 B. QUESTIONS USED TO MEASURE INTUITIVENESS AND CORRECT EXTRACTION OF INFORMATION ................................... 135 C. SCREENS OF THE FINAL PROTOTYPE ........................................... 138 D. SCENARIOS AND QUESTIONS TO MEASURE SITUATION AWARENESS...................................................................................... 148 E. ANSWERS TO FREE TEXT QUESTIONS .......................................... 160 F. NASA TASK LOAD INDEX .................................................................. 165 G. QUIS QUESTIONNAIRE: PART THREE ............................................. 167 REFERENCES ......................................................................................... 169 vii LIST OF TABLES Table Page 1. Task frequency and categories of nursing work ....................................... 37 3. Comprehension challenges ..................................................................... 42 4. Projection challenges ............................................................................... 43 5. Patient safety relevance of tasks ............................................................. 44 6. Frequently suggested information content per display element ............... 63 7. Fourth iteration - recognized display elements ........................................ 67 8. Fourth iteration - extracted information .................................................... 67 9. Fifth iteration - recognized display elements ............................................ 69 10. Fifth iteration - extracted information........................................................ 69 11. Sixth iteration - recognized display elements ........................................... 70 12. Sixth iteration - extracted information....................................................... 70 13. Seventh iteration - recognized display elements ...................................... 72 14. Seventh iteration - extracted information ................................................. 72 15. Usability evaluation of the seventh iteration ............................................. 72 16. Improvements of data presentation in the seventh prototype .................. 73 17. Presentation order of display and scenario .............................................. 80 18. Pair wise comparison to analyze main effect and interaction effects for situation awareness ................................................................................. 94 19. Questions answered incorrectly half of the time or more. ........................ 96 20. Pair wise comparison to analyze main effect and interaction effects for response time .......................................................................................... 97 21. Median response time .............................................................................. 98 22. Pair wise comparison of analyze effect of workload................................. 99 23. Pair wise comparison of analyze effect of user satisfaction ................... 101 24. Potentially saved time per hour through the clinical use of integrated displays ................................................................................................. 124 25. How was the availability of clinical information? .................................... 161 26. How was your overall sense of the patient and the treatment? .............. 162 27. What did you like best? .......................................................................... 163 28. What should be redesigned? ................................................................. 164 ix LIST OF FIGURES Figure Page 1. Model of situation awareness .................................................................... 9 2. A user centered approach to design interactive system........................... 19 3. Affinity diagram to identify opportunities for changes in information communication to increase situation awareness ...................................... 36 4. Example of information sources used and ways walked during medication administration. ....................................................................... 40 5. Overview of the integrated display development ..................................... 50 6. Goals of the rough prototyping and prototypes used ............................... 53 7. First iteration - Prototypes to identify overall information organization based in their approach of working .......................................................... 56 8. Second iteration - Prototypes with different display elements ................. 59 9. Third iteration - Prototypes surrounded by popup screens ...................... 62 10. Changes to the seventy prototype iteration .............................................. 75 11. Integrated display - Main screen .............................................................. 82 12. Paper based prototypes of the Control setting. ........................................ 84 13. Example of display-interaction ................................................................. 89 14. Situation awareness answer accuracy ..................................................... 95 15. Response time ......................................................................................... 98 16. Perceived workload (NASA TLX) ........................................................... 100 17. User satisfaction - overall reaction to the display (QUIS) ....................... 101 18. Example of new information flow through the integrated display ........... 125 19. Main screen showing PRN medication .................................................. 139 20. Changed ventilator settings and scheduled medication ......................... 140 21. Current medication empty ...................................................................... 141 22. Changed current medication orders and new scheduled medication ..... 142 23. Monitoring requirements for current medication ..................................... 143 24. Additional medication information .......................................................... 144 25. Vital sign trending - no medication trend shown .................................... 145 26. Fluid management ................................................................................. 146 27. Alarm in the other patient's room ........................................................... 147 x xi ACKNOWLEDGEMENTS I wish to acknowledge the members of my committee, Dwayne Westenskow, Nancy Staggers, Charlene Weir, Jim Agutter, and John Hurdle for their mentorship and the generous gift of their time. Additionally, I would like to thank Joyce Mitchell, the chair of my department for her support and leadership. Thanks are due to David Liu and Matthias Gorges for valuable collaboration and mentorship. The Human Factors and Ergonomics Society generously granted permission to reproduce multiple figures. Furthermore, I would like to thank Tamara Laukert, Frank Drews, Karen Nye, Lezli Matthews, Nathan Pace, John Stillman, Boaz A. Markewitz, Lorie Mitchell, James Orme, Jeanne Miner, Caroline Keller, Lauren Vanleeuwen, Amanda Sheeren, Eric Boam, Natalie Stillman-Webb, Ruth Turner, as well as all participating nurses for their thoughtfulness and assistance in this project. Finally, I am indebted to Maral Haar, and Kai Kück (Drägerwerk AG & Co KGaA) for their financial support and mentoring. Last but not least, I wish to thank my parents, Rose and Manfred Koch, who provided me with empathy and encouragement throughout my PhD program. CHAPTER 1 INTRODUCTION 2 Medical errors in ICUs Medical errors occur frequently and can result in significant harm to patients. A high number of deaths in US hospitals have been attributed to preventable adverse events.1 Patients at Intensive Care Units (ICU) are especially at risk for medical errors.2 In ICUs errors occur frequently - during 1000 ICU patient days 146 serious errors were identified.3 A portion of the errors could be attributed to ICU nurses' significant cognitive overload4 and distractions, such as constant interruptions during the care process.5 Adding to the cognitive overload is the excessive information nurses receive from various sources, such as multiple individuals working in the same space and multiple bedside displays.4 Also, contributing to the cognitive overload are bedside displays with poor ergonomics.6 They are too small,7 have poor labeling,4 and do not meet users cognitive requirements.8-10 Such devices show information hidden in complicated menu structures,9 are unintuitive to use6, 11, 12 and are difficult to learn.13, 14 Another factor which increases cognitive overload is the piecemeal fashion in which nurses need to collect task relevant information from a multitude of different devices, screens and locations. Such disparately located information makes getting the "big picture" difficult.6 Currently clinicians spend a great deal of time and energy searching through raw data trying to understand the patient's situation.10 Their decision making is hindered through ill-structured, uncertain, and potentially confliction information from multiple sources.15 Half of all critical incidents that occur in ICUs can be attributed to low situation awareness, with the other half due to low technical skills.16 3 Situation awareness was defined by Endsley17 and is a theory which can guide research about the user's understanding of the environment. It includes three levels: (i) perception, (ii) comprehension and (iii) projection.10, 85-89 The Background section discusses situation awareness further. Improved display of information could improve nurses' decision making and situation awareness. Suggestions to decrease medical errors Suggestions to decrease errors in critical care are improvements in the user interface for medical devices, automated error checks, salient alarms, and in-service training.8 Another suggestion is the consolidation and integration of all relevant information in one place.18 Used by ICU nurses, such integrated displays "hold tremendous promise for improving patient care."19 The integration of separate information from a variety of sources into one place can improve medical decisions and treatment.18 Integrated information would allow nurses to better understand the effectiveness of interventions and the patient's response to them.9 In other areas integrated information displays have been shown to increase the operator's situation awareness in anesthesia,20 power plant management,21 air traffic control,22 and might increase shared understanding among care providers.23 Displays developed to support specific care aspects were found to partially increase provider's accuracy, time efficiency, and situation awareness, e.g., 6, 20, 24-33 - the background section will discuss issues in more detail. However, to increase nurses' situation awareness at the point of care, a 4 better understanding of nurses' information needs in clinical decision making is needed,34 and what information is missing during clinical tasks.35 No research could be identified that reported detailed descriptions of an integrated information display for ICU nurses based on all relevant nursing tasks at the bedside. Missing information to develop an integrated display An integrated display should address current information deficits ICU nurses have during frequent and patient safety critical tasks. Although some publications list frequent tasks performed by ICU nurses,36-38 they do not list specific information nurses need during these tasks. Furthermore, although publications identified errors during ICU nursing care, e.g., 11, 39, 40 patient safety ratings for single nursing ICU nursing tasks could not be found. To develop an integrated display for the ICU bedside, relevant nursing tasks need to be identified, and the key information to be shown. Detailed information deficits during frequent and patient safety in nurses' situation awareness need to be identified and mitigation strategies need to be implemented. Finally, the display needs to be evaluated. Specific aims Improvements of the work environment of ICU nurses are needed, as nurses spend the most time of all health care team members in the patient's room.36 Our research attempted to further the understanding of the complex information needs nurses face at the bedside with a focus on challenges during frequent and 5 safety relevant tasks at the bedside, and to design and evaluate an integrated information display addressing these challenges. Aim 1: Identify requirements for an integrated information display for ICU nurses at the bedside. The first aim of this investigation was to acquire information about critical care nurses' work through observation of the tasks nurses perform (Chapter 3). Objectives were a) to identify frequent nursing task categories and their associated information, b) to identify patient safety critical task categories, and c) to create a vision for an integrated display. Aim 2: Develop the integrated information display and refine user requirements. The second aim was to design an integrated information display to support frequent and patient safety critical nursing tasks at the bedside (Chapter 4). Objectives were a) to identify nurses' preferred information organization and information content for an integrated display, and b) to improve intuitiveness and correct extraction of information. Aim 3: Evaluate the integrated information display. The third aim was to assess nurses' performance when using the new information display compared to using conventional control devices (Chapter 5). Objectives were to measure their a) situation awareness, b) response time, c) workload, and d) satisfaction. CHAPTER 2 BACKGROUND 7 ICU nurses' patient care activities ICU nurses are responsible for the monitoring and detection of changes in physiological functions on a minute-by-minute and hour-by-hour basis.41 Patient monitoring is only one of multiple functions ICU nurses perform. They also must document their work, give medications and support care coordination.38 Hendrich, et al.38 found that medical surgical nurses spent 78% of their time with nursing practice. During nursing practice they were found to spend 7% of their time in patient monitoring - patient assessment and reading of vital signs. More time consuming were documentation (35%), care coordination (20%), patient care activities (19%), and medication administration (17%). Therefore, nursing practice tasks might benefit from additional support. ICU nurses' patient monitoring might be improved through better support of tactical monitoring. Miller et al.41 found that nurses were more interested in shorter timeframes and attended to perceived information with a rather tactical orientation. Doctors, on the other hand, preferred attending to strategic information with focus on future patient projection and a broader range of physiological functions than nurses. Miller et al. 41 hypothesize that the traditional nursing role might contribute to their focus on short-term changes in the patient's physiological values. Overall, nurses' patient monitoring might benefit from devices which emphasize tactical monitoring. In a study about information on displays to support nurses' patient monitoring, the variables that ICU nurses determined to be the most important were in the cardiovascular, respiratory, fluids, and temperature domains.42 In 8 cardiovascular monitoring they identified heart rate, arterial blood pressure, vasoactive drug infusion rates, central venous pressure, and pulmonary capillary wedge pressure. In respiratory monitoring, they determined mode of ventilation, artificial ventilation rate, spontaneous ventilation rate, minute volume, O2 saturation, fractional inspired oxygen, as well as peak and expiratory pressure. For fluid management, nurses desired the cumulative and hourly fluid balance, hourly fluid input and output, and hourly urine and non-urine output. To support their awareness of temperature, nurses wanted core and peripheral (skin) temperature. Few of the studies listed above use a conceptual model to guide identification of information needs. The model developed by Endsley identified situation awareness as key to effective performance. This model is described below. Situation awareness Situation awareness is a theory which can guide research about the operator's understanding of the state of the environment and of relevant parameters of the system.10, 85-8917 According to Endsley,17 situation awareness includes three levels: (i) perception, (ii) comprehension and (iii) projection, see Figure 1. Success at the higher levels depends on the success of the lower levels. Perception means that the user is aware of the current state of the system e.g., a variable of the system - a nurse might see the systolic value of the blood pressure. Comprehension refers to the user's understanding of implications of 9 Figure 1: Model of situation awareness (Endsley 1995). [Reprinted with permission from Human factors, 37:1, Pp 32-64 Copyright 1995 by the Human Factors and Ergonomics Society. All rights reserved] 10 the perceived values of the system e.g., the meaning of the variable - a nurse might understand that the blood pressure is lower than it should be, considering the patient's condition. Projection depicts the user's anticipation of consequences for the system and its future development, e.g., the implication of a variable - a nurse might anticipate that if nothing changes in the renal function of the patient, he could be impaired through the current condition. The level of situation awareness influences decisions and performance of actions. In Endsley's model (Figure 1) situation awareness is influenced by both individual factors and task/system factors. Individual factors which influence situation awareness include the user's goals, expectations, information processing mechanisms, abilities and training. Task/system factors are system capacity, interface design, stress and workload, complexity, and automation. An integrated information display could address problems in interface design, complexity of the system and automation and possibly reduce stress and workload. In accidents attributed to human error, inadequate situation awareness has been identified as a primary causal factor.90 Situation awareness is diminished by the large number of data elements which operators need to monitor and combine in their heads, resulting in decreased cognitive resources available to attend to important events.10 In medical decision making, situation awareness is a critical component, and the level of situation awareness acquired by a practitioner may be critical to the outcome of the patient.91 Clinicians' inadequate awareness of data may lead to 11 insufficient comprehension of the patient status.92 Increasing the user's situation awareness has become a major goal in the development of user interfaces in a variety of fields. Integrated information displays have been shown to increase the operator's situation awareness in anesthesia,20 power plant management,21 air traffic control,22 and might increase shared understanding among care providers.23 In the evaluation of displays for nurses no report of situation awareness was found - but accuracy and detection of changes were measured which might be used to operationalize situation awareness at the bedside. Comparison with other human computer interaction models The model of situation awareness is compared with two other models which describe human computer interaction: the Staggers health human-computer interaction framework43 and the model by Despont-Gros44 to evaluate user interaction with clinical information systems. In a temporal dimension the model of situation awareness has shortcomings compared with other human computer interaction models - the situation awareness model is missing a temporal component. In Staggers' model the temporal component is called "informatics development trajectory", in Despont-Gros's model it is called "development process". In these models it is used to describe changes and maturation of user characteristics or interaction. One of these user characteristics is training: users gained more experience with systems when getting used to them and were found to prefer a higher information density 12 after sufficient training.45 On the dimension of user characteristics the situation awareness model has the highest level of detail. It cites the individual factors goals, objectives, expectations, individual information processing, memory, automaticity, abilities, experience and training are taken into consideration. The other two models do not go into detail. Similarities exist in the three models in the description of the user interaction with the system. Environmental and computer system characteristics are described in all three models, with the situation awareness analyzing them in the highest level of detail. For system characteristics, the situation awareness model takes task and system factors into consideration: system capacity, interface design, stress, workload, complexity, and automation. The other two models go into less detail. Staggers' model shows computer characteristics and actions. Despont-Gros's model has computer information system, environment and process characteristics but does not specify them further. For our research situation awareness seems to suits best as conceptual model. As our research aims to change the information presentation and availability for nurses, it influences environmental and computer system characteristics. Therefore, these system characteristics need to be shown in detail - in the way they are described in the situation awareness model. Although the missing temporal component of the model is a shortcoming, the greater level of detail in environmental characteristics should provide more benefit. To increase the operator's understanding of devices and possibly the 13 situation awareness, devices can be designed more intuitively, e.g. by applying human factors design principles. Human factors screen design principles The application of human factors principles increases intuitiveness and user friendliness of medical devices.7, 46 Shneiderman47 defined 8 golden design rules, Zhang et al.48 14 usability heuristics for medical devices, and Drews et al.22 summarized human factors design principles. Among these rules, heuristics and principles are a) proximity of related information, b) redundant coding, c) gestalt principles, d) high saliency, e) congruence with mental models, f) affordances, g) pictorial realism, h) direct manipulation, i) labeling in the users' terminology, j) addressing individual users' characteristics, k) flexibility and efficiency, l) minimalist information, and m) consistency of action. These principles are explained in the following, and illuminated with possible implications for the design of information display for nurses. Proximity of related information The display of related information in proximity to each other facilitates cognitive processing of information.49, 50 A display can show related information integrated in spatial proximity - e.g., all medication could be shown in the same display area. The two options for data integration are first, integration into a single display and second, integration into a single display element. First, data integration into a single display describes the consolidation and combination of 14 multiple data sources into a single interface - one of the cornerstones of biomedical informatics according to Stead et al.18 Second, the integration of multiple data sources into a single display element is a variant for "integral display".51 Here the graphical representation of a single display element is altered to visualize changes in multiple variables, e.g., through simultaneous change of color and brightness. Redundant coding Redundant coding of information shows data in more than one dimension to improve identification.7 In a device, color and symbols could be used for redundant coding - e.g., a specific color and symbol can be used to redundantly communicate the same message. Gestalt principles The gestalt principles allow humans to see things as belonging together when they are in spatial proximity, move together, or look alike in color, size, or shape.52 For example, on a display the users can be cued to understand that information belongs together by using similar looking display elements in spatial proximity (proximity principle). High saliency High saliency of relevant information facilitates the search processes.53 For example, an interface can use color coding in red to increase the saliency of 15 important information such as alarms because it is not common and bright. Congruence with mental models To facilitate understanding, the information organization should reflect the user's mental model11. A mental model is the internal representation of some aspects of the external world used to make predictions and inferences.54 To assess mental models of user interaction with a device, users can be asked for their expectations prior to pressing a button on the device. The final interaction can then be designed according to their mental model of the interaction. Affordances. Affordance refers to the cues provided by the environment that indicate an opportunity to act or perform a particular operation. For example, a door handle provides an affordance to pull and a big red button invites "pushing."53 Pictorial realism The information on the display should have the appearance of the variable presented by it (pictorial realism) and dynamically displayed information should reflect the users' mental model about the movement in the real system(principle of moving parts).55 To design for pictorial reality, a device should communicate a real world variable, it should look similar to the depicted value, e.g., time could be visualized by an hour-glass. To satisfy the principle of moving parts, a value which is decreasing should move down on the display. 16 Direct manipulation Direct manipulation of visible objects on the graphical user interface enhances usability.11 A device could have a touch-screen which allows the user to directly select which displayed variables to interact with. Labeling in user's terminology Labeling in the users' terminology shortens learning.48, 53 A display for nurses should not have abbreviations but label displayed variables in terms nurses use in their daily life. Observing nurses and asking them about their understanding of prototypes could identify this terminology. Congruency with user's characteristics Designs should address individual user characteristics. 11, 14, 48, 56 A display for nurses should show information which is relevant for this user group, instead of designing devices which satisfy the information needs of nurses, physicians, and respiratory therapists simultaneously. Flexibility and efficiency To accelerate performance and accommodate for differences between users, systems should allow customization and shortcuts.47, 48 A display for nurses could allow different ways to get to the same goal and customization of information content 17 Minimalist information Extraneous information distracts and slows users down, and simple or sparse information should be used with progressive levels of details.48 On an information display the initial screen could show only essential information and should reveal more information upon interaction. Consistency of action Consistent sequences of actions should be possible in similar situations. Menus, help screens, and commands should use identical terminology.47 A display for nurses should e.g. allow manipulating all different kinds of medication with the same actions - independent if they are current, scheduled or PRN medication. In conclusion, during the design phase we introduce the above principles gradually. By following the above design principles, when designing a display for nurses, it can be assumed that the resulting device is more usable, is easier to understand and more desirable for nurses. However, not all of principles can be incorporated into the design process in very early prototype stages but need to be introduced after the prototypes have reached a certain maturity. Such principles are for example consistency of action, flexibility, and efficiency. Designing a device in cooperation with the final users ensures that information is relevant, useful and understandable. In other words, feedback from users is essential. In the section below, the concept of user-centered development is presented. 18 User centered development Some of the currently used devices in the medical area were not designed with focus on the final user.7, 10, 46, 57, 58 Such devices are often not intuitive and have a low usability.46, 59-61 To create intuitive and usable displays, an iterative development cycle with user participation can be used.62, 63 Intuitive displays react in the way users expect it by showing anticipated content, 53, 63-66 and the information reflects the users' needs and tasks.66, 67 Intuitive and usable devices can be used with very little training.53 Correct identification of displayed information, low error rates and rapid decision-making indicates an intuitive design.63 User-centered development means active participation of the people who will finally use the product in the planning, design and development of the product.53 This process minimizes the learning curve and increases later accuracy and efficiency.62 Figure 2 shows an approach of user centered design which follows an iterative approach (ISO 9241-21062) with stages a) to f). In stage a) "Plan the human centered activity" the focus of the project is defined and the development is planned. In b) "Requirements gathering" the context of use and future users are determined. In c) "Requirements specification" user specific requirements and organizational requirements are identified. In d) "Develop design solution" a prototype is developed. In e) "Evaluate" user based assessment is carried out. In case the prototype does not satisfy the evaluation, the process is reiterated. Iterations can specify further context of use, user and system requirements and reiterate the design solution. This process is continued until the design solution 19 Figure 2 A user centered approach to design interactive system stages of ISO 9241-21062 satisfies the evaluation. Potential problems in user centered design include factors, such as the use of non representative users, and failure to address all potential issues. As a result, only partially satisfying solutions are designed, and proliferated display screens might result. If users participating in the design process are not representative for the final users but the design is adapted to their personality, needs, uses, behavior, and experience it might not suit the needs of the final users.61 If designers do not address all issues and feedback, the final product might not cover all necessary functions. Users might understand which problems they have, but when asked for their preferences not know how to optimally solve 20 these problems - resulting in a design solution which only partially addresses their underlying needs.61, 68 Proliferated display screens might be the result of the design attitude "since we can do it, why not include it" with systems that require navigation through multiple cluttered screens to understand the big picture.69 In addition to user centered development, design guidelines, e.g., human factors design principles, can be followed to improve the final user interface. Graphical information display for ICU nurses A systematic review of evaluations of physiological monitoring displays found that the majority of studies showed improvements in response time, accuracy, and decreased mental workload.70 Studies that have tested novel interfaces with ICU nurses have discovered improved accuracy,24, 25, 29, 71 enhanced detection of patient changes,27, 72 shortened response time,25, 26, 71 and increased user satisfaction.26, 71, 73 Graphical data representation has been found to be superior to textual interfaces, and the integration of multiple pieces of information into changing graphical shapes improved the user's performance in specific cases. Such displays were designed to support different nursing tasks: blood gas monitoring, haemodynamic monitoring, pulmonary monitoring, patient monitoring, order management, and patient management. For blood gas monitoring Doig, Agutter et al. 24, 29 found a significant increase in the accuracy of diagnosing acid base imbalances for all participants using a novel graphic display which applied gestalt principles and that with the display the status of the pH, PaCO2 and HCo3 values could be more effectively 21 communicated. In nurses hemodynamic monitoring Effken et al. 26 showed that an ecological display using gestalt principles which visualized anatomical causation and constraints compared to a traditional bar graph display did not improve recognition speed nor overall cognitive workload but lead to greater user satisfaction. In pulmonary monitoring Liu et al.74 could show that a graphical visualization of ventilator variables based on gestalt principles used by nursing students compared to a traditional representation did not improve the assessment of patient condition but increased detection of changes and made fewer errors. In patient monitoring Miller et al.27, 72 found improved detection of patient changes when patient data was ordered by physiological function, on a single sheet of paper or split between four computer screens. For nurses order management such as creating, activating, modifying and discontinuing medical orders Staggers et al.71 found that a graphical compared to a textual interface resulted in faster response time, fewer errors, easier learning, and higher user satisfaction. In pulmonary monitoring, Wachter et al. 75 found that clinical volunteers using a pulmonary metaphor display based on gestalt principles showed a faster detection of and lower workload for obstructed endotracheal tube and intrinsic peep events. Furthermore, fewer unnecessary treatments were given but improvement in diagnostic accuracy was identified. In an observational study76 ICU nurses were found to look at the display on average 1.31 times per room 22 visit - less than other care takers. For patient management Ireland et al.42 developed a summary screen which users found clear, relevant, easy to learn, and simple to use. Their summary screen showed administration data, and cardiovascular, respiratory, fluid, and temperature information. However, no outcome quantitative evaluation could be found. Evaluation methods of displays Display evaluations should be guided by a theoretical model, be based on pilot studies and a power analysis, and use realistic and complex scenarios.70 Participant's performance can be measured in multiple dimensions. Among them are situation awareness or accuracy, performance time, perceived workload, and user satisfaction.70 These variables are explained in the following. Situation awareness To measure situation awareness multiple methods can be used, dependent on the system to be tested and the experimental setting. These methods include: (i) direct system performance, (ii) direct experimental techniques, (iii) verbal protocols, and (iv) subjective measures.77 The selection of the measurement technique depends on the system tested. 78-80 (i) Measuring direct system performance focuses on reaction time to sudden change or a dangerous situation, e.g., recover from a disruption, or time to detect abnormality. This technique requires experts' agreement that the measured 23 performance depends mostly on situation awareness. However, if situation awareness is inferred from the actions that individuals choose to take or their behavior the correlation of good actions / behavior and high situation awareness is not always valid.17 (ii) Direct experimental techniques compare an individual's perception of the situation to some predefined reality. Measurement tools are real-time probes embedded as questions in conversations during the test,81 interruptions of the test and questions following the test. One technique is the Situation Awareness Global Assessment Technique (SAGAT) using multiple choice questions.82, 83 To measure situation awareness of anesthetists, Zhang et al.20 used a situation awareness questionnaire which was completed every 2.5 minutes while the simulation was paused. (iii) In verbal protocols, the participant "thinks aloud" or explains the information relied on during the scenario in a scenario replay. This technique could result in biased results as it relies on the correct interpretation and the memory of the participant. (iv) Subjective measures are self-assessments, expert judgments, peer ratings, and supervisor or instructor ratings during or after the scenario. Subjective measures ask individuals to rate their own or the observed situation awareness of participants.84 Techniques are self-assessments, expert judgments, peer ratings, and supervisor or instructor ratings during or after the scenario.77 Although easily administered, self ratings are limited by the information individuals do not know (the "unknown unknowns"), and subjective 24 measures are rather global with less detail than objective measures. Concluding the above, during the evaluation of the displays we use a direct experimental technique to measure the nurses' awareness. In our evaluation, a direct experimental technique allows comparing an individual's perception of the situation using different kinds of prototypes. The measurement technique we use is Situation Awareness Global Assessment Technique (SAGAT) using multiple choice questions,82, 83 with the speed of the scenario based on the user's response time to questions. Performance time Time measurements are dependent on the study design. Previous physiological monitoring display evaluation studies measured the time to diagnosis,29-31, 75, 85 time to detect event/change,12, 20, 30, 86, 87 time to treatment,25, 30, 31, 75 time to decision,85, 88 overall procedure times,89 and time to complete scenario.90 Experimenter effects occur as an experimenter's expectations may influence the results of his research.91 Automatic, participant driven, time measurements can control for manual time measurement biases and are used during our evaluation. Cognitive workload assessment Common subjective cognitive workload assessment instruments are NASA Task Load Index (TLX) (Appendix F),92 the Subjective Workload Assessment Technique (SWAT),93 and the Workload Profile (WP).94 All three instruments 25 were found to be equally reliable as Rubio et al.95 found that all have a high convergent validity. The instrument's intrusiveness was found to be comparable during the test. 95 However, NASA TLX seems to be more reliable to the other two instruments as subjects had problems understanding the dimensions of WP, and SWAT requires exhaustive ranking task prior to the experiment. NASA TLX uses six dimensions: mental demand, physical demand, temporal demand, performance, effort, and frustration.92 For each dimension a score from 0 to 100 is obtained using 20 step bipolar scales. A global score can be calculated by combining the single scores with operator defined different weights. NASA TLX was found to have good reliability and validity.96 It was found to have a Cronbach's alpha coefficient of more than 0.8, and the correlation coefficient between item score and total score of more than 0.6 except for performance.96 Therefore we use NASA TLX for the evaluation study. User interaction satisfaction To measure user interaction satisfaction, questionnaires such as the Questionnaire For User Interaction Satisfaction (QUIS)97 can be used. QUIS version 7.0 part 398 allows assessing six dimensions of user satisfaction of the user's overall reaction to a system. Participants are asked six questions using 9 point Likert scales to score their impression utilizing the computer system (Appendix G). Questions ask participants about their impression employing the system (terrible-wonderful, frustrating-satisfying, dull-stimulating, difficult-easy, inadequate power-adequate power, rigid-flexible). The 26 reliability of QUIS was found to be sufficiently high with Cronbach's alpha of 0.94.99 Therefore we utilize QUIS for our evaluation study. CHAPTER 3 IDENTIFICATION OF REQUIREMENTS FOR AN INTEGRATED INFORMATION DISPLAY AT THE BEDSIDE 28 Prior literature (see Background) has not provided sufficient detail regarding which information ICU nurses are currently missing during frequent and safety-relevant tasks, and a vision about how an integrated display at the bedside could best support their tasks. However, such information is essential in designing a useful integrated display. The aim of this investigation was to identify requirements for an integrated information display. Objectives were a) to identify frequent nursing tasks, combine them in task categories and identify missing information, b) to identify patient safety critical task categories, and c) to define implications and create a vision for an integrated display. Methods With Institutional Review Board approval, the study was performed at the Medical ICU of the University of Utah Health Science Center, the Surgical ICU at the George E. Wahlen Department of Veterans Affairs Medical Center Salt Lake City, and the Shock Trauma ICU of Intermountain Medical Center. The research consists of five components: 1) observations of nursing in practice settings; 2) classification of nurse activity into tasks; 3) identification of missing information for high-frequency tasks; 4) identification of the most patient safety critical task categories, and 5) definition of implications and a vision for an integrated display. Observations, classification, and identification of missing information are described together. 29 Observation of nursing task frequency and missing information Observational studies and root-cause analysis have been successful in identifying challenges and safety problems in the ICU.16 Observations have been determined to be less intrusive and to have fewer interferences with patient care than other approaches e.g., cognitive work analysis.100, 101 In a project to develop bedside displays for nurses, Miller et al.41 used cognitive analysis to determine the information used by medical and nursing staff. Wachter et al. 102 used expert consultation and literature review to determine variables used to detect adverse pulmonary events. No report was found for the development of the display of Effken et al.26 Participants Nineteen ICU nurses were observed for two hours each: ten nurses at a Medical ICU of the University of Utah Heath Science Center, five nurses at a Surgical ICU at the George E. Wahlen Department of Veterans Affairs Medical Center Salt Lake City, and four nurses at a Shock Trauma ICU of Intermountain Medical Center. The sample size was based on Holtzblatt et al.61 which recommended that there be at least four users per job role (ICU). A larger sample size was selected in the Medical ICU to get a general impression of the workflow in the ICU. Nurses were included if they worked longer than two weeks in an ICU and were taking care of critically ill patients who required technical equipment e.g., ventilators or IV pumps. No demographic data was collected. For 30 each observation, the charge nurse was asked to assign the observation team to a suitable nurse. Procedures An adapted form of participant observation103 was used. The team followed nurses into the patient room, to the central nursing station, and into the medication storage room, but not when nurses left the ICU or requested privacy. Nurses were observed, asked to explain their actions and to state their goals during each action, but different from participant observation the observatory team did not perform actions due to liability issues. Observers did not ask questions in times of very high workload to assure patient safety. The scribe recorded field notes for each action, goals and comments. For example, for medication administration one stated goal was to keep the patient safe, and a comment was that it was hard to identify new medication orders because they looked like previous orders. In case the nurse performed multiple tasks simultaneously, the scribe focused on recording the most obvious task and if possible any additional tasks performed. Time periods were selected to be mainly in the timeframe between 8am and 12pm, because in this timeframe most errors were found to occur in a previous study.104 31 Team members The observation team consisted of a biomedical informaticist with a nursing background, an architectural student, a bioengineering student, and two psychology/human factors students. Team members were trained by the biomedical informaticist. Inter-rater reliability was not measured among the team members' observations. Each observation was performed by one observer and one scribe. Data analysis Within 24 hours of the observations, the team met again and the scribe read each field note aloud and the observer entered the note into an Excel spreadsheet (Microsoft Corporation, Redmond, WA). The team used cued recall105 to reconstruct the situation in case of unclear notes. Nursing task frequency The classification of nurse activities was based on the observations field notes. Notes were sorted according to common characteristics by the research team through an iterative process of discussion and classification. Once the tasks were sorted they were assigned names by three ICU nurses. One of the observers (SK) placed each task into one of five task higher-level categories: communication, medication management, patient assessment, organization, and direct patient care. Categorical assignments were validated by an ICU nurse manager independently. Task categories were further described in nursing terms 32 by three ICU nurses through discussion and consensus. Identification of missing information To identify missing information, field notes were reviewed after organization into task categories. Information that nurses needed but which was not available to complete a task were identified in two ways. First, if a change in location occurred for the purpose of acquiring that information, an information need was recorded. Second, if a nurse was noted as verbally indicating that the information was not available, an information need was also recorded. Missing information was then assigned to one of the levels of situation awareness by the author using the following criteria. Perceptional challenges were identified when information was difficult to see or not available at the location needed. Comprehension challenges were identified when several pieces of information had to be combined, or nurses were not sure about the accuracy of information. Projection challenges were determined when information was missing to predict future developments. Identification of patient safety critical task categories Participants 10 nurses working at a Medical ICU of the University of Utah Heath Science Center, 12 nurses working at a Surgical ICU at the George E. Wahlen Department of Veterans Affairs Medical Center Salt Lake City, and 11 nurses working at a Shock Trauma ICU of Intermountain Medical Center volunteered to 33 participate. All ICU nurses working in a setting during a particular day shift were invited to participate. Of the participating nurses, 46% worked in ICU settings for more than five years, the others less. Five invited nurses refused and three nurses did not complete the questionnaire correctly - their responses were excluded. Questionnaire development The questionnaires included 34 items: 32 questions related to the previously observed tasks and two questions about information needs (see Appendix A). A nurse (SK) selected the 32 tasks as a representative sample from each of the five previously defined task categories: communication, medication management, patient awareness, organization, and direct patient care. Two additional questions addressed information needs while away from the patient: "assessing a critical patient's vital signs when in another room" and "assessing reasons for patient's alarms while in another room". Questionnaires were created using the methodology described by Waltz et al.106 Nurses were asked to rate the patient safety relevance of each task on a Likert scale 107 ranging from 1 "not important" to 7 "very important. Furthermore they were asked if they had more or less than five years work experience in ICU settings. 34 Data analysis The answers were tabulated and scored using Excel (Microsoft, Redmont), which was then used to calculate median, mean, and standard deviation for each item. Implications and vision for an integrated display Devices which support users' goals improve their usability.61, 65, 108 Therefore an integrated display should be designed to support nurses' goals and different approaches to work. To create an overarching vision, an affinity diagram was used, allowing consolidation of large amounts of consumer data, and to vision system improvements along key dimensions.61 The affinity diagram minimized the loss of ideas and supported simultaneous brainstorming by team members. Team members Two psychology students and one medical informaticist participated in the initial step of constructing the affinity diagram. Seven team members participated in the visioning process: a bioengineer, a nurse educator, a biomedical informaticist with a nursing background, architectural student, bioengineering student, and two psychology/human factors students. Procedures An affinity diagram61 was built using printed field notes and nurses' comments as individual items. The observatory team attached field notes and 35 comments to a wall and sorted them by relatedness, comparable goals, similar tasks and related problems (see Figure 3). The team wrote description notes which summarized related field notes. All seven team members then "walked the wall,"61 read through the descriptive notes and identified tasks with missing information. Visions of improved information availability, improved communication and other functions of the integrated display were recorded on post-it notes and attached to the affinity diagram. This process was iterative and consensual. Results Observation of nursing task frequency and missing information Nursing task frequency Forty-six tasks were identified and combined into five categories. Table 1 shows the observed tasks and their frequency. Using this taxonomy, nurses performed on average 23.4 tasks per hour. Communication tasks were most frequent, and included communicating with other persons, recording notes about the communication and communication with peers through charting and orders. Missing information Challenges for nurses' situation awareness were identified for the high frequency nursing task communication, medication management, and patient awareness, as well as for the three levels of situation awareness perception, 36 Figure 3 Affinity diagram to identify opportunities for changes in information communication to increase situation awareness comprehension, and projection. Communication During communication tasks nurses spent much time searching and waiting for their communication partners. They either did not know where that person was, when they would be available and at times, they were not even sure if they had the correct individual. Furthermore, they were not sure if other team members were aware of patient changes, and sometimes forgot to communicate (chart) vital signs. Medication management During medication management, orders and updates were not available at the bedside (in some ICUs), and not readable from the doorway due to small font sizes. Furthermore, nurses were gathering much information from many different 37 Table 1 Task frequency and categories of nursing work based on 38 hours of observations in 19 nurses Category of nursing work Task Tasks/hr Communication chart 1.7 talk to the patient 1.0 check/sign orders 0.9 talk with relatives 0.9 communicate with a physician 0.8 relay information to another nurse 0.8 communicate with another nurse to organize workflow 0.6 ask another nurse for advice 0.5 communicate with respiratory therapist 0.4 participate in rounds 0.3 communicate with unit clerk 0.2 socialize with another nurse 0.2 call pharmacy 0.2 record nursing note 0.1 Total 8.6 Medication management check medication order/sign order 1.6 administer medication 1.6 assess currently administered IV medication 1.5 acquire medication 0.8 check medication compatibility 0.2 record nursing note 0.1 Total 5.7 Patient awareness patient assessment 1.1 nurses rounding (at a glance assessments) 1.1 assess reason for alarm 1.0 measure urine output 0.6 record nursing note 0.7 wound care 0.2 draw arterial blood sample 0.2 assess patient's mental orientation 0.2 assess restraints 0.1 perform spontaneous breathing trial 0.03 assessing endotracheal tube position 0.03 Total 5.3 38 Table 1 Task frequency and categories of nursing work based on 38 hours of observations in 19 nurses Category of nursing work Task Tasks/hr Organization tasks set up equipment (IV pump, ventilator…) 1.4 check equipment 0.5 await pending task, e.g., admission 0.2 Total 2.0 Direct patient care support patient in activities of daily life (e.g., eating, drinking, toileting…) 1.2 suction patient's airways 0.4 bath 0.1 perform oral care 0.1 administer tube feeding 0.1 pull catheter 0.03 extubate 0.03 Total 1.9 Table 1 continued 39 locations (see Figure 4). Figure 4 shows an example of information sources used and ways walked during medication administration. A nurse who is in the patient's room (1) decides to check if new medication needs to be given. On the electronic medical record a new medication is scheduled (2), so the nurse gets the medication from the medication storage room (3). To check if the new medication can run through the same line as one of the current medication, the nurse remembers the current medication (4), and checks the IV compatibility (5a). Furthermore, the nurse checks if the new medication might cause adverse effects which need to be monitored (5b), administers the medication (if compatible) and checks the patients vital signs on the monitor(6). Patient awareness During patient awareness tasks, nurses were not notified of alarms and changes in equipment settings. Furthermore, novice nurses were sometimes unsure about assessment procedures in infrequent illnesses, alarming devices and the severity of the alarms. Situation awareness levels Table 2 to Table 4 show details on missing information for each situation awareness level. Across all categories perceptional challenges were due to unavailable information or information which was hard to discriminate (Table 2). Challenges for nurses' comprehension were largely a result of the need to40 Figure 4 Example of information sources used and ways walked during medication administration. See text for further explanation. [Human Factors and Ergonomics Society 54th Annual Meeting, San Francisco, 2010. "Integrated Information Displays for ICU Nurses: "Field Observations, Display Design, and Display Evaluation", S. H. Koch, N. Staggers, C. Weir, J. Agutter, D. Liu, D. R. Westenskow. Submission ID: 429 Reprinted with permission of the Human Factors Society.]41 combine multiple pieces of information from different locations, inaccurate or unclear information, or correct performance of procedures was unknown (Table 3). Challenges for nurses' projection were mainly based on either missing or inaccurate information needed to predict future developments (Table 4). Identification of patient safety critical task categories The average patient safety score was 6.0 on a Likert scale ranging from 1 "not important" to 7 "very important, and bathing had the lowest score (4.8). The most patient safety critical category of nursing work were medication management, followed by patient awareness, communication, organization tasks, and direct patient care. Table 5 shows that although tasks within a category overlapped, most safety critical tasks were associated with medication management (mean = 1.5) and awareness of the patient's status (mean = 2.0). Other critical tasks identified were assessment of the patient's ventilation status and the patient's vital sign trends over time. Implications and vision for an integrated display Communication For communication tasks, two improvements were envisioned: Synchronous communication with other providers and the communication of observations to increase shared knowledge (close and far through voice recognition). Visible 42 Table 2 Perceptual challenges Task Missing Information Communication Current contact information of relatives Medication management Ordered infusion rate and time Medication administration protocol Delivery status of ordered medication in medication storage room Value of last blood sugar measurement Patient awareness Waveform label (yellow and green were hard to discriminate) Alarming value on monitor (hard to discriminate due to blue background) Results of recently drawn lab values, e.g., blood gas Vital sign values (font to small to read from doorway) Organization Location of equipment Table 3 Comprehension challenges Task Missing Information Communication Location of communication partner Awareness of patient change by other involved clinicians Recently changed alarm and device settings by other clinicians Values to chart (values were inside patient's room, EMR outside) Medication management Changes in current medication orders Was unsigned medication administered by previous shift? In-line compatibility of new and currently administered medication Specific administration instructions for new medication Maximum IV infusion rate (catch up with delayed drug delivery) Patient awareness Assessment needs for patients with infrequent illnesses Alarming device and severity of alarm Patient's mental and physical state Condition of patient and alarms while at another location Patient development over the last hours Organization Utensils needed to prepare for procedure Monitor setup or adjustment of settings Reason device malfunctioned Accuracy of displayed values (measurement errors?) Direct patient care Needs of intubated patient Protocol to performance of specific (rare) tasks Patients comfort level 43 Table 4 Projection challenges Task Missing Information Communication Future availability of communication partner Medication management Scheduled medication orders which are due Possible adverse effects due to currently administered medication Remaining time of currently infusing medication (medication labels and IV pump font size to small to read from doorway) Patient awareness Future development of patient Organization Time when rounds, procedures, or consultations on nurses patient will start Time of arrival of new patient Direct patient care Future availability of another nurse / nursing assistant for help delivery status for asynchronous messages, such as pages or voice mails, could allow nurses to be aware if the recipient was aware of the message and allow to prioritize further requests (far). Medication management To improve medication management three ideas were created. Changed orders or new orders could be automatically visible to nurses to eliminate frequent login and wrong medication administration (far). Automatic display of medication IV compatibility during medication administration and adverse effects of current medication could allow safer medication management (close). Nurses should see the medication name, delivery rate and time until empty from the doorway (far). 44 Table 5 Patient safety relevance of tasks ordered by task category on a Likert scale from 1...7 (1 is most relevant 7 is least relevant) Tasks ordered by category of nursing work Median Mean Standard deviation Medication management Monitoring Medication 1 1.4 0.9 Double checking medication (right order, right medication) 1 1.4 0.8 Administering medications 1 1.4 0.8 Preparing medications 1 1.6 1.1 Total 1 1.5 0.9 Patient awareness Knowing reasons for alarms of my patient when outside the patient's room 1 1.4 1.0 Monitoring vitals 1 1.5 0.7 Assessing reasons for alarms 1 1.6 1.0 Monitoring patients Sp02 1 1.7 1.1 Ventilator check 1 1.7 1.0 Monitoring patients trends (over 24 hours) 1 1.8 1.1 Patient assessment 1 1.8 1.1 Monitoring correct position of ventilation tube 2 1.8 0.9 Knowing a critical patient's vital signs when outside of the room 2 1.8 0.8 Monitoring urine output 2 2.0 1.4 Monitoring wound status 2 2.0 1.3 At a glance assessments 2 2.2 1.1 Breathing trial 2 2.3 1.3 Keeping notes of vital signs or tasks that need to be done 2 2.4 1.2 Total 1 1.9 1.1 Communication Communication with doctor about patient 1 1.6 0.9 Communication with doctors about orders 1 1.6 1.2 Communication with other nurses about patient 2 1.9 1.2 Communication between nurse and RT about patient 2 1.9 1.2 Communication with patients 2 2.0 1.2 Rounds 2 2.0 1.2 Communication with pharmacy 2 2.0 1.2 Charting 2 2.6 1.7 Communication with patient's family 2 2.8 1.7 Total 2 2.0 1.3 Direct patient care Suctioning 1 1.7 1.2 45 Table 5 Patient safety relevance of tasks ordered by task category on a Likert scale from 1...7 (1 is most relevant 7 is least relevant) Tasks ordered by category of nursing work Median Mean Standard deviation Drawing blood 2 2.2 1.5 Oral care 2 2.2 1.4 Feeding 2 2.3 1.4 Bathing 3 3.2 1.5 Total 2 2.4 1.5 Organization tasks Communication to organize procedure 2 2.0 1.0 Setting up equipment 2 2.4 1.2 Total 2 2.2 1.1 Table 5 continued 46 Patient awareness For patient management the following changes to information communication were imagined: The patient's vital signs should be easily distinguishable from the doorway to support nurses' rounding (far). Alarms in the nurses' second patient should be automatically shown at the nurses' present location (far). Direct review of the patient's past vital signs in combination with past drug delivery could allow easier titration of medications to desired effect (close). Organization To improve organization tasks, the detection of errors in equipment could be supported by visual step-by-step instructions- comparable to error detection support in Xerox machines (close). Direct patient care For direct patient care, the vision was to provide a list of tools needed to perform and step-by-step procedure instructions to guide inexperienced nurses through infrequent procedures. Discussion Medication management received the highest rating in terms of patient safety concerns and the second highest rating in terms of frequent tasks. Therefore, it is particularly important issue. Communication and awareness of the patient were performed with high frequency. Information deficits were identified in all three 47 levels of situation awareness. Nurses' perception, comprehension, and projection of changes in patient and treatment were challenged due to distance of related information, lack of visible treatment change communication, and information which was laborious to retrieve. Displays should therefore primarily support medication management and additionally improve communication and patient awareness, and should focus on all three levels of situation awareness. Support should include automatic change notification, close display of related information, context sensitive decision support, improved awareness of changes in nurse's second patient, and implications for future work for the nurse. CHAPTER 4 DEVELOPMENT AND FINE TUNING OF AN INTEGRATED INFORMATION DISPLAY 49 The previous chapter found that medication management, awareness of the patient's state, communication, and identification of problems in the nurse's second patient needed most attention when designing an information display for the bedside. This chapter defined a vision for an integrated display, and the proactive and reactive approaches of nurses' work which should be considered when designing an integrated display. This chapter aimed to develop an intuitive integrated information display with useful information content for nurses. Figure 5 shows that the development process consisted of two major steps: rough prototypes identifying design concepts (Figure 5b) and refinement of intuitiveness and accuracy (Figure 5c). Rough prototypes The initial step with roughly designed prototypes had three iterations of prototype designs. Users were engaged at strategic points which allowed the developers to receive focused feedback on the specific prototypes. Overall goals Intuitiveness and later adoption increases with compatibility of the device with the user's past experiences.109 Therefore the aims of the rough prototype iterations were to probe for and improve information organization and information content for an integrated information display at the bedside. Figure 5 Overview of the integrated display development. Identification of information need, iterative display design, and evaluation compared to existing devices. [Human Factors and Ergonomics Society 54th Annual Meeting, San Francisco, 2010. "Integrated Information Displays for ICU Nurses: "Field Observations, Display Design, and Display Evaluation", S. H. Koch, N. Staggers, C. Weir, J. Agutter, D. Liu, D. R. Westenskow. Submission ID: 429 Reprinted with permission of the HF Society.] 51 52 Figure 6 shows the goals of the three iterations of rough prototyping. The first prototype iteration tried to identify nurse's preferences on overall information organization. The second design iteration aimed to identify preferences on information visualization. The third iteration focused on identifying information content nurses wanted to see on display elements and information they expected when interacting with the display at different types of ICU's. Overall methods Prototyping results in products which better meet users' expectations and needs.61, 110 Paper prototyping suits early stages of design and lets the designer refine the user requirements, functionality, and the information architecture, e.g. based on interviews with the final user.110 Instrumentation Setting With Institutional Review Board approval, the study was performed at the Medical-, Surgical-, and Burn Trauma ICU as well as the Neurological Critical Care of the University of Utah Heath Science Center. Interviews were conducted in the ICU break rooms between 9:00pm and 12.30am. Instruments Prototypes were designed to provide information for medication management, awareness of the patient's state, and communication. The 53 Figure 6: Goals of the rough prototyping and prototypes used. The first iteration aimed to identify nurses preferred information organization. The second iteration compared different visualizations, and the third iteration focused on specific information content and interaction preferences54 previously defined implications and vision for an integrated display (see previous Chapter) guided rough information content, and information hierarchy of close and far views. Information which had to be visible from afar was mainly related to patient awareness and awareness of current medication. Information visible from nearby was often related to medication management. Rough prototypes were sketched on a computer, printed, and discussed with individual nurses. The instruments used to collect this high level information were general in nature (interviews, discussions) as were the graphic examples of the displays. As the iterations became more and more refined, the instruments to gather data were adapted to the stage of refinement. Different questions were pertinent at different stages dependent on the aim of the iteration. Procedures Nurses currently working at ICUs were invited to participate in the interviews by the interviewers. Interviews were conducted one-on-one and were administered to different nurses simultaneously by two interviewers: an information-architect, and a medical informatician / nurse. Interviews consisted of a) introduction - the goals of the study and the prototype(s), b) interview - questions dependent on the goals of the interviews, and c) wrap up -nurses were thanked. Interviews were structured according to a list of display elements which was used to guide the sequence of questions. Nurses were asked open-ended questions59 about their understanding and opinion of each display element, and could expand upon details of their interest. The answers and opinions were 55 recorded on printed spread sheets organized by display element and prototype. First iteration The first iteration aimed to identify nurses' preferences on information organization. Methods Participants Four nurses working at the Medical ICU, three female and one male, participated in the first round of prototype interviews. Their average age was 32 years (standard deviation SD 8.9), their daily computer use was 5.6 hrs (SD 4.3), and their years working at ICU settings were 4.1 (SD 2.7). Procedures In the first iteration, the prototypes with different information organization were explained to nurses and nurses were asked to select their preferred prototype: "Which of the prototypes do you like most for your daily work?" Prototypes Figure 7 shows the three prototypes were designed based on nurses' proactive and reactive approaches to work (see previous Chapter). The prototype shown in Figure 7a focused on support for proactive work approaches, the one in Figure 7b concentrated on reactive work attitudes. The prototype in Figure 7c.56 Figure 7 First iteration - Prototypes to identify overall information organization preferences based on nurses' approach of working. a) proactive vital sign centered - orders, laboratory test results and medication are shown next to the vital sign which they might influence. It allowed nurses to continuously monitor the patient trend and the influence of medication over time, and to predict future actions. b) reactive information organization - nurses are notified about urgent tasks and values they should notice. Values were abnormal vital signs and scheduled medications which both moved gradually to the left side to the red area depending on the amount of attention needed and deviation. c) traditional organization - a close resemblance to known patient-monitoring displays with syringes that show the medications being delivered and scheduled. 57 tried to satisfy both approaches simultaneously in a traditional information organization. The prototype in Figure 7a supported proactive work and shows orders, laboratory test results and medication next to the vital sign which they might influence. It allowed nurses to continuously monitor the patient trend and the influence of medication over time, and to predict future actions. Figure 7b shows a prototype to support reactive work by notifying nurses of urgent tasks and values they should we aware of. Values were abnormal vital signs and scheduled medications which both moved gradually to the left side to the red area depending on the amount of attention needed and deviation. Figure 7c shows the prototype to support both proactive and reactive approaches by enhancing a traditional patient monitor. It has a close resemblance to known patient-monitoring displays, with additional syringes showing the medications being delivered. Proactive nurses can drill down by exploring additional information available upon request and predict future care plans. Reactive nurses are notified of care steps and abnormal values by countdowns and automatic notifications. Results During the first iteration, three of four nurses preferred the enhanced conventional information organization display (Figure 7c). They indicated that it resembled their current patient monitors, had easy-to-see current and scheduled medications, and looked uncluttered 58 Second Iteration The second design iteration aimed to identify preferences on information visualization. Methods Participants Five female nurses working as the Medical ICU participated in the second round of prototype interviews. Their average age was 32.3 years (standard deviation SD 1.5), their daily computer use was 5.1 hrs (SD 4.8), and their years working at ICU settings were 5.6 (SD 6.2). Procedures In the second round, participants were asked to select preferred display elements: "Which of these display elements would you prefer for your daily work?" Prototypes Figure 8 shows prototypes with different display elements supporting specific nursing tasks. These prototypes focused on examples of information content for medication management, ventilation, communication, and trending. 59 Figure 8 Second iteration - Examples for different information representations for medication management, communication, and trending60 Results In the second iteration, four of five suggested adding medication trends directly alongside vital sign trends. Some nurses complained that the exact values on the trend were hard to identify. The majority selected black as preferred background color, and to constantly see the medication infusion rate on currently active medication infusions. Third iteration The third iteration focused on identifying information content nurses wanted to see on display elements and information they expected when interacting with the display. The study was expanded on additional types of ICUs. Methods Participants Sixteen nurses, 12 female and 4 male, participated in the second round of prototype interviews. Five nurses were working at the Burn Trauma ICU, five at the Neurological Critical Care Unit, and six at the Surgical ICU. Their average age was 29.7 years (SD 10.8), their daily computer use was 4.5 hrs (SD 3.2), and their years working at ICU settings were 4.3 (SD 2.7). Procedures In the third round, nurses were asked about their understanding of display elements, possible interaction and suggested information content. Interviewers 61 pointed on each display element and asked nurses what they expected to see when pressing the element on the main screen. Then the pop-up screen for the element was shown and nurses were asked if the information was sufficient, or if they would add or remove something. Questions were: "What do you think this is?", "What do you expect to happen if you touch this?", and "Is the information shown sufficient for task X - or which information would you want to add or remove?" Prototypes Figure 9 shows a prototype designed using the results of the prior interviews to ask nurses for suggestions and possible interaction with the prototype. Pop-up screens were used in the empty central area to give information to support medication management, trending, charting, and communication tasks. Results Table 6 shows information content nurses suggested for display elements and when interacting with them. Nurses demanded that the level of details on each element should be adaptable to individual nurses' monitoring preferences and to the patient's condition. To increase accuracy, nurses decided that infusion rate and alarm settings should be changed using buttons instead of sliders. When asked about the function and meaning of display elements, some nurses were not immediately able to identify intended function and shown values of some elements. 62 Figure 9 Third iteration - Third prototype surrounded by popup screens which show possible information content upon interaction. Popup screens of supported tasks are displayed in the central black area and discussed with nurses. The figure shows in the top right corner the screen for medication assessments, and clockwise direct communication, display settings, alarm in the second patient, and trending 63 Table 6 Frequently suggested information content per display element Display element Expected or suggested information content Trending Past alarms Value alarming variable, alarm duration Waveform at a specific time Charting of specific events on the trend Patient awareness (while in the room) Change alarm settings, scale waveform, zero ECG or blood pressure transducer Automatic detection of artifact in vital signs Normal waveform for comparison Influencing factors, e.g., medication, paced patient Patient awareness (while away in the other patient's room) Kind and severity of alarm Current vital signs Silence alarm or relay to coworker to silence alarm Optionally: Trend Medication administration IV compatibility with current medication Doctor's order Concentration of active ingredient Select/change infusion rate and administer bolus Medication assessment Adverse side effects Time started, last time given Maximum infusion rate dependent on pat. metabolism Associated laboratory test results Charting Tab with access to current charting system Automatic charting of displayed values with confirmation Fast charting of specific values, e.g., urine output Communication Leave message for next nurse working with patient Notification of changes in orders without interaction Contact coworker by function, e.g., respiratory therapist Discussion and implications The interviews allowed identifying nurses' preferences on information organization and information content. However, some nurses could not intuitively identify the function or shown values of some display elements. Therefore, the intuitiveness and usefulness of the display might be improved through more formal evaluation and improvements. 64 Refined prototypes The second development step had four display iterations which were refined based on nurses' feedback. Overall goals The aim of this development phase was to improve intuitiveness of display elements and reading accuracy of values shown by these elements. Overall, we aimed to increase both measures to an average of 95% correct answers and to archive a high perceived usability for the final prototype design iteration. Overall methods Questionnaires can be used to solicit detailed feedback from larger user groups, and to expose all respondents to uniform stimuli.106, 111 Questionnaires were developed which showed essential elements and details of the display. Setting With Institutional Review Board approval, the study was performed at the Medical-, Surgical-, and Burn Trauma ICU as well as the Neurological Critical Care of the University of Utah Heath Science Center. Instruments A nine page questionnaire measured intuitiveness and correct extraction of display information and collected comments. A cover letter asked nurses to 65 complete the questionnaire alone without help from others and introduced them to the purpose of the display. Questionnaires were created using InDesign (Adobe, San Jose, CA), and were designed according to a multiple choice guideline112 to have the question text in the multiple choice stem and short answer options. We operationalized intuitiveness as the rate of correct identification of display elements, and usability as the rate of correct information extraction. Thirty-four multiple choice questions probed intuitiveness for depicted display elements, or correct extraction of the depicted information (see Appendix B). Twenty questions asked nurses to name display elements. Fourteen questions asked nurses to retrieve values shown by the display elements. First, nurses named display elements of the prototype's main screen. Then nurses identified details about the display elements and elements on additional display screens. The questionnaires showed different states of the patient and treatment, e.g., a vital sign trend which showed the influence of a medication, a new order, current medication which was nearly empty, or IV incompatible medication. Appendix B reports the questions used. Usability was measured for the last prototype design iteration and included in the last questionnaire. Nurses were asked for the eight main display elements on a 7-point Likert scale. "I prefer this presentation of information over the presentation I currently use" (strongly disagree, disagree, somewhat disagree, neither agree nor disagree, somewhat agree, agree, strongly agree). 66 Procedures Questionnaires were distributed to the charge nurses in each ICU at the beginning of the night shift (around 7pm). The charge nurse distributed the questionnaires and collected completed questionnaires. At 1am, completed questionnaires were collected. Instructions for nurses asked them to complete the questionnaires alone without the help of their colleagues. Excel (Microsoft, Redmond) was used to calculate the average of correct answers. Each question had the same weight and contributed equally to the result. Nurses' comments were not statistically analyzed but incorporated - if possible - in the next design iteration. Nurses' usability rating (7-point Likert scale) was summarized for agreement. Fourth iteration The aim of this evaluation was to identify areas of improvement for recognition and extraction of information. It was performed based on the fourth prototype. Methods Seven nurses participated in the first questionnaire. One was working at the Medical ICU, four at the Surgical ICU, and two at the Neurological Critical Care. Their average age was 40.3 years (standard deviation SD 15.1), their daily computer use was 4.5 hrs (SD 4.5), and their years working at ICU settings were 10.8 (SD 12.8). 67 Results Table 7 and Table 8 show that deficits existed in the intuitive recognition of current, scheduled, and PRN medication, as well as trending of vital signs and medication and in the correct extraction of information in all elements of the display. This iteration aimed to measure if improvements of the sixth prototype in specific display areas could improve nurses' recognition and extraction of information in these areas. Table 7 Fourth iteration - recognized display elements (percentage correct) Correctly recognized Average 86% Trending 86% Currently administered medication 76% Scheduled medication 17% PRN medication 83% Additional medication information 100% Fluid balance 100% Table 8 Fourth iteration - extracted information (percentage correct) Correctly recognized fourth iteration Average 59% Trending 62% Currently administered medication 52% Scheduled medication 43% PRN medication 43% Fluid balance 67% 68 Fifth iteration Methods Sixteen nurses participated in the second questionnaire. One was working at the Medical ICU, five at the Neurological Critical Care, and ten at the Surgical ICU. Their average age was 32.7 years (standard deviation SD 8.3), their daily computer use was 4.3 hrs (SD 4.9), and their years working at ICU settings were 4.9 (SD 5.3). Results Table 9 and Table 10 show that major deficits exist regarding correctly identifying display elements and extracting information. As the questions of the fifth iteration did not cover the same elements as in iteration four, the results cannot be compared with the previous iteration. Sixth iteration The sixth evaluation step aimed to measure if improvements in specific display areas could increase recognition and extraction of information in these areas based on the sixth prototype. Methods Twenty-two nurses participated in the third questionnaire. Five were working at the Medical ICU, seven at the Surgical ICU, five at the Neurological Critical Care, and five at the Burn Trauma ICU. Their average age was 32 years 69 Table 9 Fifth iteration - recognized display elements (percentage correct) Fifth iteration Average 88% Trending 88% Currently administered medication 92% Scheduled medication 80% PRN medication 88% Additional medication information N/A Fluid balance N/A Table 10 Fifth iteration - extracted information (percentage correct) Fifth iteration Average 76% Trending 78% Currently administered medication 90% Scheduled medication 75% PRN medication 75% Fluid balance 100% (standard deviation SD 10.5), their daily computer use was 3.6 hrs (SD 3.6), and their years working at ICU settings were 6.2 (SD 9.9). Questionnaires covered a selection of display elements. Results can therefore not be compared to the results of the first and last evaluation. Results Table 11 and Table 12 show that some deficits exist regarding correctly identifying display elements and major deficits regarding the correct extraction of information. As the questions of the sixth iteration did not cover the same elements as in iteration four and five, the results cannot be compared. 70 Table 11 Sixth iteration - recognized display elements (percentage correct) Sixth iteration Average 92% Trending 92% Currently administered medication 86% Scheduled medication 71% PRN medication 77% Additional medication information 98% Fluid balance 100% Table 12 Sixth iteration - extracted information (percentage correct) Sixth iteration Average 86% Trending 97% Currently administered medication 61% Scheduled medication 95% PRN medication 95% Fluid balance 100% Seventh iteration The overall evaluation step aimed to measure if the improvements of the display overall would result in a recognition and extraction of information of at least 95% and to measure perceived usability when using the seventh iteration. Methods Twenty-three nurses participated in the fourth questionnaire. Five were working at the Medical ICU, ten at the Surgical ICU, three at the Neurological Critical Care, and five at the Burn Trauma ICU. Their average age was 34.7 years (standard deviation SD 9.2), their daily computer use was 7.4 hrs (SD 4.4), 71 and their years working at ICU settings were 4.7 (SD 6.4). Results Correct recognition Table 13 and Table 14 show that compared to the fourth iteration the overall percentage of correctly recognized elements reached to 96% and the correctly extracted information was 97%. Compared to the fourth iteration, correctly recognition increased on average by 10% with the largest increase of 83% in the recognition of scheduled medications. The correct extraction increased on average by 38% with the largest increase of 57% in scheduled and PRN medication. Perceived usability. Table 15 shows that 86% nurses preferred the integrated display over the displays they currently use in the ICU. Table 16 lists relevant improvements in human factors principles between the fourth and the seventh prototype. Many display elements differ on a smaller scale such as labels and icon shape. Improvements were made in labeling, the inclusion of multiple indicators for similar values, higher saliency of relevant information, and better resemblance of display elements to the depicted values. Figure 10 depicts these changes for the elements used in medication management. 72 Table 13 Sixth iteration - recognized display elements (percentage correct) Seventh iteration Average 96% Trending 94% Currently administered medication 97% Scheduled medication 100% PRN medication 87% Additional medication information 95% Fluid balance 100% Table 14 Seventh iteration - extracted information (percentage correct) Seventh iteration Average 97% Trending 95% Currently administered medication 100% Scheduled medication 100% PRN medication 100% Fluid balance 100% Table 15 Usability evaluation of the seventh iteration Main screen Communication Medication Additional med. Info Fluid balance Trending Alarm other patient Ventilator I prefer this presentation of information over the presentation I currently use. Somewhat - strongly agree 86% 86% 91% 71% 95% 78% 91% 95% 73 Table 16 Improvements of data presentation in the seventh prototype iteration compared to the fourth iteration and improved human factors principle Element Fourth design iteration Seventh design iteration Human factors principle Communica-tion -Voicemail icon looked like speaking person -Messages were unlabeled icons -Voicemail icon looked like recorded tape -Messages were labeled icons -Mental model54 -Labeling7 Medication -Current, scheduled and PRN medication had same location and comparable icons -In the current medication, the time until empty looked like fluid level in medication bag -Different location for scheduled and PRN medication and list representation -In current medication, icons were changed to hour glasses -Proximity of related information49, 50 -Pictorial realism55 -Mental model54 -Redundant coding7 Medication additional information -Nurses comments indicated the need for medication charting -Medication charting was added -Elements were rearranged by relatedness -Proximity of related information 49, 50 Fluid display -Intake and output were ordered by source -Balance looked similar to other data elements -Sources of intake and output were ordered according to positive or negative -Overall balance was separated and printed in bold at the side of table -Proximity of related information49, 50 -High saliency53 -Addressing user character-istics 11, 14, 56 74 Table 16 Improvements of data presentation in the seventh prototype iteration compared to the fourth iteration and improved human factors principle Element Fourth design iteration Seventh design iteration Human factors principle Trending -Trending curve scale had small numbers -Medication trend shown as blocks, small scale -Trending curve scale had larger numbers -Medication trend was shown as small bubbles and enlarged scale -Separating space was added to the trend -High saliency53 -Mental model54 -Labeling7 Table 16 continued Figure 10 Changes to the seventy prototype iteration compared to the third iteration 75 76 Discussion and implications Through the iterative design process of the integrated display, the availability and accessibility of information could be improved, and a display could be created which nurses preferred over the one they currently have. However, it needs to be evaluated it the final display prototype can influence nurses' situation awareness e.g. by improving the availability of information. During the design phase we could not test the influence of the prototype on situation awareness, or if the prototype improves task performance time, workload, and satisfaction. Therefore, a formal evaluation of the display compared to using conventional displays was performed and is described in the following chapter. CHAPTER 5 EVALUATION OF AN INTEGRATED INFORMATION DISPLAY 78 The integrated display aimed to improve the information presentation and availability for nurses, and thereby influences environmental and computer system characteristics. It focused especially on supporting frequent and safety-critical work areas of medication management, assessment of the patient's state, and team communication. Therefore, an evaluation should focus on measuring the influence of the change in system characteristics on nurses' work. The aim of this investigation was to assess nurses' performance when using the new information display compared to using conventional devices as a control. Objectives were to measure their a) situation awareness, b) response time, c) workload, and d) user satisfaction. Information shown on the integrated display and included in the evaluation came from the patient monitor, the electronic medical record, a medical reference library, the ventilator, and infusion pumps. However, it still has to be shown if the integrated information display improves human performance when completing real nursing tasks. Methods Setting This study was conducted at the University of Utah Health Science Center, a 383-bed tertiary care medical hospital and regional referral center. The study was performed in the Burn Trauma ICU (BTICU) in the BTICU break room. The unit was selected as the nurse manager agreed to let the unit's nurses participate during their breaks. On the unit work a total of 60 nurses and 20 nurse 79 assistants. It has twelve beds and six nurses working per shift. Patients were pediatric and adults: thermally, chemically, or electrically injured, and from other medical fields requiring critical care services. The average daily census is about 11, annual admissions between 375-400, and the BTICU is verified as a burn center by the American College of Surgeons and is a Level I Trauma Center. Participants Participants were invited to participate if they were currently working at the ICU during two consecutive night shifts of the tests, and if they were working in the role of a nurse. Excluded were nurse assistants, learning nurses, and nurses who were only working a single night in a row. Night shifts were selected because nurses were less likely to be interrupted during the test and the break room was available. The institutional human subjects review board approved the study (ClinicalTrials.gov identifier: NCT00714012). Familiarity of nurses with the displays Nurses were asked prior to the evaluation for their experience with the control devices. The familiarity with the patient monitor was a medial of 2.3 years, the electronic medical record 3 years, and medication database 2 years. They were not familiar with the infusion pump and the ventilator. Several nurses had participated in the design of the integrated display. To compensate for possible familiarity with displays, all participants received standardized training on all displays prior to the test independent of their familiarity. The information shown 80 on the Integrated display was completely new to nurses as it was different from the data displayed during the design phase. Study design The study used a counter-balanced 2 display (Control vs. Integrated) by 3 scenario (Medication management, Patient awareness, Team communication) repeated-measure design. The design was counter-balanced across display type and scenario. Each subject was exposed to one of the 6 possible combinations of the two design factors using Latin square, see Table 17. Instrumentation Displays The practicability of evaluations using paper based prototypes is high - users evaluating the same device on paper and on computer screens found the same percentage of usability problems although it took them 30% longer.113 Prototypes with varied information were used, dependent upon the questions (see Appendix C). Paper based prototypes were the method of choice because of not all data Table 17 Presentation order of display and scenario Participant # First Display Second Display Scenario sequence 1,7 INTEGRATED CONTROL A B C 2,8 INTEGRATED CONTROL C A B 3,9 INTEGRATED CONTROL B C A 4,10 CONTROL INTEGRATED A B C 5,11 CONTROL INTEGRATED C A B 6,12 CONTROL INTEGRATED B C A 81 shown on the Integrated display was available through interfaces from existing devices. Paper based prototypes of the displays were created using Illustrator CS4 (Adobe, San Jose) and printed in the size of the original devices. Screens of each device were combined as booklets with tabs labeled according to the information provided. The first page of the booklet showed the initial device screen and the following pages showed additional screens. The initial and scrolled down screens were both included as separate tabs for devices where users had to scroll down to see information at the bottom of the screen. Data presentation tool Questions were presented on a custom web-based measurement tool that recorded whether the answers were correct and recorded the time taken from opening the question to submitting the answer. To simulate the time critical ICU environment, a 60sec hourglass was presented alongside with the question. If participants asked about the hourglass, they were instructed to answer as fast as possible but that their answers would be accepted even if the time expired. Integrated display Figure 11 shows the Integrated display prototype with vital signs, vital sign trends, alarm history, the vital signs for the nurse's second patient, medications being delivered, scheduled and PRN medication orders, medication IV compatibility and interaction, potential side effects of medications, fluid balance, 82 Figure 11 Integrated display - Main screen which shows vital signs, ventilator settings in the center (two were recently changed in yellow), fluid balance below, and on the right current and scheduled medication (one changed and one new order in purple83 and ventilator settings. The Integrated display prototype booklet had a total of 12 screens/pages. Related information was displayed in special proximity, e.g., for medication the current rate, order, and side effects to monitor. See Appendix C for the screens of the Integrated display. Control displays Figure 12 shows the Control displays. Control devices were selected based on current use at the BTICU. The ventilator currently in use could not show recent changes in settings, and the current infusion pump could not display the time until the delivered medication was empty. Therefore alternate devices were selected (see below) which provided adequate functionality for the test. The patient monitor (IntelliVue MP70, Phillips, Andover, MA) showed the patient's vital signs on the initial screen, and on additional screens vital signs of the nurses' second patient, past alarms, and a vital sign trend. The patient monitor prototype booklet had a total of 3 pages/screens. The electronic medical record (PowerChart®, Cerner, North Kansas City, MO) showed the patient's demographics on the main screen, and on additional screens fluid balance, the electronic medical record, and changes in orders. The prototype had 7 pages/screens. The medication reference (Micromedex® healthcare series, Thomson Reuters, NY) had 19 pages. On additional screens were IV compatibility, interaction, and potential side effects of current medication, as needed medication (PRN) and scheduled medication. 84 Figure 12 Paper based prototypes of the Control setting. Clockwise: Ventilator, medication reference - opened at the potential side effects tab, electronic medical record - opened at the scheduled medication tab, infusion pumps, and patient monitor 85 The ventilator (Evita XL®, Drägerwerk, Lübeck, Germany) showed ventilator settings on the main screen and recent changes in the settings on an additional screen. The prototype had 2 pages. The infusion pump (Outlook 100®, B Braun, Melsungen, Germany) showed the medication being delivered and the time left until it was empty on the main screen. The prototype was displayed on a single page. Measurements and data collection Situation awareness Situation awareness was measured as the accuracy of participant's answer to each question. Response time Response Time (seconds) - the time taken by the participant to answer each test question, from when the question is presented until the participant submitted his/her answer. Workload NASA Task Load Index92- measurement of the participant's perceived cognitive workload for all scenarios was completed after using each display. NASA TLX uses six dimensions: mental demand, physical demand, temporal demand, performance, effort, and frustration. (Appendix F) For each dimension a score from 0 to 100 is obtained using 20 step bipolar scales. A global score can 86 be calculated by combining the single scores with operator defined different weights. NASA TLX was found to have a high validity and reliability with Cronbach's alpha higher than 0.80.96 It was selected because it is easier to understand than the Workload Profile (WP),94 and less exhaustive than Subjective Workload Assessment Technique (SWAT), 93 having a comparable intrusiveness.95 User satisfaction QUIS98 Version 7 part 3 (Appendix G) was used to assess nurses' perceived usability of the displays with additional free-text questions. QUIS was found to have a high reliability and validity (Cronbach's alpha = 0.94),99 see Background. Four free text questions were used to additionally assess nurses' opinion and experience with both the Integrated and Control displays. • How was the availability of clinical information? • How was your overall sense of the patient and the treatment? • What did you like best? • What should be redesigned? Scenarios For the evaluation we used a direct experimental technique to measure nurses' situational awareness and their understanding of the situation. We employed an adaption of the Situation Awareness Global Assessment Technique (SAGAT) 82, 83 (see Background) with multiple choice questions organized in 87 three scenarios which were answered at the participant's own pace. 17 questions were developed by the author in consultation with a psychologist and ICU nurses (see Appendix C). They covered the three levels of situation awareness: Perception was defined as recognition accuracy of values on the displays, comprehension as accurate situation model, and projection as the accuracy of future estimation of events. Questions were deliberately created to be complex with a high number of simultaneous medications, multiple screens to look at and addressing information presentation deficits in the Control devices. Such deficits were e.g. that scheduled continuous infusions almost looked identical as running infusions, that medication interaction was listed alphabetically ordered by the first interacting medication, and that trends were hard to discriminate due to the small and overlapping display areas. Questions were organized in three scenarios and covered the most frequently performed and patient safety critical task areas identified in the observation study (Chapter three). Each scenario had 5-6 questions with 4-6 answer options each. Answers included the option "I do not know," "None" or "None of the above." for the majority of questions. The three scenarios were medication management, patient awareness, and team communication and are described in the following in further detail. 88 Medication management A busy colleague asked the participant to administer a medication through the same line as currently in use for other medications. The participant was asked to identify the drug name, dose and potential drug-drug interactions of the next PRN medication. An example using both displays for medication administration is shown in Figure 13. Patient awareness The participant was asked to assess the patient's condition using past alarms, vital signs values and fluid balance. The participant was asked to assess the condition of the nurse's second patient (in the adjoining room) without leaving the current room. Team communication The participant was asked to identify changes made by another health care team member to ventilator settings and medication orders. The participant was asked to tell a colleague which infusion pump medication needed to be refilled and which PRN medication was due within the next 20 min. Verification of scenarios Three ICU nurses reviewed the scenarios prior to the evaluation and verified that questions could be answered when using the prototypes. They adapted the scenarios to nursing language and improved scenario realism. They then rated 89 Figure 13 Example of display-interaction for medication administration with the currently used devices at ICUs (Before - Control devices) and display interaction using the Integrated display (Integrated). Arrows indicate the new location of the information. Before (Control devices): Currently, different devices are used to show 1. the medication order, 2. medication being delivered, 3. medication compatibility, 4. potential side effects, and 5. the patient's vital signs. Integrated: On the Integrated display, related information is displayed in close proximity. A. Nurses see the medication order directly. B. They scan the new medication and automatically see medication compatibility with current medication, potential side effects, and vital signs. [Human Factors and Ergonomics Society 54th Annual Meeting, San Francisco, 2010. "Integrated Information Displays for ICU Nurses: "Field Observations, Display Design, and Display Evaluation", S. H. Koch, N. Staggers, C. Weir, J. Agutter, D. Liu, D. R. Westenskow. Submission ID: 429 Reprinted with permission of the Human Factors Society.] 90 the scenarios to be realistic 6.5 (1=unrealistic and 7=realistic), clear 5.25 (1 = unclear … 7 = clear), and comprehensive 5.25 (1 = not comprehensive …7 = comprehensive), and reached the final cut of score of 5 for the evaluation. Procedures Nurses participated in the study during two consecutive nights. During the first night by using the first display, and during the second night using the remaining display (see Table 17). During the first night, participants were consented as required by the institutional human subjects review board (ClinicalTrials.gov identifier: NCT00714012). Prior to the study, participates received standardized training using all devices and all relevant functions and screens for the study through a self paced power point presentation. They subsequently answered a competency test. In case they answered all competency questions correctly they proceeded with the study. If they did not answer all questions correctly they received additional training and repeated the competency test. Failure to complete the test a second time was an exclusion criterion. During the study display prototypes were presented on a table. Participants read a clinical situation on a computer screen, looked at the paper-based displays, and answered the questions. After using each display, nurses completed the post tests NASA TLX, QUIS and answered free text questions. During the next night nurses performed the second part of the study by using 91 the second display (see Table 17). The same procedures were used as described above: Nurses received standardized training on the second display, passed the competency test, participated in the study with the second display, and answered post-tests questions about the second display. Data analysis All statistical analysis was performed using Matlab analytical software (release R2008b, The MathWorks, Natick, MA). Fisher's exact test suits for the analysis of small categorized samples, and its assumptions are a directional hypothesis, independent observations, random sampling, and dichotomous observations.114 Fisher's exact test did not allow simultaneous analysis for effects and interaction effects, so pair wise comparison and Bonferroni correction was used. Results of Fisher's exact test are p-values. Repeated measure Friedman's ANOVA is a non-parametric test that suits to analyze data that are not normally distributed and which has no equality of variance. It uses the ranks of the data rather than their raw values.115 In Matlab Friedman's ANOVA exact test does not allow simultaneous analysis for effects and interaction effects through a single analysis, so pair wise comparison and a Bonferroni corrected α-critical were used. Results of Friedman's ANOVA are p-value and χ2-values. 92 Situation awareness Fisher's exact test was calculated for the accuracy of the decision representing situation awareness. We used a 2x3 Fisher's exact test comparing the two displays for all three scenarios at a Bonferroni corrected αcrit0.05/3=0.0083. Additionally we calculated a 2x3 Fisher's exact test comparing the two displays for all three levels of situation awareness at a Bonferroni corrected αcrit0.05/3=0.0083. Response time Our measured response time neither met the assumption of normality nor of equality of variance for the standard 2x3 ANOVA. Therefore, we used a repeated measure Friedman's ANOVA (α=0.05). Seven comparisons of response time (global times, times the three scenarios, and times for the three situation awareness levels per display) were performed using a Bonferroni corrected αcrit=0.05/7= 0.0071. Workload Our measured data for workload scores neither met the assumption of normality nor of equality of variance for the standard 2x3 ANOVA. Therefore a repeated measure Friedman's ANOVA (α=0.05) was calculated. Additionally for workload the summed sub scale workload was calculated using equal weights for each sub scale. 93 User satisfaction Our measured data for the QUIS scores neither met the assumption of normality nor of equality of variance. Therefore, we used a repeated measure Friedman's ANOVA (α=0.05). Nurses' answers to the free text questions were classified into the two categories positive and neutral/negative by the author. Results Twelve BTICU nurses (8 female, 4 male) with a median age of 31.5 years (range 23 - 57) participated. They had worked in an ICU setting for a median of 3 years (range1 - 24 yrs). Their median nursing expertise was 7 (1=novice, 9=expert), competence 7 (1=low competence, 9=high competence), and confidence 7 (1=low confidence, 9=high confidence). Participants were using computers on average 5 hours per day. Two participants had to repeat the training and quiz before entering the study, but no one was excluded. The median study duration was 23 min (range 15-47 min). Situation awareness The average situation awareness level for the Control displays was 65% (ranging from 47% to 71%), and 86% with the Integrated display (ranging from 71% to 94%) Using the Integrated display nurses made their decisions 21% more accurate than using the Control displays. Table 18 shows the pair wise comparison of the correct answers to the 94 Table 18 Pair wise comparison to analyze main effect and interaction effects for situation awareness level and scenario (Fisher's exact test). Significant results are marked by asterisks (p≥0.001). Additionally the table indicated percentages of correct situation awareness questions Control p Integrated Overall 65% (132/204) <0.001* 86% (175/204) Perception * Display 65% (50/60) <0.001* 89% (64/72) Comprehension * Display 68% (41/60) 0.006* 90% (54/60) Projection * Display 61% (44/72) 0.028 79% (57/72) Medication management * Display 40% (29/72) <0.001* 81% (58/72) Patient treatment * Display 74% (53/72) 0.107 85% (61/72) Team communication * Display 83% (50/60) 0.153 93% (56/60) situation awareness questions for display, scenario and situation awareness level. Statistically significant results were found overall (p=1.1E-6 Fisher's exact test), for medication management (p<0.001), perception (p<0.01), and comprehension (p=0.06). The overall accuracy for the medication management scenario was 60% and for situation awareness level projection 70% indicating problems with this scenario and this situation awareness level. Situation awareness levels and scenario are correlated. Figure 14 shows the accuracy of answers to situation awareness questions by scenario and situation awareness level graphically. Table 19 shows the results for the questions that were answered incorrectly more than half the time. 95 Figure 14 Situation awareness answer accuracy. White=Control, grey=Integrated 96 Table 19 Questions answered incorrectly half of the time or more. The question numbers (e.g., A2 ) refer to the Appendix where all questions including correct answers are reported Question Control incorrect Integrated incorrect A2: "Please check which medications need to be administered within the next 5 minutes." 12/12 0/12 A4: You decide to administer Metronidazole and Drotrecogin together with the currently infusing meds (Hydrocortisone and Vancomycin). However, as the patient has only 2 lines you need to decide which of the lines to use. For which of the following medication combinations do you expect to see IV in-line incompatibility? Assume that not tested meds are incompatible. 9/12 5/12 A3: The attending is concerned that pharmacy has not checked drug-drug interaction for Gentamycin as the order was rushed. To protect your patient: Do you expect future drug-drug interaction while administering Gentamycin together with the currently infusing meds (Hydrocortisone and Vancomycin)? 8/12 4/12 B3: "Based on the SPO2 value during last HR alarm: What would you expect the first patient's SpO2 level to be if there was another HR alarm in the future?" 8/12 4/12 A6: "The patient says he has nausea and pain - please push a PRN medication. What issues are you most concerned about in this situation?" 7/12 4/12 A5: "Which of the following side effects of the currently administered medication Insulin and Gentamycin would expect to see and monitor?" 6/12 0/12 97 Response time Table 20 shows the pair wise comparison of the response time for display, scenario and situation awareness level. The median response time for the Control displays was 34sec (SD 51, ranging from 5.7 sec to 523 sec), and 25sec for the Integrated display (SD 26, ranging from 5.9 sec to 251 sec). Using the Integrated display nurses made decisions 26% faster than using Control. Nurses overall response time was significantly faster when using the Integrated display (p=1.8E-7, χ2=27.25, repeated measure Friedman's ANOVA). Among scenarios, medication management and team communication were answered significantly faster. Among situation awareness levels, perception questions were answered significantly faster (repeated measures Friedman's ANOVA). Table 21 shows the raw response time per display, scenario and situation awareness level, see also Figure 15 for a graphical representation. Situation awareness levels and scenarios are correlated. Table 20 Pair wise comparison to analyze main effect and interaction effects for response time (repeated measure Friedman's ANOVA). Significant results are marked with asterisks (p≥0.001) Factor p χ2 Display <0.001* 27.25 Perception * Display <0.001* 17.51 Comprehension * Display 0.011 06.41 Projection * Display 0.035 4.42 Medication management * Display 0.001* 12.67 Patient treatment * Display 0.035 4.43 Team communication * Display 0.001* 10.21 98 Table 21 Median response time (α-critical = 0.007). P-values are calculated using repeated measure Friedman's ANOVA Control (sec) Difference (sec) Integrated (sec) Display 34 9 25 Perception * Display 29 11 18 Comprehension * Display 34 9 25 Projection * Display 49 18 31 Medication management * Display 54 16 38 Patient treatment * Display 30 3 27 Team communication * Display 24 8 16 Figure 15 Response time -Time from seeing question until answer was submitted. White=Control, grey=Integrated 99 Workload The summed workload score was better for the Integrated display (7.1) than for the Control display (9.9). The median TLX score for the Control displays was 11 (ranging from 1 to 20). The Integrated display got a better median score of 5 (ranging from 1 to 18). Table 22 shows that nurses reported significantly less effort (p=0.03, χ2=4.45, repeated measure Friedman's ANOVA) and lower frustration (p=0.01, χ2=6.4) when using the Integrated display. The other measures of workload did not show a significant difference. Figure 16 shows the data graphically. User satisfaction The median QUIS score was lower for the Control displays (5.2, ranging from 2 to 9). The Integrated display got a higher median score of (7.7 ranging from 2 to 9). Table 22 Pair wise comparison of analyze effect of workload (repeated measure Friedman's ANOVA). Significant results are marked with asterisks (p≥0.05) NASA TLX measure Control (median) p χ2 Integrated (median) Mental workload 14.5 0.13 2.27 8.5 Physical workload 4 0.3 0.81 3 Temporal workload 12.5 0.2 1.60 7.5 Performance 6.5 0.5 0.4 5 Effort 14.5 0.03* 4.45 8.5 Frustration 10.5 0.01* 6.4 3.5 100 Figure 16 Perceived workload (NASA TLX) Table 23 shows the analysis with repeated measure Friedman's ANOVA. Significant differences existed for the dimensions Terrible…wonderful (p=3.9E-7, χ2=8.3), Frustrating…satisfying (p=6.7 E-3, χ2=7.36), Dull…stimulating (p=0.02, χ2= 5.33), and Inadequate power…adequate power (p=3.9E-3, χ2=8.33). See Figure 17 for a graphical representation of the data. Free text questions There were more positive comments for the Integrated display than for the Control display (see Appendix E for all answers). Positive answers about the availability of clinical information were 11 vs. 5 (Integrated vs. Control), e.g., "Great. It is very helpful to have all different kinds of information available from one source- instead of having to look several different places and compare." Equally higher was the number of positive comments about the sense of the Control Integrated 101 patient 12 vs. 5 (Integrated vs. Control). An exa