The development of online patient asistance at a remotely-controlled kiosk.

A touchscreen information kiosk was implemented to address patient concerns of not receiving enough health information from their healthcare providers. Included in this kiosk was a remote control assistance feature, which allowed a help-giver to view and take control of the kiosk from another computer. Four research questions were addressed in this paper: 1) Will patients use a touchscreen kiosk to look for health information? 2) Is online patient assistance using remote control a feasible alternative to help given in person by healthcare staff? 3) Does remote control assistance improve the patient's experience with the kiosk? and 4) What are the ramifications of using remote control technology for patients, librarians, healthcare providers, and system developers? The methods were largely qualitative and structured around the four research questions. Limited quantitative analysis was performed through a counterbalanced, repeated measures study done at an outpatient clinic (subjects = 64) to examine whether remote control assistance improved patients' experiences with kiosk use. The study found that: 1) High scores on patient questionnaires indicated general satisfaction with and acceptance of the touchscreen kiosk (median scores of fours and fives on a five-point Likert scale). 2) When asked to compare remote control assistance against other ways of receiving health information, patients rated it higher (median scores of fours and fives) against all other methods (e.g., brochures, television, friends, Internet) except against physicians or other health providers (median score of three). 3) Based on patient questionnaires, remote assistance improved their satisfaction, increased ease of use, and raised the number of appropriate document retrieved from 5.5% to 86.7%, It was found that remote assistance gave patients convenient access to the librarians' skills and encourages them to become active seekers of their own educational materials. Such encouragement could have long-term impacts for creating a better-informed patient population. As compared to routine assistance give over the telephone, librarians using remote control were providing better service while spending the same time per patient. For healthcare provider, the remote kiosk can complement their existing arsenal of patient education resources. Further studies are recommended to explore other aspects of remote control assistance for patient education kiosks.

THE DEVELOPMENT OF ONLINE PATIENT ASSISTANCE AT A REMOTEL Y-CONTROLLED KIOSK by I-Kai Frank Lin A thesis submitted to the faculty of The University of Utah in partial fulfillment of the requirements for the degree of Master of Science Department of Medical Informatics University of Utah May 2001 Copyright © I-Kai Frank Lin 2001 All Rights Reserved THE UNIVERSITY OF UTAH GRADUATE SCHOOL SUPERVISORY COMMITTEE APPROVAL of a thesis submitted by I-Kai Frank Lin This thesis has been read by each n1ember of the following supervisory committee, and by majority vote has been found to be satisfactory_ Chair: ")-25··0 ( THE UNIVERSITY OF UTAH GRADUATE SCHOOL FINAL READING APPROVAL To the Graduate Council of the University of Utah: I have read the thesis of I-Kai Frank Lin in its final form and have found that (1) its format, citations, and bibliographic style are consistent and acceptable~ (2) its illustrative materials including figures, tables, and charts are in place; and (3) the final manuscript is satisfactory to the supervisory committee and is ready for submission to The Graduate School. Date Bruce E. Bray , Chair. Supervisory Committee Approved for the Major Department ~r">.yvtA~ Reed M. Gardner . Chair/Dean Approved for the Graduate Council ~6:....\ S.C~--. David S. Chapman Dean of The Graduate School ABSTRACT A touchscreen information kiosk was implemented to address patient concerns of not receiving enough health information from their healthcare providers. Included in this kiosk was a remote control assistance feature, which allowed a help-giver to view and take control of the kiosk from another computer. Four research questions were addressed in this paper: 1) Will patients use a touchscreen kiosk to look for health information? 2) Is online patient assistance using remote control a feasible alternative to help given in person by healthcare staff? 3) Does remote control assistance improve the patient's experience with the kiosk? and 4) What are the ramifications of using remote control technology for patients, librarians, healthcare providers, and system developers? The methods were largely qualitative and structured around the four research questions. Limited quantitative analysis was performed through a counterbalanced, repeated measures study done at an outpatient clinic (subjects 64) to examine whether remote control assistance improved patients' experiences with kiosk use. The study found that: 1) High scores on patient questionnaires indicated general satisfaction with and acceptance of the touchscreen kiosk (median scores of fours and fives on a five-point Likert scale). 2) When asked to compare remote control assistance against other ways of receiving health information, patients rated it higher (median scores of fours and fives) against all other methods (e.g., brochures, television, friends, Internet) except against physicians or other health providers (median score of three). 3) Based on patient questionnaires, remote assistance improved their satisfaction, increased ease of use, and raised the number of appropriate document retrieved from 56.5% to 86.7%. It was found that remote assistance gave patients convenient access to the librarians' skills and encouraged them to become active seekers of their own educational materials. Such encouragement could have long-term impacts for creating a better­informed patient population. As compared to routine assistance given over the telephone, librarians using remote control were providing better service while spending the same time per patient. For healthcare provider, the remote kiosk can complement their existing arsenal of patient education resources. Further studies are recommended to explore other aspects of remote control assistance for patient education kiosks. v To my parents, who have sacrificed so much for my education. TABLE OF CONTENTS ABSTRACT.................................................................................................................... IV ACKNOWLEDGMENTS ................................................................................................. ix INTRODUCTION... ....... ......................... ............. .............. ............................................... 1 RESEARCH QUESTIONS...................................................................... 4 REVIEW OF THE LITERATURE...... .................................................. ............. ............ 5 METHODS... ................... ....................................................................................... 7 Kiosk Development and Pilot Evaluation...................................................... . . .. 7 Kiosk Support Features............................................. ........................ .......... ....... 7 Remote Assistance Evaluation.......................................................................... . . 8 Implications of Kiosk Technology for Healthcare.............................................. 12 RESULTS ........................................................................................................................ 13 Kiosk Development and Pilot Evaluation ... ;................................................. . . . .. 13 Kiosk Support Features........... ....... .... ....... ....... ......... .............. ....... ......... .... ....... 15 Remote Assistance Evaluation.......................................................................... .. 16 DISCUSSION .................................................................................................................. 22 Implications for Patients ..................................................................................... 22 Implications for Librarians .......................................................................... 23 Implications for Healthcare Providers ............................................................. 24 Implications for System Designers ................................................................. 25 Problems ......................................................................................................... 25 Privacy ............................................................................................................ 26 Computer Experience ........................................................................................... 27 Remote Control vs. Reference Librarian .......................................................... 27 Evaluation of Retrieved Documents ............................................................... 28 Limitations ........................................................................................................ 28 CONCLUSION ................................................................................................................ 29 Appendices A. PILOT KIOSK QUESITONNAIRE ........................................................................... 31 B. KIOSK SCREENSHOTS ...................................................................... 33 C. OUTLINE OF KIOSK TUTORIAL ......................................................... 41 D. CONTROL SESSION QUESTIONNAIRE ................................................. 42 E. EXPERIMENTAL SESSION QUESTIONNAIRE ......................................... 43 F. DOCUMENT RATING FORMS .............................................................. 44 REFERENCES ................................................................................................................ 45 Vin ACKNOWLEDGMENTS I would like to thank the University of Utah, and especially the Department of Medical Informatics, for giving me this chance to complete my thesis. Thanks must also be given to the staff members at the U Family Health Clinic and librarians Jennifer Bosgraaf and Kasandra Mauger at the Fox Eccles Medical Library. Special thanks are given to members of my committee, who have shown tremendous patience and willingness to work with me in completing this project. INTRODUCTION Patient education is an important component of modem health care. In one observational study of seven clinical sites, it was found that of the time health providers spent talking to patients, 37% was used for patient education. 1 The practice of teaching patients seems to be consistent across the various fields of medicine, as both primary care and specialty care clinics were sampled for the study. Patient education material gives patients a degree of knowledge and autonomy with regard to their own health care. This is important because a person's control over his or her life n1ay be threatened during an illness.2 Becoming informed is a common method by which patients try to regain control in these situations.3 Although the exact role an individual wishes to play in his or her own health care will vary, one thing remains consistent across the studies: the desire for information about care is generally high.4 In a focus group study conducted by Tang, the general information needs of patients were explored.5 The participants of the focus group indicated they wanted more information about their illness and treatment plan than they typically received during the physician encounter. One possible explanation is that patients often formulate their questions after the initial office visit. Patients look to answers to their questions in a variety of places including friends, relatives, pharmacies, libraries, and the World Wide Web. 2 The focus group participants indicated that they preferred to receive material that their physician had endorsed. This is of particular importance given the amount of inaccurate or outdated medical information on the Internet. A study was done in 1998 to assess the accuracy of medical information on the Web with regard to treatment of pediatric diarrhea.6 Sixty Internet web sites were examined, including medical centers, news services, and health departments. Only 20% contained information that conformed to current American Academy of Pediatrics treatment guidelines for pediatric diarrhea. The authors concluded that practitioners needed to warn their patients about the quality of all medical information obtained from the Web, even those that seem to come from reliable sources. Despite its potential for misinformation, the Web can still be an invaluable medium for distributing health information. In 1999, Hoch evaluated information on epilepsy found on Internet web forums and message boards.7 Unlike static web pages that are not frequently updated, this web forum was staffed by caregivers who answered questions posted by patients. Only 6% of the information was found to be objectively inaccurate. Another important reason for educating patients is to remind them of relevant medical facts. This is important because patients often do not remember everything that was discussed during the office visit, sometimes forgetting even the most significant findings of the encounter. In a study of patient-physician recollections of medical diagnoses, Scheitel found that patients "failed to report 680/0 of all health problems and 54% of the most important health problems diagnosed by the physician" within three weeks.8 The authors speculated that this was a primary reason for patient noncompliance and suggested postencounter handouts as a possible solution. 3 The University of Utah Hospital has been searching for a way to address patients' unanswered informational needs. An ideal patient learning center should contain comprehensive medical information as well as be staffed by a full-time medical librarian to help individualize searches. However, this was not feasible because of space and other resource limitations, and alternative approaches were considered. The goal of this project was to create something that would serve the purposes of a patient learning center while considering space and other resource constraints. The feasibility and practicality of a touchscreen computer kiosk was tested as a medium for delivering health information to patients in a hospital-based, outpatient environment. Furthermore, an experimental remote control assistance feature was added to address situations where patients were not able to find information that satisfied their needs. Through this feature, a remote helper (e.g., a medical librarian) could view and take control of the kiosk's computer screen to assist with information searches. This proj ect examines the usefulness of remote control assistance in a patient information kiosk, and presents a discussion of the ramifications and implications for its future use. RESEARCH QUESTIONS Due to the exploratory nature of this study and the fact that it was not primarily designed to include experimental interventions per se, research questions, rather than hypotheses, were employed. The research questions that guided the data analysis were the following: 1. Will patients use a touchscreen computer kiosk to look for health information in an outpatient clinic setting? What are the user interface, navigation, and content issues that might be encountered? 2. How will patients use available assistance to improve their experience with the computer kiosk? Is remote live assistance a feasible altenlative to help given by local personnel? 3. Does remote assistance improve the patient's experience with using the kiosk? 4. What are the ramifications of using this remote assistance kiosk in the clinic for patients, librarians, and healthcare providers? What are the technological implications for future system development? Although much of the data presented in this study is qualitative in nature, limited amounts of quantitative data were also collected. The majority of these came from an experimental study that addressed the question of whether remote assistance improved the patient's experience with using the kiosk. REVIEW OF THE LITERATURE Computers have been used in patient education for many years, with varying degrees of success under a variety of conditions. Existing literature showed that many results at least equaled those using traditional methods, if not surpassing them.9 - 11 Computers have also been more successful than traditional methods in certain aspects, such as having a greater distribution range and increased user interaction with the presentation. In a project sponsored by the New York Online Access to Health (NOAH), computers were shown to be an inexpensive and effective way of providing bilingual health information to a broad public. 12 A computer can present health information to patients in a number of ways. Whether it be a simple static page of text or a fully developed multimedia kiosk, it is important that the patient be comfortable with the method of presentation. Touchscreens have previously been used in healthcare settings with a fair degree of success. A study done in 2000 found that 95% out of 1,440 patients felt "comfortable" with using a touchscreen to fill out history information. 13 Other studies were less quantitative, but generally acknowledged the touchscreen to be an acceptable medium of information exchange. 14 ,15 Survey results from internal testing of this project's prototype also showed that patients had no trouble using a touchscreen. 6 In a pilot study done by Lewis in 1997, it was shown that a touchscreen kiosk is both feasible as an interactive learning tool and as a useful method to deliver diabetes information to patients. 16 In the study, 91 % of the users thought the touchscreen kiosk would be a beneficial way to receive patient information, and a majority preferred it to more traditional media such as pamphlets and television. Remote control software has traditionally been used for technical support at information desks l7 or for various forn1s of distance learning. 18,19 The software solves two main problems encountered during remote operations: 1) the difficulties in communicating exact onscreen contents to the other user and 2) the inability to demonstrate a desired action. No studies were found that specifically discussed using remote control software to assist with information searches. A variant of remote control assistance is the live online chat assistance that is b ecom.lng I.ncre.asIn g common Clo r tech n'I caI support on the We b .20'2 1 U S.I ng an onscreen chat box, technical support personnel can communicate directly with customers in real time. This is similar to getting help over the phone except all the interactions are through the chat box on the computer screen. Remote control assistance expands on this idea by allowing a ren10te operator, the medical librarian in this project, to take full control of the computer at which assistance is needed. METHODS The methods employed in this project were largely qualitative in nature and were structured around the four research questions mentioned. However, a more quantitative approach, in the form of an experimental study, was taken to address the question of whether remote assistance improved the patient's experience with using the kiosk. Kiosk Development and Pilot Evaluation The first research question of whether patients will use a computer to look for health information was addressed by developing an interactive, touchscreen patient kiosk prototype in October 1998. Initially, the scope of the kiosk was broad and included patient education material as well as customer service content, such as maps of the hospital and answers to frequently asked questions. Internal testing led to several cycles of iterative design and ultimately resulted in a kiosk prototype that was piloted in the summer of 1999 in a hospital-based outpatient clinic. Feedback from 20 patients was obtained by way of a one-page questionnaire (Appendix A) that each patient filled out after using the kiosk. Kiosk Support Features The second research question asked what kinds of assistance features could improve the patients' experiences with the computer kiosk. Automated help was tested by way of an online tutorial of the kiosk's features. Since one purpose of the kiosk was to 8 save valuable personnel resources, it was important that the kiosk be a stand-alone system requiring little maintenance. The ability of the kiosk to be an independent provider of health information was tested by leaving it unattended throughout most of the pilot study. Informal feedback was obtained from clinic and patient education personnel. A live, remote control assistance feature was later added to improve the ability to locate health information. Remote Assistance Evaluation An evaluation of the remote control assistance feature was the focus of the third research question. A formal study compared the experiences of patients who used the kiosk with and without the remote assistance option. Kiosk Tool Description The kiosks used were two Pentium®-class computers running Windows 95®. Both computers had 10 Mbps Ethernet connections to the Internet. The two touchscreens were a 19" and a 17" Microtouch® monitor. The kiosk program was a free trial version ofNetKey® Version 3.0 by Lexitech.22 NetKey® was a kiosk browser program that ran the core interface functions of the kiosk computer, including calling a web browser to display Web content. It also provided additional features such as URL blocking and session logging. The web browser was Internet Explorer® 5.0. Graphics were mostly designed using Microsoft's Image Composer®, and the HTML was built using Homesite® by Allaire Corp. The remote control software was Symantec's PC Anywhere® Version 9.0. 9 To request a remote assistance session, the kiosk user pressed a button on the screen labeled "Live Help" (Appendix B). The user would then be given the telephone number to the medical library and be instructed to call using a telephone located next to the kiosk. Upon receiving the call, a medical librarian activated PC Anywhere and started a remote control session. In most cases, the librarian talked to the user on the phone while helping with the search on the kiosk screen. Database Selection Databases used in the pilot included the MicroMedex CareNotes® and the patient handouts available through the Office of Patient Education. During evaluation of the remote assistance, the database selection was expanded at the librarian's request to include PubMed by the National Library of Medicine, online medical textbooks by StatRef®, and a medical dictionary on Medterms.com. Subject Selection and Recruitment The subject pool included all patients who had a scheduled appointment at the Family Health Center from May to August 2000. No consideration was given to the subject's race, gender, or medical diagnosis in determining eligibility for the study. Minors under the age of 18 were excluded from the study, as were subjects who were physically or mentally unable to operate a computer, keyboard, and mouse. Patient contact information was generated through the clinic's scheduling program, and everyone who had an appointment the following day was contacted and recruited by telephone. Information listed included the patient's age, gender, and reason for visit, although only age was used to screen recruitment. 10 Study Desi gn The study employed a counterbalanced repeated measures design, a methodology that used the subjects as their own control to minimize the effects of demographic variables.23 Recruited subjects participated in both the control and the experimental sessions. The two sessions differed in that subjects had the option of using the remote control assistance only during the experimental session. Counterbalancing was achieved by having approximately equal numbers of subjects perform each of the sessions first. So for instance, half of the subjects participated in the control session first, while the other half first did the experimental session. Counterbalancing was important because it minimized the effects of practice, fatigue, and carry-over on the subjects. Study Procedure After signing the informed consent form, subjects were assigned to either the experimental or the control group first. Ideally, the assignment would be sequential so that every odd-numbered subject would be using the control kiosk first. However, scheduling conflicts and room availability problems made the two groups only approximately equal. Subjects were then given a short scripted tutorial (Appendix C). To simulate a kiosk that was unattended, the investigator left the room and was not available for questions. Having the investigator leave also gave patients the perception of privacy for more sensitive medical questions. Data Collection The subjects were instructed to write down their search topic on a piece of paper, so that the results could be evaluated vis-a-vis the search goals. For an added sense of 11 privacy, the piece of paper was sealed in an envelope. At the end of each session (control and experimental), the subjects filled out an exit survey. The investigator recorded the starting and ending times for each session. The subject's privacy was protected by the fact that there was no identifying information contained in the surveys, as they were kept confidential by the investigator. Approval from the Institutional Review Board was obtained prior to data collection. Statistical Methods and Data Analysis24 Several sets of data were generated from this project. The first came from the pairs of patient questionnaires, one for the control and one for the experimental session (Appendix D & E). Both questionnaires contained itemized questions that asked the patients to rate individual aspects of the kiosk on a five-point Likert scale. The differences between each subject's two questionnaires were calculated, and these data were analyzed using a paired-sample Wilcoxon Signed Rank test. Because a meaningful mean cannot be generated from Likert-scale data, the use of a paired-sample t test was rejected. Similar analyses were done for other parts of the questionnaires. The second part of the control questionnaire asked for the subject's demographic information, while the experimental session questionnaire asked the subject to rate various aspects of the remote assistance feature using the Likert scale. Median or mean values were computed for these two sections. The variables of age, education, and hours per week of previous computer use were tested for correlation with ratings of the computer kiosk using a Pearson correlation coefficient. Printed copies of all documents retrieved by subjects were sent to a medical librarian and an internist physician who assessed their accuracy and relevancy. This was achieved by evaluating each docun1ent against the subjects' search goals that they had written down at start of the experiment. A standardized form was used to record each rater's scores (Appendix F). The raters were also asked to place the documents into one of four reading level categories. The rating scores were then analyzed using a Mann­Whitney test. The threshold for statistical significance was set at the alpha=O.05 level for all tests, as per the standard in scientific study. Implication of Kiosk Technology for Healthcare 12 Observation and exit interviews identified the ramifications of using the remote assistance kiosk in the clinics. Formal, tape-recorded exit interviews were done with each of the librarians involved in the study. Informal interviews were conducted with most of the patients following kiosk use, and informal discussions were held with healthcare providers and system developers. RESULTS Kiosk Development and Pilot Evaluation Feedback from the first prototype identified many design issues that were worthy of discussion. For instance, text size was a problem because a small font was hard to read for many older patients. Since most of the kiosk's content was hypertext with embedded links, smaller fonts resulted in text that was physically smaller on the screen. This made choosing a link on the touchscreen more difficult. On the other hand, a large font size limited the amount of material that could be displayed. Cascading Style Sheets (CSS) and Internet Explorer's Accessibility options were used as a solution because they allowed the link text to be selectively enlarged. Similar concerns were encountered with the buttons used on the kiosk's user interface. Smaller buttons were harder to press, while larger ones limited the space available to display content. Similarly, greater number of buttons made the interface more complex while fewer numbers decreased the functionality. For data input, the prototype kiosk had only the touchscreen. Textual input was done via an alphabetical onscreen keyboard. Both the mouse and the physical keyboard were taken away to simulate a stand-alone kiosk. An online kiosk design guide was used for these design decisions.25 Database selection was another important issue. Adam.com and Healthwise were not considered due to high licensing costs. Drkoop.com provided reliable medical information free of charge, but a document formatting incompatibility prevented it from working on the prototype kiosk. A final decision was made to include only the Micromedex Carenotes@, for which the hospital already had a subscription. Patient handouts, available through the Office of Patient Education, were also included via a hypertext link to the Office's homepage. The combined factors of cost, accessibility, formatting, and document reliability determined the final database selection. 14 Session logs as well as user surveys were used for analysis. Likert scale questions showed that patients were generally interested in using the kiosks and were satisfied with most aspects of their experience. However, several problems were identified during this prototype period. First, patients found the onscreen, alphabetical keyboard confusing, preferring to use a physical keyboard in the standard QWERTY configuration. Second, although the majority of searches produced some results, patients had trouble finding information that was both relevant and detailed enough for their needs. For instance, a patient looking for treatment options on plantar warts found a document that, though containing accurate medical information, had only a cursory discussion on current treatments. Third, a significant nun1ber of searches failed because of misspelling. As an example, patients spelled diabetes more than 10 different ways (diabetis, dibetis, diabetice, etc. ). Fourth, access from the kiosk to other Internet sites should have been restricted. Session logs showed that patients accessed questionable content from sources not included in the study during the night hours when the kiosk was left unattended. Physical security and lock-down of the kiosk equipment appeared to be needed. Based on the results of the pilot implementation, several changes were made to the kiosk design. The onscreen keyboard was dropped in favor of the more traditional mouse and keyboard combination, although the touchscreen remained available. The kiosk interface was simplified so that it had fewer buttons which were also bigger and more streamlined in function. The customer service content was also dropped, as it was not relevant for the later parts of the proj ect. Kiosk Support Features 15 Observation of the patients, as well as informal exit interviews, provided valuable data regarding the second research question. Although there was an online tutorial included with the first prototype, patients tended to bypass it and instead asked nearby staff members for help when they had questions. However, this created a problem because clinic staff was not knowledgeable about the kiosk. The clinic staff consequently referred most of the questions to the Office of Patient Education, which was in tum swamped by these requests. During the later part of the pilot period, the kiosk was moved to the front desk area of the hospitaL Similar problems occurred, as the front desk volunteers had to refer many of the kiosk problems back to the Office of Patient Education. In order to improve the kiosk's support capabilities, a live, remote control assistance function was added. It quickly became apparent that patients preferred talking to a live person, rather than using an automated online tutoriaL Although staff from the Office of Patient Education was able to answer many of the questions, their work process was not set up to handle the volume of requests generated at the kiosk. Furthermore, it was difficult for the educator to assist many of the patients effectively because it was hard to demonstrate how to operate the computer kiosk and its online databases over the telephone. 16 Addition of remote control assistance to the kiosk solved many of these issues. Medical librarians at the Fox Eccles Clinical Library provided excellent assistance because they were familiar with health information searches .. They were also ideal candidates because the medical librarians were already dealing with similar requests over the phone on a daily basis. Remote control assistance allowed the librarian to view and take full control of the kiosk computer, and in this way enabled them to assist patients by directly demonstrating how the searches should be done. Finally, remote control assistance allowed the kiosk to be independent of its physical setting, as the often untrained local staff were no longer needed to support the kiosk. Responses from the patient surveys indicated that remote live assistance was a feasible alternative for providing support for the kiosk. On a scale of one to five, with one being "not helpful" and five being "very helpful," patients rated remote control assistance a median score of five. Furthermore, when asked to compare the remote live assistance with other methods of receiving health information, patients indicated that they preferred remote assistance to most. The results are summarized in Table 1. On a scale of one to five, with one being "much worse than" and five being "much better than," patients gave remote control assistance median scores of four or above against all other methods. The only exception was when compared against "physician or other health providers," for which a median score of three indicated parity in the patient's preferences. Remote Assistance Evaluation Data from the patient questionnaires were used to evaluate the value of remote assistance in the computer kiosk. Table 1. Median Scores on Patient Questiolll1aires Comparing Preference for Remote Control Assistance Against Other Methods of Receiving Health Information Method Compared Against Reference librarian at a library Other sources on the Web Brochures or handouts at clinics Friends or family merrlbers Television or radio Magazine or newspaper articles Physician or other health providers Sample Size Median Score 4 4 4 5 5 4 3 Of the 400 or more patients contacted by the investigator, approximately 20% agreed to participate in the study. The no-show or cancellation rate was approximately 25%, yielding a final sample size of 64. The investigator estimated a "medium" effect size of 0.5, which translates to a 0.25 difference in the scores between the control and 17 experimental groups. A sample size of 64 was therefore sufficient to yield a power of 0.8 with alpha set at the .05 leveL Subject Demographics For the 64 subjects recruited, the mean age was 38.6 with a range of 18 to 64. It was interesting to note that all of the clinic's patient population over the age of 64 refused to participate in the experiment. For those subj ects who had prior computer experience, the mean hour per week of computer use was 27.4 with a range of 1 to 80. Seven of the subjects (10.9%) did not use a computer regularly. The median education level was II college or beyond. " 18 Patient Satisfaction Results from the questionnaires indicated that patients were generally satisfied with the kiosk. On a scale of one to five, with one being "very unsatisfied" and five being "very satisfied," the median scores for all eight aspects of the kiosk were at least four. These satisfaction scores were increased when subjects had the remote assistance option available to them, as indicated in Table 2. Scores for "ability to find information," "material seems correct," "kiosk's overall ease of use," and "kiosk compared to other methods" all received median scores of five, the highest possible. The relevancy and accuracy of the material found remained at four. Because the study had a repeated measures design, each subject participated in both the control and the experimental sessions, resulting in there being two questionnaires per subject. A difference data set was then calculated by subtracting the scores on one of the subject's questionnaires from his or her other questionnaire. A positive difference would indicate that the subject rated his or her experience in the experimental session higher than in the control session. A difference score of zero would mean that the subject's experiences during both sessions were similar. Table 2. Median Score Comparisons on Patient Questionnaires Between the Control and the Experimental Sessions Ability to find J.J.J...L'V.LJ.JL .......... "H.JJ.J. Material found was relevant Material found in enough detail Material easy to understand Material seems correct Overall rating of kiosk Kiosk's overall ease of use Kiosk compared to other methods No Assistance 4 4 4 4 4.5 4 4 4 With Remote Assistance 5 4 4 4 5 4 5 5 19 Wilcoxon Signed Rank tests were then performed on this difference data set against the hypothesis that the median of the difference was zero. If this hypothesis could not be rejected, then the subject's scores on the two questionnaires were not different on a statistically significant level. On the other hand, a rejected hypothesis would indicate that subjects had rated their experiences differently. Results of the Wilcoxon Signed Rank tests are listed in Table 3. Demographic Correlations Each of the three demographics variables were then tested for correlation against the survey data using Pearson correlation coefficients. Analysis of the data indicated that age was positively correlated with preference for using remote assistance (p 0.046). Hours of weekly computer use, however, was negatively correlated with the kiosk's perceived ease of use (p = 0.036) Usage Order Effects on Time Use Results from a paired-sample t test indicated the average time spent on the kiosk to be not statistically different between the control and experimental sessions (p = 0.179). Table 3. Results of the Wilcoxon Signed Rank Tests on the Difference Data Question Kiosk compared to other methods Kiosk's overall ease of use Ability to find information Material found in enough detail Material seems correct Overall rating of kiosk Material found was relevant Material easy to understand Mean Difference 0.27 0.32 0.33 0.30 0.17 0.22 0.15 0.12 P-Value 0.015 0.032 0.058 0.063 0.113 0.134 0.394 0.481 20 For the control, subjects spent an average of lOA minutes per session. When they had the remote assistance option available, average time was 11.6 minutes. A chi-square test found no statistically significant association between time spent per session and the order in which the subjects did the experiment (p = 0.329). Usage order and final kiosk preference were also found to be unassociated by chi-square testing (p = 0.072). Results for the chi-square tests are listed in Tables 4-5. Document Retrieval Rate and Evaluation Analysis of the retrieved documents was incomplete due to the fact that a portion of the document data was lost and therefore not included. Based on this partial data set, an analysis of the session logs showed that the document retrieval rate was 56.7% in the control sessions and 86.7% in the experimental. Data obtained from the document ratings were analyzed using a Mann-Whitney test. Documents found in the control and experimental sessions were not found to be statistically different in terms of their Table 4. Chi-square Results of Usage Order vs. Kiosk Preference Order of Use Experimental First Control First Sum Prefer Control 4 8 12 No Preference 3 8 11 Prefer Experimental 23 15 38 Table 5. Chi-square Results of Usage Order vs. Time Spent Per Session Order of Use Experimental First Control First Longer Control 6 11 No Difference 7 4 11 Longer Experimental 16 16 32 Sum 30 31 Sum 29 31 accuracy or relevancy to the subjects' searches (p > 0.50). The reading level, however, was found to be higher in documents retrieved with remote assistance (p < 0.001). 21 DISCUSSION Results from the surveys showed that patients felt comfortable and were generally satisfied with using the kiosk. Issues identified during the pilot period shaped the final design of the kiosk. The need for improvement led to the implementation of remote live assistance, which became a feasible support feature that was accepted by both the patients and staff. Further analysis of the data showed that remote assistance improved patient satisfaction, increased ease of use, and raised document retrieval rate. The implications of these findings are significant. Implications for Patients Analysis of the data showed that patients were generally satisfied with the kiosk technology as a health information tool. In addition, they found that remote control assistance made the kiosks easier to use and increased their ability to find information. This was reflected in the high scores on the patient questionnaires as well as in the improved document retrieval rate. Although the accuracy and relevancy of materials found with remote control assistance did not increase, a median score of four on a five point Likert scale indicated that patients were still satisfied with what they found. Freeform comments, as well as informal exit interviews with patients, revealed reasons for the kiosk's success. First, patients liked the convenience that remote control assistance offered, as it allowed direct access to the librarian's search skills without physically going to the library. Second, patients were able to find information that was 23 individually tailored to their needs. As compared to patients doing there own searches, it also took the librarians less time to locate the appropriate information since they were already trained for the situation and were familiar with the available databases. Third, the patients improved their own skills by observing how the librarians did the searches. This learning effect could have long-term impact on the patient education goal as the patients were being empowered to become independent information finders. Fourth, and perhaps most importantly, having the kiosk in the clinic exposed the patients to health information and encouraged them to become active seekers of their own education. Even though many would not visit a medical library , many were willing to try the kiosk as they waited in the clinic. Having health information accessible at a convenient location could potentially produce a more informed patient population that is better able to make decisions in vital health care issues. This could lead to tremendous savings in the broader healthcare goal, as prevention and chronic management becomes more efficient and powerful. Placing the kiosks in the clinics had the added benefit of increasing patient satisfaction because the perceived waiting time was shorter for those busy using the kiosk. Care must be taken, however, to assure the patients that they would not be forgotten by the nursing staff while on the kiosk, as that could be a source of anxiety. Implications for Librarians There were also important ramifications for the librarians. Like the patients, the medical librarians delivering the remote control assistance found the kiosk to be a useful tool. An analysis of free form comments and a formal exit interview indicated overall satisfaction. Features that the librarians found useful included the convenience provided by remote assistance, as well as the ability to teach patients through direct demonstration. 24 They were also pleased with the opportunity for direct patient contact and felt that giving remote assistance through the kiosk complemented their role as an information gateway in a hospital setting. It is worth mentioning that the level of service was improved without a concomitant increase in time spent; the patients did not spend more time on the kiosks when using remote control assistance, and the librarians indicated that the time spent on servicing kiosk users was comparable to helping traditional telephone users. Suggestions for improvement included adding more document databases, as well as possibly increasing the number of library staff available to answer telephones should the demand for remote control assistance increase. Implications for Healthcare Providers The application of information kiosks and remote control assistance could have both positive and negative effects for the healthcare providers. Patients may be more inquisitive about their conditions if they have already done preliminary research on the computer kiosk. This could in turn translate into increased time spent per patient and a resulting decrease in productivity. In cases where the information brought in is incomplete or inaccurate, the extra time spent on the patient might simply be wasted on separating the good information from the bad. Giving physicians, or a trusted party, control over the kiosk's contents can avoid these situations, as physicians would already approve of and be familiar with the information. For the nursing providers, the computer kiosk can be used to address answers to frequently asked medical, self-care, or procedural questions. As compared to more traditional patient handouts, information contained on the kiosks is easier to update and distribute. For instance, patients can potentially access the information from their homes through the Internet. Having computerized versions of the patient handouts also allows patients to browse and search through the available material using keywords. For the nursing provider, the kiosk can become a powerful addition to their arsenal of patient education resources. Implications for System Developers 25 For the system developer, the kiosk built in this project can serve as a basis for future models, particularly if technological improvements are considered. For instance, the benefits of voice integration, using a voice over IP protocol, can be examined for advantages over a traditional telephone line when used in the kiosk. Furthermore, using remote control to manage the kiosk on an administrative level may be advantageous. Maintaining the software can be performed remotely and will become an important issue should multiple kiosks be deployed in geographically separated regions. Future studies should consider putting patient surveys online. This will facilitate data collection as well as ensure completeness, for patients can be systematically prompted to fill in missing information. With onscreen surveys, patients can also rate their own documents. Answers to subjective measures, such as the document's relevancy to the patient's search goals, can be more accurately established through this method. Problems Despite being a helpful resource for most patients, the computer kiosk and its remote assistance feature were frustrating for some. One subject, who was apparently daunted by the kiosk interface, stormed out of the experiment session. The patient had given up after a brief period of initial failure and refused to participate any further in the experiment. Unfortunately, the individual was too upset to explain what difficulties were 26 encountered, and a viewing of the session logs did not reveal any clues. Another incident involved a subject who did not know what a hyperlink was. Therefore, when the search results came back in the form of several links, the patient did not know what to do and waited in front of the computer for approximately 10 minutes. Even though the use of hyperlinks was demonstrated in the scripted tutorial given to all the subjects, it may not be sufficient for some people. The investigator later explained how the hyperlinks worked and the experiment proceeded. The most common technical problems encountered during the experiment sessions involved getting the network printer to print. On one occasion, PCAnywhere@ froze, but the situation was quickly resolved by restarting the computer. Other than these few isolated incidents, the computer kiosk ran well during the experiment period. Privacy A surprising obstacle was the number of potential subjects who refused to participate due to privacy issues. Several expressed concerns that their names would be taken down and their search queries recorded. One person feared that the government would abstract details of his health condition from the searches he would do during the experiment. He was afraid that future employers or insurance companies could then use this information against him once it is stored into a database. This individual could not be persuaded to participate, despite repeated assurances from the investigator that all information was kept confidential. This privacy issue should be kept in mind in future development and evaluation endeavors. 27 Computer Experience It was noted that people with less computer experience reacted more favorably towards using the remote control assistance (the help feature). It was therefore not surprisingly that age, which is commonly associated with less computer experience, was positively correlated with preference for the remote control assistance. The feeling that it would be especially helpful for those less skilled at computers were echoed by the librarians in their exit interviews. It also was not surprising that hours per week of computer use were negatively correlated with perceived ease of use. This could be explained by the fact that the kiosk interface had to be simplified, at the cost of functionality, to make it more attractive to those with less experience. Computer literate individuals also tend to be more familiar with computerized information searches and would therefore find the remote control assistance to be less needed. Remote Control Assistance vs. Reference Librarian Patients rated the remote control assistance higher as a method of getting health information than to even meeting reference librarians at a library. Since a medical librarian operates the help-giver end of the remote assistance, it is surprising that patients had such preference. It is conceivable that they preferred the computer kiosk because the anonymity made it easier to ask sensitive medical questions. Another possibility is that patients perceive going to the library as an extra task and liked the convenient access to the librarians through the kiosk. 28 Evaluation of Retrieved Documents Although data from the patient surveys showed the patients' satisfaction with the kiosk, results from the document ratings were rather disappointing. It was hoped that there would be a clear improvement in document accuracy, as well as relevancy, when using remote control assistance. The higher reading level could be explained by the fact that patients often used the remote control assistance after they had exhausted their own means at finding infonnation. Since StatRef was the database that was the most complex but had the most detailed infonnation, the librarians frequently used it when giving remote assistance. As StatRef was a collection of electronic medical texts, it was understandable why the reading level was higher. Limitations This study was limited in its analysis by the incomplete data set used for the document ratings. A relatively small sample size also limited the extent of quantitative analysis. Furthennore, the sample used in this study might not be representative of the population in general. For instance, none of the potential subjects over the age of 64 were included in the study, although they were invited to participate. Perhaps they felt intimidated by computers and did not trust new technology enough to participate. The median education level for the 64 subjects who did participate was also unusually high, at "college and beyond," suggesting a possible selection bias. This demographic finding is significant considering that a good portion of the clinic's patient population comes from low-income families. CONCLUSIONS The project demonstrated that patients are willing to use a touchscreen computer kiosk to access health information in a clinic setting. Furthermore, it showed that remote control assistance is an effective and useful tool to provide search assistance for the kiosk users. Results from patient surveys indicated that remote assistance improved patient satisfaction, increased ease of use, and raised document retrieval rate. Both the patients and the help-givers found the application useful, and the technology has important ramifications for the patients, librarians, healthcare providers, and system developers. As with most research, this study did not answer all the possible questions that could be raised. For instance, it would be interesting for future studies to examine whether the level of the help-giver's expertise influences a patient's experience with using the kiosk. Additional studies could look at a broader demographic range of subjects, especially those with less education. Another interesting experiment would be to deploy multiple kiosks equipped with remote control assistance and see how the increased workload impacts the efficiency of the library's services. The ultimate test would be to see if there is any long-term impact for patients who have experienced remote control assistance. Many of the subjects expressed interest in the technology and inquired about having similar access to remote control assistance from their personal computers at home. Perhaps another study can look at the feasibility of using remote control assistance from private residences. 30 A task is always more efficient when perfoffi1ed by an individual specifically trained for the task. In the realm of locating health information, a medical librarian is an expert. Remote assistance technologies help patients to conveniently tap into the skills of the librarian in real time. Perhaps in the future, a full remote assistance service would be available where the patient only needs to ask the question at the kiosk. The query would then be executed remotely by the librarian and the results printed at the patient's kiosk. Further studies are needed to determine the feasibility and usefulness of such a system. APPENDIX A PILOT KIOSK QUESTIONNAIRE Thank you for helping with this study. Please take a minute to complete this survey about the Patient Learning Center. 1. Please indicate on a scale of 1 to 5 with 1 being very dissatisfied to 5 being very satisfied, how satisfied you are with various aspects of the Patient Learning Center. Aspects of the Patient Learning Center Very Very Dissatisfied Satisfied a. Ability to find desired information 1 2 3 4 5 b. Ease of use 1 2 3 4 5 c. The accuracy of the content 1 2 3 4 5 d. Ability to navigate through the site 1 2 3 4 5 e. Touch screen monitor 1 2 3 4 5 2. Compared with other methods of receiving patient education information, the Patient Learning Center is: much better than other ways somewhat better than other ways about the same as other ways worse than other ways much worse than other ways 3. Did you use the "Live Help" function? Yes No If yes, how helpful was this function to you on a scale of 1 to 5 with 1 being not helpful to 5 being very helpful? Not Helpful 1 2 3 4 Very Helpful 5 4. With 1 being much worse and 5 being much better, how would you compare the Live Help feature against the following methods of getting help from someone: 32 Other methods of getting help Much Much Worse Better a. Live librarian (face to face) 1 2 3 4 5 b. Live librarian on a telephone 1 2 3 4 5 c. Live librarian via a chat box * 1 2 3 4 5 d. Online text or help manual 1 2 3 4 5 e. Paper text or help manual 1 2 3 4 5 * chat box = a way for you to ask the librarian questions through messages typed into the computer. The librarian's response is displayed as text on your screen. 5. Please let us know a little bit about you. a. Gender: (Please write "M" for male and "F" for female) b. What was your age at your last birthday? c. How many years of schooling have you completed? (For example, if you have completed high school, write 12) d. How many hours on average do you spend using a computer per week? (Please write 110" if you don't use one.) e. Do you have access to the World Wide Web (Internet)? (Either at home, school, work, or other places) f. Do you currently use e-mail? Please indicate yes or no. 6. Other comments: Thank you for your time! APPENDIXB KIOSK SCREENSHOTS Starting Screen Patiellt E(iucatioll fOI~ University of Utah Health Sciences Center Basic Search Screen Basic Search: Micromedex CareNotes Instructions: Please enter the ke)7'INord(s) of the disease or drug that you want infonnation on. Hit the Search key when you are finished. 34 Advanced Search Screen Advanced Search: Please choose one of the fonowing SIAl' INTRANET P_edby: mON SERVER'~ Online access to 31 titles including: Basic & Clinical Pharmacology, The Merck Manual of Diagnosis & Therapy, Taylor's Family Medicine, Geriatric Medicine, Medical Immunology, Williams Obstetrics, Rudolph's Pediatrics, Review of General Psychiatry, DSM N. Ellenhom's Medical Toxicology, and Smith's General Urology. MedLine Plus A pre-selected list of authoritative sources ofhealth infonnation on the web. Compiled by the National Library of Medicine, MedLine Plus is the consumer health version of Me dLine , the database ofjoumal articles that is used by health professionals and researchers. Dictionay of medical tenus, useful for looking up quick infonnatlon on difficult medical terminology. Includes brief explanation of common and uncommon words used in medicine. Compiled by the people at University Hospital, Office of Patient Education Browse through patient education handouts and pamphlets by category. AJ1 35 Sample StatRef® Search AHFS Drug Information 2000 Mosby's GenRx 10th Ed. (2000) USP 01 Vol. 1- Drug Info. forthe Health Care Pro., 20th Ed. (2000) USP 01 Vol. 11-Advice for the Patient 20th Ed. (2000) . The Medical Letter (1988-2000) The Handbook of Adverse Drug Interactions (2000) Ellenhorn's Medical Toxicology, 2nd Ed. (1997) Stein Internal Medicine, 4th Ed. (1994) Family Medicine, 5th Ed. (1998) Griffith's 5 Minute Clinical Consult 8th Ed. (2000) Geriatric Medicine, 3rd Ed. Precision: I OR ~,A.t least one searc;h wordrnust (Jccur II Enter word proximity only tOt Hear "X" search i 36 Sample Medline Plus Search FDA Approves Once-a-Week Prozac Associated Press By LAURAN NEERGAARD AP Medical Writer Tuesday, February 27, 2001 WASHINGTON (AP) - The government approved a once-a-week version of Prozac Tuesday for long-term depression treatment - but cautioned it's too soon to know if weekly doses are as effective as once-a-day Prozac, Depression often requires long-term treatment. Yet many patients quit medication as soon as they feel better, prompting a relapse. So some psychiatrists have longed for once-a-week medication. theorizing patients would be more likely to keep taking their medicine if it didnl mean swallowing so many pills. 37 Sample MedTenns.com Search Procedures & Tests I Medications I MedTerms Dictio MemberAdvantage I SmartMed Sites dioineNet Home;. Medto'!rms Home> Dictional)' > SSRI mmaD!a •• Search I : (I Advanced Surch I Help I Privacy I About Us .A.II",rgies Arthritis Asthma Cholesterol Diabetes: Digestion HIV Thvroid SSRI: Abbreviation for selective serotonin reuptake inhibitors, commonly prescribed drugs for treating depression. SSRIs affect the chemicals that nerves in the brain use to send messages to one another. These chemical messengers, called neurotransmitters, are released by one nerve and taken up by other nerves. Neurotransmitters that are not taken up by other nerves are taken up by the same nerves that released them. This process is termed ~I!II= ••• "reuptake." SSRIs work by inhibiting the reuptake of serotonin, an action iii which allows more serotorun to be available to be taken up by other nerves. 38 Sample Search Using Office of Patient Education Materials Advanced Search: Univ. of Utah Patient Education Materials Click a Category or use the Search box above U Bum Center U Cancer/B:MTlRematology U Cardiology/Cardiovascular SurgerylPreventive Cardiology U Community Resources U Diabetes U Diagnostic TestslProcedurelEguipment U Discharge U Genitourinary System II Home Care U Infectious DiseaseN accmes/AIDSIlITV U Male Reproductiv"e SystemIHealth Issues U MedicationfM:edication Instructions U N eurofMusculo/SkeletallOrthopedic U NutritionlGI u OB/GYNlInfertilitylWomen's Health U Ophthalmology U Pain u Parenting/Child CarelPediatrics U Patient RightslPatient Information U Poison Control U Preventive U PsychosociallBereavement U Rehabilitation U Respiratory U Skin CarerN ound Care U Surgery U Transplant 39 Remote Assistance Request Screen An online librarian will help you with using the Patient Learning Center. The online librarian will guide you through your search for the health information you·re looking for. Please touch here if you wouIdJike the online librarian to help you search forinfonnation usingl'elDote control assistance. Remote control assistance is prov:icl!d through Timbuktu Pro. During remote control assistance, a clinica1librarianwill tw control of the computer that you are using, after asking for your pennission. The librarian will then guide you through your search. It is a teclmology that has traditionally been used fortechnlc81 support at computer help desks. This study will test the feasibility of applying remote control assistance to improve patient education semes. 40 APPENDIXC OUTLINE OF KIOSK TUTORIAL 1. Demonstrate how to type in a search term using "Prozac" as an example 2. Demonstrate the two methods that can be used to execute a query: a. By pressing the "Enter" key on the keyboard b. By pressing the "Search" button onscreen 3. Demonstrate how hyperlinks work by following a few links 4. Demonstrate how the "Up" and "Down" scroll buttons work 5. Demonstrate how the "Page up" and "Page down" buttons work 6. Demonstrate how the "Back" and "Forward" buttons work 7. Demonstrate how to print a document by pressing the "Print" button 8. Demonstrate the three methods that can be used to start a new search: a. By pressing the University Hospital logo b. By pressing the "New Search" button c. By pressing the "Basic Search" button 9. Demonstrate how to use the "Advanced Search" features 10. Demonstrate how to request remote assistance by hitting the "Live Help" button APPENDIXD CONTROL SESSION QUESTIONNAIRE Subject #: Session: A or B 1. Please indicate how satisfied you are with the following aspects of the Patient Learning Kiosk, with 1 being very dissatisfied to 5 being very satisfied. This survey is only for your first session with the Learning Kiosk. Very Very Dissatisfied Satisfied a. Ability to find information on the topic 1 2 3 4 5 you are interested in b. The material found is relevant 2 3 4 5 c. The material is in enough detail 2 3 4 5 d. The material is easy to understand 2 3 4 5 e. The material seems correct and accurate 2 3 4 5 f. Overall, how would you rate the material 2 3 4 5 you found g. Overall, how would you rate the 2 3 4 5 Learning Kiosk's ease of use? 2. Compared with other methods of receiving patient education information, the Patient Learning Center is: much better than other ways somewhat better than other ways about the same as other ways worse than other ways much worse than other ways 3. Please let us know a little bit about you. Male: Female: Age: ___ _ How many hours per week do you use a computer either at home, work, or school? ___ Average number of hours per week. Or, __ I do not use a computer. What is the highest education level that you have completed? Elementary School _ Junior High _ High School College or beyond Thank you for your time! APPENDIXE EXPERIMENTAL SESSION QUESTIONNAIRE Subject #: Session: A or B 1. Please indicate how satisfied you are with the following aspects of the Patient Learning Kiosk, with I being very dissatisfied to 5 being very satisfied. This survey is only for your second session with the Learning Kiosk. a. Ability to find information on the topic you are interested in b. The material found is relevant c. The material is in enough detail d. The material is easy to understand e. The material seems correct and accurate f. Overall, how would you rate the material you found g. Overall, how would you rate the Learning Kiosk's ease of use? Very Dissatisfied 1 2 2 2 2 2 2 2 3 3 3 3 3 3 3 4 4 4 4 4 4 4 2. Compared with other methods of receiving patient education information, the Patient Learning Center is: much better than other ways somewhat better than other ways about the same as other ways worse than other ways much worse than other ways 3. Did you use the "Live Help" function? Yes No Very Satisfied 5 5 5 5 5 5 5 IF YES, how helpful was the "Live Help" to you on a scale of I to 5 with 1 being not helpful to 5 being very helpful? Not Helpful Very Helpful 1 2 3 4 5 4. With 1 being much worse and 5 being much better, how would you compare the Live Help feature against the following methods of getting help from someone: Other methods of getting help Much About Worse Same a. Reference librarian at a library 1 2 3 4 b. Other sources on the Web 1 2 3 4 c. Brochures or handouts at clinics I 2 3 4 d. Friends or family members 1 2 3 4 e. Television or radio 1 2 3 4 f. Magazine or newspaper articles 1 2 3 4 g. Physician or other health providers 1 2 3 4 Thank you for your time! Much Better 5 5 5 5 5 5 5 APPENDIXF DOCUMENT RATING FORMS Question: -------------------------------- The patient wanted general/detailed information. 1. In your opinion, how accurate is the information presented in the document? Not Accurate 1 2 3 4 5 Very Accurate 6 7 2. In your opinion, what grade level is the document written for? Elementary School (6th grade) Junior High (8th grade) High School (12th grade) College or beyond 3. In your opinion, how well did the document address the patient's topic, taking into consideration whether the patient wanted general or detailed information? Not Well 1 2 3 4 Very Well 5 6 7 REFERENCES 1. Tang PC, Jaworski MA, Fellencer CA et al. Clinician information activities in diverse ambulatory care practices. Proc AMIA Annu Fall Symp. 1996: 12-6. Langer EJ, Abelson RP. The psychology of controL Beverly Hills, CA: Sage Publications, 1983. 3. Stone GC, Cohen F, Adler NE. Health psychology: a handbook: theories, applications, and challenges of a psychological approach to the health care system. San Francisco, CA: Jossey-Bass Publishers, 1979. 4. Tang PC, Newcomb C. Informing patients: a guide for providing patient health information. J Am Med Inform Assoc, 1998; 5(6): 563-70. 5. Tang, PC, Newcomb C, Gorden S et aL Meeting the information needs of patients: results from a patient focus group. Proc AMIA Annu Fall Symp. 1997: 672-6. 6. McClung HJ, Murray RD, Heitlinger LA. The Internet as a source for current patient information. Pediatrics, 1998; 101(6): E2. 7. Hoch DB, Norris D, Lester JE et al. Information exchange in an epilepsy forum on the World Wide Web. Seizuer, 1999; 8(1): 30-4. 8. Scheitel SM, Boland BJ, Wollan PC et al. Patient-physician agreement about medical diagnoses and cardiovascular risk factors in the ambulatory general medical examination. Mayo Clin Proc, 1996; 71 (12): 1131-7. 9. Lewis D. Computer-based approaches to patient education: a review of the literature. JAmMed Inform Assoc, 1999; 6(4):272-82. 10. Cain NJ, Fuller HJ. The hospital library online--a point of service for consumers and hospital staff: a case study. Bull Med Libr Assoc, 1999; 87(3): 261-3. 11. McCray, JC. Delivering health information statewide via the Internet in a collaborative environment: impact on individual member institutions. Bull Med Libr Assoc, 1999; 87(3): 264-9. 12. Voge S. NOAH--New York Online Access to Health: library collaboration for bilingual consumer health infonnation on the Internet. Bull Med Libr Assoc, 1998; 86(3): 326-34. 46 13 . Westman JH, Hampel H, Bradley T. Efficacy of a touchscreen computer based family cancer history questionnaire and subsequent cancer risk assessment. J Med Genet, 2000; 37(5): 354-60. 14. Sechrest RC, Henry DJ. Computer-based patient education: observations on effective communication in the clinical setting. J Biocommun, 1996; 23(1): 8-12. 15. Pearson J, Jones R, Cawsey A. The accessibility of information systems for patients: use of touchscreen infonnation systems by 345 patients with cancer in Scotland. Proc AMIA Symp. 1999: 594-8. 16. Lewis D, Nath C. Feasibility of a kiosk-based patient education system in a busy outpatient clinic setting. Diabetes Educ, 1997; 23(5): 577-81, 585-6. 17. A friendly 'ghost' helps users. Communications News, 2000; 37(8): 56. 18. Holland MP. Collaborative Technologies in Inter-University Instruction. J Anl Soc InfSci, 1996; 47(11): 857-862. 19. Tucker G. A dynamic distance learning program. Media and Methods, 1998; 34(3): 11-16. 20. Roxy.com and everything wireless are now Roxy Satellite & Wireless. Roxy.com Web site. July 1, 2000. Available at: http://www.roxy.com!. Accessed July 1, 2000. 21. Road Runner help desk launch page. Road Runner Web site. Dec 2000. Available at: http://help.rr.com/html/rrwho.html. Accessed Dec 14,2000. 22. Netkey Inc. infrastructure for kiosk solutions. Lexitech.com Web site. Oct 1999. Available at: http://www.lexitech.com!. Accessed Oct 1999. 23. Girden ER. ANOVA : repeated measures. Sage university papers. Quantitative applications in the social sciences; no. 07-084. 1992, Newbury Park, Calif.: Sage Publications. vi, 77. 24. Shott S. Statistics for health professionals. Philadelphia, PA: Saunders, 1990. 25. Lund A. Design Book. 1997. Available at: http://nuinfo.nwu.edulmscs/projects/610e15.html. Accessed Dec 1999.