Improving the Rural Provider's Ability to Manage Complex Diseases: Implementing and Facilitating the Use of Project ECHO

Purpose: To discover the challenges and potential solution(s) for the rural primary care providers using project ECHO (Extension for Community Healthcare Outcomes) to better manage common complex diseases. Rationale/Background: Research has demonstrated that those living in rural areas have higher mortality rates when compared to those living in metropolitan areas. The Project ECHO model was developed and designed to educate and enable rural primary care providers with the ability to manage common complex diseases and could be a potential solution. Project ECHO uses teleconferencing technology to connect an interdisciplinary team of experts (the "hub") with primary care providers to discuss patient conditions. Therefore, creating a positive online learning experience is important to engage providers in Project ECHO. Project ECHO is now nationwide and includes a hub in Utah; however, the Project ECHO model has not been fully implemented in Utah rural areas. Methods: Project ECHO was implemented in a clinic located in rural Cache Valley Utah. The E-Facilitation Model for creating positive online learning environment was used to guide and assess the implementation of Project ECHO. This E-Facilitation Model theorizes that three main components (preparation, engagement and ensuring value) must exist for meaningful online engagement/learning to occur. Data was collected using semi-structured interviews using the E-Facilitation model as a guide. The participants answers were then analyzed using a combination of direct and indirect content analysis. Outcome: Four providers including 2 DNPs, 1 PA, and 1 Physician agreed to participate in the project. Pre-implementation: Interview results revealed that only one of the providers had previous knowledge and had used Project ECHO. All four Participants responded positively they felt confident they could logon onto a Project ECHO session and have a reliable computer with reliable internet connection. The provider who has participated in Project ECHO also expressed that the ECHO sessions is engaging, relevant, and valuable. Lacking time was listed as the main barrier or suspected barrier to participating in Project ECHO. In addition, two providers expressed some resistance to participating on a live stream and felt uncomfortable being seen by others online. Implementation: All four providers successfully completed the online learning module introducing them to Project ECHO, how to register for it, and how to sign on for a session. Post Implementation: Despite providers reporting an overall perception of Project ECHO after the learning module none logged into a Project ECHO session. All reported time barriers and lack of clinic support as the main barriers to attending a Project ECHO session. Conclusion: Despite an overall positive perception of Project ECHO following implementation, none participated, citing lack of time and lack of clinic support as their main barriers. The provider who previously has used Project ECHO demonstrated strong motivational factors to helping underserved population, was the least experienced and was also introduced to Project ECHO in school. Possible solutions to help implement Project ECHO include: increasing clinic/employers support for the service, introduce Project ECHO to new graduates/students and although motivation may be a key factor to using Project ECHO, more research is needed to fully understand the correlation.

Running head: FACILITATING PROJECT ECHO Improving the Rural Provider's ability to Manage Complex Diseases: Implementing and Facilitating the Use of Project ECHO Deven Willardson Project Chair: Jia-Wen Guo Content Experts: Christina Choate, Sue-Chase Cantarini, and Rebecca Wilson University of Utah In Partial Fulfillment of the Requirements for the Doctor of Nursing Practice 1 FACILITATING PROJECT ECHO 2 Abstract Purpose: To discover the challenges and potential solution(s) for the rural primary care providers using project ECHO (Extension for Community Healthcare Outcomes) to better manage common complex diseases. Rationale/Background: Research has demonstrated that those living in rural areas have higher mortality rates when compared to those living in metropolitan areas. The Project ECHO model was developed and designed to educate and enable rural primary care providers with the ability to manage common complex diseases and could be a potential solution. Project ECHO uses teleconferencing technology to connect an interdisciplinary team of experts (the "hub") with primary care providers to discuss patient conditions. Therefore, creating a positive online learning experience is important to engage providers in Project ECHO. Project ECHO is now nationwide and includes a hub in Utah; however, the Project ECHO model has not been fully implemented in Utah rural areas. Methods: Project ECHO was implemented in a clinic located in rural Cache Valley Utah. The EFacilitation Model for creating positive online learning environment was used to guide and assess the implementation of Project ECHO. This E-Facilitation Model theorizes that three main components (preparation, engagement and ensuring value) must exist for meaningful online engagement/learning to occur. Data was collected using semi-structured interviews using the EFacilitation model as a guide. The participants answers were then analyzed using a combination of direct and indirect content analysis. Outcome: Four providers including 2 DNPs, 1 PA, and 1 Physician agreed to participate in the project. Pre-implementation: Interview results revealed that only one of the providers had previous knowledge and had used Project ECHO. All four Participants responded positively they felt confident they could logon onto a Project ECHO session and have a reliable computer with reliable internet connection. The provider who has participated in Project ECHO also expressed that the ECHO sessions is engaging, relevant, and valuable. Lacking time was listed as the main barrier or suspected barrier to participating in Project ECHO. In addition, two providers expressed some resistance to participating on a live stream and felt uncomfortable being seen by others online. Implementation: All four providers successfully completed the online learning module introducing them to Project ECHO, how to register for it, and how to sign on for a session. Post Implementation: Despite providers reporting an overall perception of Project ECHO after the learning module none logged into a Project ECHO session. All reported time barriers and lack of clinic support as the main barriers to attending a Project ECHO session. Conclusion: Despite an overall positive perception of Project ECHO following implementation, none participated, citing lack of time and lack of clinic support as their main barriers. The provider who previously has used Project ECHO demonstrated strong motivational factors to helping underserved population, was the least experienced and was also introduced to Project ECHO in school. Possible solutions to help implement Project ECHO include: increasing clinic/employers support for the service, introduce Project ECHO to new graduates/students and although motivation may be a key factor to using Project ECHO, more research is needed to fully understand the correlation. FACILITATING PROJECT ECHO 3 Introduction The field of medicine is constantly changing as the wealth of medical knowledge continues to grow. New discoveries and best practices are continually developing. One of the biggest challenges providers face is maintaining best practice. Evidence has shown that people are not always receiving the highest level of evidenced based care. In fact, it has been estimated that "On average, Americans receive appropriate, evidence-based care when they need it only 55% of the time" (Arora, et al., 2017, p. 30). This suggests that medical practice isn't keeping up with our most current medical knowledge (Arora, et al, 2017). While this evidenced-based gap has a profound effect on most individuals. Evidence suggests that individuals who live in rural areas experience even worse outcomes. Researchers using statistics from the Centers for Disease Control (CDC) identified the top five causes of mortality in the United States. These five causes were attributed to 62% of all deaths and included: heart disease, cancer, chronic lower respiratory disease, cerebrovascular diseases (stroke), and unintentional injuries (Phillips & Barclay, 2017). In addition they found that those who lived in nonmetropolitan (rural) areas experienced significantly higher mortality rates for the five conditions mentioned above, than those in metropolitan (urban) areas (Phillips & Barclay, 2017). For example, the investigators found that the rate of mortality for potentially preventable heart disease in rural areas was 42.6%, while in urban areas the rate was much lower at 27.8%. Similar comparisons were also reported for cancer and lower respiratory disease (Phillips & Barclay, 2017). Several studies have been completed that demonstrate the health disparities experienced by rural populations. Hashibe, et al. (2018) compared the five-year survival rate as well as the risk of death and number of cancer diagnosis in the state of Utah, comparing rural to urban populations throughout the state. They found that the rate of cancer was slightly decreased in the FACILITATING PROJECT ECHO 4 rural population. However, those who lived in rural areas had a 5.2% lower five-year survival rate as well as a 10% increased risk of death (Hashibe, et al., 2018). While another study found that those who live in rural areas experience higher rate of COPD when compared to those in urban communities (Croft, et al., 2018). Those living in rural areas also had a higher rate of hospitalization as well as nearly twice the mortality rate when compared to those living in urban areas (Croft, et al., 2018). The cause of this discrepancy in health is multi-factorial. Researchers have identified two potential causes as lack of access to specialists in rural areas (Charlton, Schlitchting, Chioreso, Ward & Vikas, 2015) and the gap in knowledge among primary care providers regarding best evidenced based practice for those suffering with chronic diseases (Arora, et al, 2017). One potential solution to help increase access to specialists in rural areas and close the healthcare knowledge gap among primary providers is the Project ECHO (Extension for Community Healthcare Outcomes). Project ECHO was first introduced in New Mexico in 2003. The Project ECHO model uses videoconferencing technology to conduct a collaborative learning environment. Project ECHO teams conduct frequent sessions that are largely case based. Providers attend these HIPPA compliant sessions and discuss patient conditions (Project ECHO, 2017). Project ECHO was first designed to help treat patients diagnosed with Hepatitis C. Most of these patients were not receiving treatment as there were a shortage of specialists to treat the entire population. Many, if not most, also lived in rural areas (Arora, et al, 2017). In fact, those who wanted to be treated for Hepatitis C often had to wait six months or more and at times drive up to 250 miles for an appointment (Arora, et al., 2011). Project ECHO has been proven to be safe and a highly effective learning model. A study published in 2011 compared the effectiveness of treating or curing Hepatitis C between the FACILITATING PROJECT ECHO 5 University of New Mexico (UNM) HCV clinic and the rural clinics using ECHO. The results demonstrated that the rural clinics using Project ECHO were just as effective as the UNM HCV clinic with patients receiving sustained viral responses at rates of 58.2% and 57.5% respectively (Arora, et al., 2011). Since Project ECHO began, it has grown to include several different specialties including diabetes, hypertension, cardiovascular, behavioral health, and more. New studies have verified that the model can be used safely and effectively in treating these various conditions. For example, the Veteran Affairs (VA) studied the outcomes of difficult to control high risk diabetic patients being treated through their Specialty Care Access Network-Extension for Community Healthcare Outcomes (SCAN-ECHO). They followed thirtynine patients being treated over fifteen months by two providers using SCAN-ECHO. The mean HbA1c improved from "10.2 ± 1.4% to 8.4 ± 1.8% (p < 0.001) over the average follow-up period of five months" (Watts, Roush, Julius & Sood, 2015, p. 221). Another Replication of the ECHO model called ECHO-AGE demonstrated benefits among those living in long term care facilities. This pilot study followed 47 residents with an average age of 82 and measured, of those 83% had a history of dementia. Most of the cases presented had increased "agitation, intrusiveness, and paranoia" (Catic, et al., 2014, p. 938). The study found that those who followed the recommendations of the ECHO team had fewer hospitalizations and sites reported a much higher clinical improvement 74% vs 20% (Catic, et al., 2014). Several studies have also shown an increase in provider knowledge measured through pre and post quizzes, through self-reflection or a combination of both in treating several chronic conditions. Among those include hypertension (Masi, et al., 2011), human immunodeficiency virus (HIV) (Wood, et al., 2016), substance abuse and pain management among American FACILITATING PROJECT ECHO 6 Indians (Katzman, Fore, et al., 2016), and most recently Autism (Mazurek, Brown, Curran, & Sohl, 2016). The Project ECHO model can educate and enable primary care providers (especially those living in rural areas) with the ability to manage common complex diseases like Hepatitis C (Arora, et al, 2017). Limited research has been conducted to determine barriers to Project ECHOs' use among non-users. Researchers Knapp and Pangarkar described the limitations providers face attending SCAN-ECHO (2015). They cited high clinical caseloads, lack of financial support or lack of reserved time to participate in the service and limited access to proper telehealth technology as the main barriers to participation (Knapp & Pangarkar, 2015). During a focus group among those who act as facilitators for an Army ECHO program they cited two main barriers that they feel contributes to lack of ECHO attendance. These barriers include "limited dedicated time" and "multiple competing priorities" (Kotzman, Galloway, et al., 2016). An ECHO model implemented to help treat substance abuse disorders cited three barriers to provider participation. They cited lack of reimbursement for attending sessions, decreased productivity and, in this case, providers being hesitant to participate in addiction treatment due to possible stigma or "practical barriers" (Komaromy, et al., 2016, p. 23). Problem The Cache Valley Community Health Center serves an underserved patient population and offers services to those who do not have insurance. Many of their patients experience several barriers to seeing any specialists due to a lack of certain specialists in the area, lack of insurance or inability to pay, long wait times, and others. Despite Project ECHO's rapid growth, some locations in Utah including Cache County, Utah, where the Cache Valley Community Health Center is located, have yet to fully adopt its use. According the Project ECHO hub, located in FACILITATING PROJECT ECHO 7 Salt Lake City, Utah, very few providers have every registered or even used this service in the Cache Valley area. Because there has been limited research on the barriers to implementation of Project ECHO, this quality improvement project will consist of implementing Project ECHO and identifying barriers and facilitators to provider engagement with the service, using the Efacilitation model created by Dzinotyiweyi (2015) as the framework. E-Facilitation Model Online learning is becoming more common. Several studies and research has been done to determine how to make online learning more meaningful and engaging. Researcher Dzinotyiweyi (2015) conducted research to determine what components were needed to create positive online learning environment. Dzinotyiweyi compiled his findings to create the EFacilitation Model (2015) see figure 1. The E-Facilitation Model is composed of three main concepts. These include Preparation, Engagement and Ensuring value. Figure 1: E-Facilitation Model Adapted From Dzinotyiweyi (2015). FACILITATING PROJECT ECHO 8 According to Dzinotyiweyi (2015), the preparation stage of the model involves all the tasks that prepare one to have a meaningful online discussion. These tasks include: ensuring that participants have adequate computer and online access, comfortable using technology, computer support available, and a content expert present to make the discussion meaningful and engaging. The engagement portion ensures that participants are all invested and engage in the discussion. This refers to the vibrancy of the discussion. It is imperative that participants feel comfortable participating and that everyone is respectful of others, including sensitivity to different cultures and diversity. Lastly, the ensuring value portion of the model must be in place to justify the existence of the online group. Participants must feel that the discussion is worth their time and that it truly adds value of some kind or another (Dzinotyiweyi, 2015). Once the three main concepts (preparation, engagement, and ensuring value) are accomplished then optimal online interaction can occur. The project will utilize the EFacilitation Model to assess for any barriers to the three components (Preparation, Engagement, and Evaluating value) as mentioned above, thus determining if Project ECHO is perceived as a positive and meaningful online learning environment (Dzinotyiweyi, 2015). Specific Aims The purpose of this project is to enable the rural provider the ability to manage common chronic and acute diseases using project ECHO. Thus, providing best evidenced based practice to rural patients, decreasing the healthcare gap experienced by those living in rural areas, and decreasing overall medical costs to the patient. Methods Context The Cache Valley Community Health Center (CVCH) is located in Cache County, Utah. It, along with their sister clinic Bear Lake Community Health Center (BLCHC), extends from FACILITATING PROJECT ECHO 9 Northern Utah to Southern Idaho and even Western Wyoming. The clinic(s) follows a medical home model and offer a variety of services. The clinic offers medical, dental, OB/GYN, mental health and has onsite pharmacy. The clinics within the CVCH were elected to be involved with this pilot study. The CVCH sees a very diverse patient population including insured, uninsured, several different ethnicities, races, varying financial capabilities and a large percentage of patients who would be considered low socioeconomic status. Two of the clinics were involved in the study. Both clinics were located in Cache County, Utah. There were eight total eligible participants but only four of them accepted the invitation to participate. These four providers involved in the study were family practice providers. The participants included two nurse practitioners, one physician, and one physician assistant. Intervention Project ECHO was introduced and implemented to all four participants in the Cache Valley Community Health Center. An interactive online learning module was created using Microsoft Power point and distributed to each of the participants. The module was emailed to each provider and was estimated to only take 20 - 30 minutes to complete. Before the module was disturbed it was verified for accuracy and effectiveness by several content experts including a Utah Project ECHO program coordinator, nursing PhD researcher with informatics focus, DNP faculty who has an emphasis in nursing education who has done extensive research involving telemedicine and another PhD researcher who has extensive work creating online learning modules for health care professionals. During the implementation period the providers were contacted bi-weekly to determine and resolve any barriers, which also served as a reminder to use the service. Study of the Intervention FACILITATING PROJECT ECHO 10 Due to the small sample size (4) it was determined that an interview would be the best method to evaluate the effectiveness of the intervention. The participants were interviewed both pre-implementation and post-implementation using questions that were derived from the EFacilitation Model (Dzinotyiweyi, 2015). For those who were unfamiliar with or have never used Project ECHO the questions focused on the barriers and facilitators of the preparation phase in the E facilitation model, noted above (Dzinotyiweyi, 2015). Following the intervention, these participants were then asked questions related to the remaining two components, engagement and ensuring value (Dzinotyiweyi, 2015). Those who had used Project ECHO were asked questions related to all three components of the E facilitation model including preparation, engagement and ensuring value both pre and post intervention (Dzinotyiweyi, 2015). The providers responses were then examined using a combination of direct and indirect content analysis. For direct content analysis we pulled key ideas/concepts form the E-Facilitation model for both pre and post implementation and categorized the data that corresponded to these key concepts. Any data that did not correlate with the concepts of the E-Facilitation model were analyzed and categorized using indirect content analysis. These responses were then analyzed to determine if the intervention reduced/resolved the providers barriers discovered in the first interview as well as any outlying or unforeseen barriers/facilitators to engaging in Project ECHO. See Appendix A for the pre-implementation code book and Appendix C for the postimplementation code book that were developed and used during the data analysis. Measures The series of questions that were included during the interviews with the providers were developed by the author and content experts. It was initially desired to use an already established and tested survey or interview questions specific to Project ECHO. However, because this topic FACILITATING PROJECT ECHO 11 has yet to be studied among non-users and no reliable tool could be identified. A comprehensive literature review was conducted to determine barriers to continuing education in general. Questions were then drafted using the literature review along with the E-Facilitation model (Dzinotyiweyi, 2015. These questions were presented to two content experts both with experience applicable to this topic. These experts include Jia-Wen Guo who has a PhD in Nursing and expert in the field of informatics along with Susan Chase-Canterini who as a DNP with an emphasis in nursing education and has done extensive research involving telemedicine. Both content experts were involved in critiquing and editing the questions prior the interviews with the subjects involved in the project. In conjunction with the intervention (learning module), additional steps are in place to help remind providers about the benefits and feasibility of Project ECHO. After the providers register for Project ECHO they also receive email reminders about seminars they are interested in. These email reminders provided by the Project ECHO team are essential to keeping the providers engaged in the program, thereby contributing to a meaningful and lasting change. We also had weekly to bi-weekly contact with each provider to assess if they had logged on and any potential barriers they had that week to logging on for a session. To ensure the accuracy of the data gathered, the providers were interviewed by the author personally and, after receiving participants permission, each session was recorded. The author also took steps to ensure that data gathered was analyzed correctly by obtaining feedback from the content experts noted above. The outcome of the analysis was then verified through interrater reliability. Lastly, by working with the Project ECHO team in Utah, the author was able to verify the participants who registered and logged into a Project ECHO session by checking Project ECHO's records. Analysis FACILITATING PROJECT ECHO 12 Both the pre-implementation and post-implementation interviews were transcribed and analyzed using a combination of direct and indirect content analysis using the E- Facilitaiton model as the framework (Dzinotyiweyi, 2015). A code book was developed with the identified key themes. The key themes or categories were then defined. These findings were verified using inter-rater reliability by a fellow colleague to code the transcript based on the code book. Ethical Considerations The University of Utah Institutional Review Board determined this study to be exempt from human subject's review. Confidential information was not collected, and all the subjects volunteered to participate. Results The results of this project will be discussed in the following format: pre-implementation interview results, implementation period results, and post-implementation interview results. Please see Appendix A for the code book used for the pre-implementation data and Appendix B for the concept map of the data results. Pre-implementation For the initial interview or the pre-implementation interview there were three main categories or themes that emerged from the data along with several sub-categories or themes. The information in this section will be displayed in the following format. First, participant/provider demographic information followed by the main categories/themes including Preparation Facilitators, Preparation Barriers and lastly, other outlying data. Within each main category or theme the sub-themes or codes that were discovered will be revealed. Demographic Information FACILITATING PROJECT ECHO 13 There were four providers who participated in the study. Refer to table 1 to see the complete demographic information of each participant. Each participant will be referred to as provider (P) 1, 2, 3 or 4 or (P1), (P2), (P3), or (P4). (P1) is a 27-year-old, Caucasian, Female, Family Practice Nurse Practitioner and at the time had been in practice for nearly one and a half years. (P2) is a 39-year-old, Caucasian, Male, Family Practice Nurse Practitioner who at the time had been in practice over nine years. (P3) is a 40-year-old, Caucasian, Male, Family Practice Physician Assistant who at the time had been in practice over six and a half years. Lastly, (P4) is a 50 year old, Male, Family Practice Physician (DO), who at the time had been in practice over 16 years. Preparation Facilitators Tasks, activities, tools and attitudes that properly prepare the subject to participate and engage in the online discussion/lecture. Sub-themes within this category and be thought of as anything that helps facilitate, support and/or motivate the subjects to participate in Project ECHO. The sub-themes discovered include: knew/used Project ECHO prior to intervention, positive perception of project echo, access to reliable computer and internet connection, comfortable using technology, content expert present at each project echo session, and positive perception of telecommunication systems. Knew/used Project ECHO prior to intervention. From the interview seven codes or subthemes were identified as belonging to the main theme, Preparation Facilitators. Of all the four participants, (P1) was the only one who had ever heard of Project ECHO and also the only one who had any experience using it. (P1) was introduced to Project ECHO "when I was a student" and has attended "probably 3-5 sessions." FACILITATING PROJECT ECHO 14 Positive perception of Project ECHO. (P1) also had a very positive perception of Project ECHO "Project ECHO set me up with the right information so I could manage [my patients] care" and several instances mentioning in different variations that Project ECHO "was really, really helpful." Access to reliable computer and internet connection. All four providers did agree that they had access to a reliable computer and internet both at home and in clinic. Comfortable using technology. All four providers expressed that they felt comfortable using computer technology. (P1) rated herself 8 on a 0-10 scale, with 0 being not comfortable and 10 being very comfortable, but in the same sentence said that she is "not tech savvy." (P2) rated himself 7.5 on a 0-10 scale and stated that he is "pretty comfortable" using computer technology. (P3) did not rate himself on a 0-10 scale but stated that he was "relatively comfortable" but "sometimes [computer technology] "can be a real hassle." He also mentions that he prefers online leaning to that of a live conference "I just prefer to read or listen to something and get the CME that way…not for convenience, that's just my style." (P4) rated himself 6 on a 0-10 scale. He mentions "I have done CME through continuing board certification…[it's] good on my own time at my own pace. Content expert present at each Project ECHO session. (P1) felt that there was a content expert present at each session. She stated "I really like with Project ECHO how you have a whole panel of people and… you have that accessibility to specialists and people who have a whole lot more experience than me. So, I think that that is tremendously valuable." The other three providers could not comment on this section as they had not yet used Project ECHO. FACILITATING PROJECT ECHO 15 Positive perception of telecommunication systems. Three of the four providers made a total of 11 statements displaying a positive perception of telecommunication systems. (P1), (P2), (P3) all mentioned that it gives more access to specialists for those living in rural areas. They all stated that they felt telecommunication/telemedicine was going to continue to grow. (P2) and (P3) felt that telecommunication systems would be really helpful in managing "psychiatric" or "behavioral health" patients. (P1) and (P3) also had personal experience using telehealth technology in school. Preparation Barriers Tasks, activities, tools and attitudes that limit or prevent the subject to participate and engage in the online discussion/lecture. Sub-themes within thThink of this as anything that is a barrier or discourages the subjects to participate in Project ECHO. The sub-themes discovered include: lack knowledge of Project ECHO, perception of, or actual time limitations, uncomfortable with format of meeting/discussion, and negative or neutral perception of telemedicine/telecommunication. Lack knowledge of Project ECHO. It is obviously apparent that not knowing about Project ECHO or never hearing about is a barrier to using Project ECHO. Pre-intervention three of the providers (P2), (P3), and (P4) have "never heard" of Project ECHO. Perception of, or actual time limitations. Three of the four providers listed time limitations as their actual or suspected barrier to participating in Project ECHO sessions. (P1) who has attended project ECHO sessions sited this as her main barrier. "My biggest barrier is seeing patients when ECHO is scheduled, if it was a lunch hour or day that I was off I would be able to attend. Or really early in the morning…there was an ECHO about migraine and headache, that was at like 7:00 AM or something so I could log on and get FACILITATING PROJECT ECHO 16 information while I am getting my first patient ready…if the time was better I think I could attend more" This theme of time concerns and barriers is also evident as (P3) explained, (referring to attending live online education seminars) "I think a lot of it was just you know, you're so busy…like I have a patient here right now and I'm already behind, I really just don't have the time to devote to…something that is live." For other providers not only is there time limitation barriers with live seminar and education there is a convenience factor that also goes along with it. (P4) stated "I like doing things on my own time…at my own pace" but admits that there is a challenge of "finding the time to do it" referring of course to live seminars and/or education. Uncomfortable with format of meeting/discussion. Interestingly, the perceived format of the meeting like how it is designed and some of the unknowns of how the meeting is conducted. For example, (P2) stated: "I think…part of why I wanted to do this (referring to the project) was [to] get outside my box because live has not always worked out best for me. I think some of the barriers are…know[ing] when to jump in…and how to best communicate over the video." (P3) also agrees that the format/structure of the meeting greatly influences his desire to participate: "Other barriers to time would be my perception on how [the meetings] are structured…I think the format would be too…it would depend if it was more of a lecture or if it was more of an expert panel or question and answer [session], I guess it would just depend." Negative or neutral perception of telemedicine/telecommunication. As aforementioned all the providers had a very positive outlook on telecommunication systems. However, two of the providers also made comments that were more negative or neutral in nature regarding telemedicine/telecommunication. For example, (P3) stated: "Some of the disadvantages are… not [being] able to… touch something or hear something as well (referring to telemedicine), I think that sometimes creates some diagnostic errors just again because you're not able to engage all your senses to…assess somebody. FACILITATING PROJECT ECHO 17 While (P3) statement may not be directly related to Project ECHO those perceptions could be a barrier to participating in Project ECHO. (P4) made comments that were more neutral in nature towards telemedicine/telecommunication systems. He stated that he has "not had much experience with [telemedicine]" and realizes that telemedicine can be "difficult to get insurance to cover it." Having a negative or neutral perception of telemedicine or telecommunication systems can play a significant role in one's ability to try something like Project ECHO. Other Underlying Factors There were a total of five sub-themes that were discovered that did not meet the criteria to be under the preparation facilitators or preparation barrier categories. However, these findings provided some possible insight as to why a provider may or may not participate in Project ECHO sessions. The sub-themes included in this category include: frequency of contact with specialists, collaboration with specialists is important, motivated by patient population, negative perception of collaboration with specialists, and negative/neutral live online seminar experience. Frequency of contact with specialists. All the providers in this section had nearly daily collaboration with specialists. Most of the daily contact was "written communication" while those who communicated directly to a provider via telephone was reported around two times per week. Collaboration with specialists is important. It was very apparent that all the providers recognized the importance of collaboration with specialists all of them stated that this is "very important." Motivated by patient population. (P1) was the only provider who made comments that demonstrated that she was highly motivated to participate in Project ECHO to help her patients and it seems to be her primary motivating factor. On four separate statements she made it abundantly clear as stated in the following statement: FACILITATING PROJECT ECHO 18 (referring to a pt who was misdiagnosed with anxiety and had schizophrenia) "Project ECHO set me up with the right information, so I could manage his care because we really didn't have access to anywhere else for him... [and our] psychiatrist is booked up for months and months. So, it was really really helpful… that I could use that resource (Project ECHO) to figure out how I could manage his care safely." She continues that "I felt like the specialists had an understanding of people who have limited resources and limited access that's really [the] population that benefits the most with Project ECHO." She realizes that her patients have "very limited resources…so if I can collaborate effectively with specialists I save my patient…time and money and enhance their care." (P1) appears to be very motivated to attend Project ECHO in order to help her patients. Negative perception of collaboration with specialists. Two of the providers expressed frustrations when collaborating with specialists. (P1) expressed "sometimes when I talk to specialists I kind of feel stupid" but never felt that way about Project ECHO "Project ECHO wasn't like that at all." (P3) experienced frustrations contacting specialists "It can…be a really big hassle contacting them. You can spend a huge amount of time trying to contact somebody and, in the end,, still have trouble getting a hold of them." Whether or not the frustrations these providers expressed contacting/collaborating with specialists will be a factor with engaging in Project ECHO is still yet to be determined. Negative/neutral live online seminar experience. All of the providers expressed some negative or neutral experience to live online seminars. (P1) and (P4) both mentioned that "technical issues" is their biggest complaint with live online seminars that they've done in the past. (P3) was more neutral as he has never attended any live online seminars. (P2) mentions that while he likes to be able to "just listen in" he often finds the online seminars "boring" and hard to be as "involved." He also expresses, "I prefer the [seminars] in person just because of the collaboration that you have with the other providers when you are doing CME together" (P2) also felt that it was "harder to schedule time to do online training" and if he were to choose FACILITATING PROJECT ECHO 19 between live online seminar to previously recorded ones he would prefer the recorded ones as they "are almost better than the live ones…because I can pause and stop and move around if I need to." Project ECHO is a supportive learning environment. Providing a supportive learning environment helps providers to be engaged and fulfill the engagement step of the E-Facilitation Model (Dzinotyiweyi, 2015). (P1) who previously participated in Project ECHO felt that the specialists from the Project ECHO panel were "very supportive." When (P1) was asked how comfortable she was asking questions she replied: "it was always an open format and never got frowned upon for not knowing everything." She also felt that Project ECHO understood her patients and the financial struggles they have. Speaking of a previous experience contacting a specialist: "I told him because my patient doesn't have insurance and they can't afford that and he was like why are you talking to me I don't even know what to do and um project echo wasn't like that at all" She felt that the providers from Project ECHO were there to help her: "the presenters never had a problem going over time, one of the presenters sent me a whole hand book, it was really helpful. She followed up on everything and talked with me a week later about how the patient was doing and so it was really, really helpful" The providers from Project ECHO helped her to be engaged and were engaged and invested in the providers seeking help and their patients. Perception of value. This concept helps fulfill the Ensuring Value portion of the EFacilitation Model (Dzinotyiweyi, 2015). (P1) demonstrates that she does find value in Project ECHO and that there are "demonstrated benefits" (Dzinotyiweyi, 2015, pg. 159). She has recommended Project ECHO to other providers in her clinic and feels like Project ECHO is "tremendously valuable." Implementation Period FACILITATING PROJECT ECHO 20 During the two-month implementation period none of the providers participated in a live Project ECHO session. However, (P3) did register for the service and did watch some previous recordings of the didactic sessions. (P2) and (P4) didn't register and (P1) was already registered prior to the implementation. All of the providers reported watching and participating in interactive online learning model, introducing them to Project ECHO and provided positive feedback about the module. Post-Implementation (P1), (P2), and (P3) all participated in the post implementation interview while (P4) did not. There were seven key categories or themes that were identified of those two were also present in the pre-intervention intervention interview data. These two themes included: perceived or actual time barriers and motivated by patient population. The remaining five themes included: lack of clinic support, Project ECHO improves access and/or quality care, Project ECHO can/may lead to professional development, confident in ability to use Project ECHO, and Project ECHO topic must be relevant/useful for patient population. Please see Appendix C for code map used for interview data and Appendix D for the concept map of the data results. Perceived or actual time barriers. All three of the providers (P1),(P2) and (P3) cited the reason they did not attend a Project ECHO session was due to time barriers, citing this as the "biggest barrier." Even though (P4) didn't participate in the final interview he also expressed that time was the biggest barrier during our bi-weekly follow ups. (P1) gave more insight to the problem stating: "it's not so much the time but the timing" and "[fitting Project ECHO] into my clinic days is pretty hard." (P2) described similar frustrations: "I mean once your registered you can figure out the technology… pretty easily [but] it's just designating the right amount of time to do it." (P3) recognized that many of the sessions were during lunch but that its still hard to FACILITATING PROJECT ECHO 21 attend as "lunch is super unpredictable" further stating that another issue is not being reimbursed for their time stating: "People are willing to improve skills but there's a little bit of a limit to that. People don't necessarily want to lose their lunch or go in on day off. I didn't really think about that until now but that may be another little bit problematic that would be much easier to send specialist with no more reimbursement for those efforts ." All three providers felt that they would be more likely to attend if they had a designated time to do it. Lack of clinic support. Mirroring the perceived or actual time barriers is that there is a lack of clinic support for the implementation of Project ECHO. All three providers felt that if there was more clinic support that they would be more likely to attend. (P1) stated: "If I could block out that time so I could log on and do an echo session I would definitely utilize that." (P2) gave a very clear statement as to what that process may look like stating: "I think if here were set time a couple hours per a week you know for example you can attend a meeting that's not on your time off or time away that would be beneficial." (P3) expressed similar ideas and gave an example of what a previous clinic did with a similar program stating: "I worked at this… facility and they had this like a similar program through Harvard university the whole clinic they blocked out time for people to attend these kind of education sessions… they were almost more mandatory, so it was easier to attend because they blocked out that time." Clinic support seems to be a major contributor with regards to Project ECHO Engagement. Motivated by patient population. Providers that are motivated to use Project ECHO due to a specific patient or their patient population continues to be a key factor. (P1) cited some frustrations getting patients in to see specialists in a "timely manner." Referring to the frustration of waiting to get a patient in to see a gastroenterologist she states: "by the time you get them in we needed to be five steps ahead." Further expressing that Project ECHO can save the patient a FACILITATING PROJECT ECHO 22 lot of money and "running around too." She expresses that Project ECHO once had a headache and neurology Echo that was helpful. Further stating if that one was to start again she had several patients that could benefit as getting into a head ache clinic is a "miserable process" and takes "6-7 months." (P3) also provided more insight to motivation to use the service and stated: "if I were to have like a Hepatitis C patient… I think my preference at this point would still be to have them go see a gastroenterologist [but] if that were not the [possible] I think [Project ECHO] would be really useful." Project ECHO improves access and/or quality care. Even though none of the providers participated in a Project ECHO session during the implementation period they all felt that Project ECHO improves access and or quality care. (P1) who has used the service in the past recalled how it has helped her provide more "efficient care" and that she can have her "patients taken care of in a few weeks rather than waiting almost three months just to consult with a specialist." She also provided more insight and mentioned that the wait time is not the only barrier with the specialist stating: "With the community clinic in particular there are so many patients that are tremendously helped by doing [Project ECHO] sessions... its not a reality to get them to SLC to get them to see a specialist." (P2) also cited that Project ECHO is a "really neat way to collaborate for patients that may not otherwise have the same access to care." Lastly, (P3) gave a very similar response: "I think you could gain some access to specialty care especially with a patient without resources to see a specialist… which could benefit somebody immensely if [they] didn't have that [resource]. Project ECHO can/may lead to professional development. All three made statements consistent with the concept that Project ECHO can help them be a better provider, improve their skills and increase their knowledge. (P1) expressed that she feels the frequent collaboration with FACILITATING PROJECT ECHO 23 the specialists in Project ECHO is "extremely valuable." Project ECHO has also helped her counsel her in areas that she has felt "uncomfortable." (P2) expressed that by using Project ECHO there is "potential improvement of chronic care management based on individual diagnosis and then consultation with specialists." Lastly, (P3) further emphasized this concept and felt that Project ECHO could help on e "hone in on your diagnostic skills" and "allow you to be a better clinician." Confident in ability to use Project ECHO. All three providers felt confident in their ability to log on and use Project ECHO. (P1) remembers that they recently changed the technology some to log onto a Project ECHO session since she last logged on but was "confident" that she could figure it out. (P2) also stated: "the technology we can figure out." Project ECHO topic must be relevant/useful for patient population. The topic(s) that are offered by Project ECHO can be a potential motivator or barrier to engagement with the service. All the providers had topics offered by Project ECHO that were useful and relevant to their practice. Both (P1) and (P3) stated that the psych mental health was probably the most useful to their practice. (P3) thought that the sessions for Hepatitis C one could be useful but doesn't have and could be a potential barrier to engaging in Project ECHO stating: "sometimes not having a patient like [with] Hep C that… actively trying to come up with a treatment plan... is a disincentive to attend something...specific to Hepatitis C." However, later stated "honestly HEP C seems to come in spurts… we try to get them to gastroenterology but that's not always that easy especially, those without insurance, so you know being able to talk to someone would be super helpful. (P3) found that the gastroenterology and Hepatitis C are "still very much of interest." Both (P1 and P3) had an interest in cardiology which isn't currently offered by the Utah Project ECHO. While (P1) expressed that a desire for the headache and neurology ECHO would FACILITATING PROJECT ECHO 24 be offered again as she had several headache patients that could currently use the service as the are in "limbo" until they can get into a neurologist. Discussion Summary Our results demonstrate how difficult it can be to implement a new service like Project ECHO into a clinic. None of our four participants engaged or logged into a live Project ECHO session during the implementation period. However, one provider has used Project ECHO in the past and another provider did register to use the service as well as watched previous recordings of Project ECHO. There was one provider of the four who did not complete the postimplementation interview. The online learning module seemed to be affective in introducing Project ECHO as three of the providers who completed the second interview demonstrated a greater understanding of Project ECHO and an increased desire to use the service. However, the module was not able to overcome all the barriers discovered in the pre-implementation phase. Lack of time and lack of clinic/administrative support continues to be the greatest cited barrier to Project ECHO engagement. However, some outlying data including motivation to help specific patient or patient population seems to be key to engaging in Project ECHO. In addition, the topic(s) that are presented have to be applicable to current patients being seen by the provider. The overall perception of Project ECHO suggests that it meets the criteria outlined in the E-Facilitation model and its subsequent steps (Preparation, Engagement and Ensuring Value) (Dzinotyiweyi, 2015). Providers did report having access to the right technology and felt comfortable using it. They had an overall positive perception of Project ECHO, recognized that it can increase access to quality care, and lead to professional development. The provider who used the service in the past also felt that the service was supportive, engaging, prepared, relevant and FACILITATING PROJECT ECHO 25 valuable. However, some aspects of the preparation step seem to be lacking on the clinic or providers in order for them to fully engage in Project ECHO. Although these items are not specifically present on the current E-Facilitation Model (Dzinotyiweyi, 2015) it seems reasonable to include designated time, clinic/admin support, and motivation to the preparation step in order for meaningful online interaction to occur (Dzinotyiweyi, 2015). Relation to Other Evidence Limited research has been done to fully understand the barriers and facilitators implementing Project ECHO. In a study by Knapp and Pangarkar (2015) the identified three main barriers that seem to have contributed to the lack of SCAN-ECHO attendance. These three barriers include: high clinical caseloads, lack of financial support or lack of reserved time to participate in the service and limited access to proper technology. Time barriers were also a key finding to this study or more specifically the lack of scheduled time to attend Project ECHO. High clinical caseloads did not come up as a barrier in this study although one could conceive that this is also a time related barrier. Lastly, lack of access to proper technology was not a key finding in this study. All providers reported that they had access to a reliable computer and internet connection. All felt comfortable and confident in their abilities to use computer technology and access Project ECHO (Knapp & Pangarkar, 2015). Overall, time barriers were also listed as a possible barrier for an Army ECHO program (Kotzman, Galloway, et al., 2016). Facilitators for the ECHO program cited two main barriers that they feel contributes to lack of ECHO attendance. These barriers include "limited dedicated time" and "multiple competing priorities" (Kotzman, Galloway, et al., 2016). These findings echoed the findings found in our study. An ECHO model implemented to help treat substance abuse disorders cited three barriers to provider participation. These barriers include: lack of reimbursement for attending sessions, FACILITATING PROJECT ECHO 26 decreased productivity and in this case providers being hesitant to participate in addiction treatment due to possible stigma or "practical barriers" (Komaromy, et al., 2016, p. 23). Lack of reimbursement was not found to be a key finding in our study. Decreased productivity was not a key finding in our study but seems to be related to time barriers. Hesitancy to participate due to stigma was not a finding of our study (Komaromy, 2016). However, the topic(s) offered by Project ECHO did come up as both potential facilitator and perhaps barrier to attending Project ECHO.A couple of providers felt if they didn't have a patient with a condition that fit into the current Project ECHO topics that would impact weather they would attend or not. Several studies have also shown that Project ECHO is a safe and effective model that increases the quality of care and the knowledge of primary care providers. Studies cited in the introduction demonstrate that Project ECHO has been effective in improving several chronic conditions including Hepatitis C (Arora, et al., 2011), Diabetes (Watts, et al., 2015), and Dementia (Catic, et al., 2014). Other studies demonstrate an increase in provider knowledge measured through pre and post quizzes, through self-reflection or a combination of both in treating several chronic conditions including hypertension (Masi, et al., 2011), human immunodeficiency virus (HIV) (Wood, et al., 2016), substance abuse and pain management among American Indians (Katzman, Fore et al., 2016), and most recently Autism (Mazurek, Brown, Curran, & Sohl, 2016). This study did not directly measure any knowledge increase as none of the participants attended a Project ECHO session during the implementation period. However, the provider who has attended Project ECHO did feel like Project ECHO helped her provide better care and help her be more efficient. She, along with the other two providers who completed the final interview, all expressed that Project ECHO had the ability to increase their knowledge and abilities and improve their diagnostic skills. Interpretation FACILITATING PROJECT ECHO 27 None of the participants logged into a Project ECHO session during the implementation period. Time and lack of clinic support were the two main barriers discovered and seem to be the strongest inhibitors of provider engagement. Another possible barrier to engaging in Project ECHO is not having a patient that would benefit from the currently offered Project ECHO sessions. In the past the provider who participated in Project ECHO did so when she had a patient who she felt would benefit from the service or perhaps really needed the service. There is a strong motivational component that needs to be further explored. It seems that providers who feel stuck with a patient or are very motivated to help that patient would be more likely to use Project ECHO. More could have been done during the implementation period to encourage participation such as helping the participants participate in a test call, which may have alleviated some of the uneasiness of using a new telecommunication program This required very little time commitment from the participants and even though one of the providers did not complete the entire study they all were receptive of the project. Overall, the project was very inexpensive and could easily be replicated. However, further interventions need to be done to increase the likelihood of engagement such as administrative support e.g. allowing allotted time to participate in the ECHO sessions, time management techniques, and as aforementioned testing the Project ECHO software with the participants or having a designated Project ECHO champion to help troubleshoot and encourage other participants. Future studies may also benefit by choosing a clinic that is actively using Project ECHO and complete a study of what motivates the providers to engage in the service or perhaps what systems they have in place that facilitates participation. Limitations The most significant limitation to the study was that nobody used Project ECHO, which made it difficult to assess all the areas of the E-Facilitation model (Dzinotyiweyi, 2015). We FACILITATING PROJECT ECHO 28 attempted to overcome this by asking questions related to their perception of Project ECHO in relation to the E-Facilitation model. Another limitation was the small number of providers that participated. We initially had four but only three completed the entire study. There was also a lack of diversity of the participants all living in the same geographic region. Some bias could also have occurred as one of authors has worked with most of the participants, as well as some subjectivity in the coding process of the data analysis. Conclusion The Project ECHO model has been proven to be a safe and effective tool to disseminate information to primary care providers and help co-manage patients with common complex diseases. The Project ECHO model is rapidly expanding throughout the country and globally. However, engagement in some areas like Cache Valley Utah has been low. Results from this study demonstrate the challenges of implementing Project ECHO with the main barriers including lack of time and lack of clinic/administrative support. Providers continue to be unengaged even though Project ECHO offers interesting and relevant topics. Some providers felt that there could be additional topics of interest included that may increase engagement. Lastly, there is a motivational component that may also drive providers to engage in Project ECHO particularly if they feel stuck or desire quicker outcomes for the patient due to a long wait time to see a specialist. Project ECHO did seem to meet the criteria cited in the E-Facilitation Model (Dzinotyiweyi, 2015) including the preparation, engagement and ensuring value stages. However, because this study was a very small sample size and none of the providers participated in Project ECHO during the implementation period, more research will need to be done to validate these findings. FACILITATING PROJECT ECHO 29 Further research will need to be done to validate the barriers and facilitators discovered during this study. Further studies may control for the barriers by initiating clinic support to give providers an allotted time to attend Project ECHO as well as teach time management skills to determine if there are any other outlying barriers. Studies may also contact clinics who are currently using Project ECHO more routinely and determine how they have overcome these barriers or what systems they have in place that make them more successful at using it. FACILITATING PROJECT ECHO 30 References Arora, S., Thornton, K., Komaromy, M., Kalishman, S., Katzman, J., & Duhigg, D. (2014). Demonopolizing Medical Knowledge. Academic Medicine, 89(1), 30-32. doi:10.1097/acm.0000000000000051 Arora, S., Thornton, K., Murata, G., Deming, P., Kalishman, S., Dion, D., . . . Qualls, C. (2011). Outcomes of Treatment for Hepatitis C Virus Infection by Primary Care Providers. New England Journal of Medicine, 364(23), 2199-2207. doi:10.1056/nejmoa1009370 Catic, A. G., Mattison, M. L., Bakaev, I., Morgan, M., Monti, S. M., & Lipsitz, L. (2014). ECHO-AGE: An Innovative Model of Geriatric Care for Long-Term Care Residents With Dementia and Behavioral Issues. Journal of the American Medical Directors Association, 15(12), 938-942. doi:10.1016/j.jamda.2014.08.014 Charlton, M., Schlitchting, J., Chioreso, C., Ward, M., & Vikas, P. (2015). Challenges of Rural Cancer Care in the United States. Oncology. Retrieved from http://www.cancernetwork.com/oncology-journal/challenges-rural-cancer-care-unitedstates/page/0/2 Colorafi, K. J., & Evans, B. (2016). Qualitative Descriptive Methods in Health Science Research. HERD: Health Environments Research & Design Journal, 9(4), 16-25. doi:10.1177/1937586715614171 Croft, J. B., Wheaton, A. G., Liu, Y., Xu, F., Lu, H., Matthews, K. A., . . . Holt, J. B. (2018). Urban-Rural County and State Differences in Chronic Obstructive Pulmonary Disease - United States, 2015. MMWR. Morbidity and Mortality Weekly Report, 67(7), 205-211. doi:10.15585/mmwr.mm6707a1 Dzinotyiweyi, M. (2015). Development of an iterartive process model of e-facilitation to Support lifelong learning. South Aferica International Conferance on Educational Technologies. Hashibe, M., Kirchhoff, A. C., Kepka, D., Kim, J., Millar, M., Sweeney, C., . . . Mooney, K. (2018). Disparities in cancer survival and incidence by metropolitan versus rural residence in Utah. Cancer Medicine. doi:10.1002/cam4.1382 Katzman, J. G., Fore, C., Bhatt, S., Greenberg, N., Salvador, J. G., Comerci, G. C., . . . Karol, S. (2016). Evaluation of American Indian Health Service Training in Pain Management and Opioid Substance Use Disorder. American Journal of Public Health, 106(8), 1427-1429. doi:10.2105/ajph.2016.303193 Katzman, J. G., Galloway, K., Olivas, C., Mccoy-Stafford, K., Duhigg, D., Comerci, G., . . . Arora, S. (2016). Expanding Health Care Access Through Education: Dissemination and Implementation of the ECHO Model. Military Medicine, 181(3), 227-235. doi:10.7205/milmed-d-15-00044 FACILITATING PROJECT ECHO 31 Knapp, H., & Pangarkar, S. (2015). Utilizing the ECHO Model in the Veterans Health Affairs System: Guidelines for Setup, Operations and Preliminary Findings. Future Internet, 7(4), 184-195. doi:10.3390/fi7020184 Komaromy, M., Duhigg, D., Metcalf, A., Carlson, C., Kalishman, S., Hayes, L., . . . Arora, S. (2016). Project ECHO (Extension for Community Healthcare Outcomes): A new model for educating primary care providers about treatment of substance use disorders. Substance Abuse, 37(1), 20-24. doi:10.1080/08897077.2015.1129388 Masi, C., Hamlish, T., Davis, A., Bordenave, K., Brown, S., Perea, B., . . . Johnson, D. (2011). Using an Established Telehealth Model to Train Urban Primary Care Providers on Hypertension Management. The Journal of Clinical Hypertension, 14(1), 45-50. doi:10.1111/j.1751-7176.2011.00559.x Mazurek, M. O., Brown, R., Curran, A., & Sohl, K. (2016). ECHO Autism. Clinical Pediatrics, 56(3), 247-256. doi:10.1177/0009922816648288 Phillips, D., & Barclay, L. (2017, February 10). Rural US Persons at Risk for Death From 5 Leading Causes. Retrieved May 29, 2017, from http://www.medscape.org/viewarticle/875155?nlid=113524_2713&src=wnl_cmemp_170 327_mscpedu_nurs&uac=121062PR&impID=1317109&faf=1 Project ECHO. (2017). Retrieved July 28, 2017, from http://healthcare.utah.edu/echo Watts, S. A., Roush, L., Julius, M., & Sood, A. (2015). Improved glycemic control in veterans with poorly controlled diabetes mellitus using a Specialty Care Access NetworkExtension for Community Healthcare Outcomes model at primary care clinics. Journal of Telemedicine and Telecare, 22(4), 221-224. doi:10.1177/1357633x15598052 Wood, B. R., Unruh, K. T., Martinez-Paz, N., Annese, M., Ramers, C. B., Harrington, R. D., . . . Spach, D. H. (2016). Impact of a Telehealth Program That Delivers Remote Consultation and Longitudinal Mentorship to Community HIV Providers. Open Forum Infectious Diseases, 3(3). doi:10.1093/ofid/ofw123 FACILITATING PROJECT ECHO 32 Appendix A Pre-Implementation Interview Code Book Code/Category Knew/used Project ECHO prior to intervention Positive perception of Project ECHO Access to reliable computer and internet connection Comfortable using technology Content expert present at each Project ECHO session Positive perception of telecommunication systems Lack knowledge of Project ECHO Perception of, or actual time limitations Uncomfortable with format of meeting/discussion Negative or neutral perception Definition Participant has heard of and used Project ECHO prior to the intervention Any statement, thought process or phrase that demonstrates and overall positive perception of Project ECHO Any individual who reports and feels that he/she has access to both reliable internet and a reliable computer either at home, work or both. Refers to someone with average or greater computer skills and demonstrates this comfort level either by directly saying so or indirectly through their actions or stated actions Refers to the individuals perception that a content expert is present at each Project ECHO session Any statement, phrase or thought process that demonstrates a positive perception of telecommunication systems in general, not specific to Project ECHO Refers to a person who has no previous knowledge of Project ECHO prior to the intervention Refers to any perception that a provider/participant may have that they do not have enough time or actual time barriers to attending or using Project ECHO. Any statement or thought process that expresses concerns about attending a live online discussion strictly based on the format of the meeting or concerns about being an active participant. Any statement or thought process Example "I have used Project ECHO 2-3 times this last year" "I found Project ECHO to be very helpful" "Project ECHO can help those in rural areas" "I see the potential of Project ECHO" "Dr. Smith states that he has access to reliable internet and a computer" "I am fairly comfortable using technology" "overall I am great with computers" "I enjoy learning through an online platform" "The experts in the Project ECHO sessions were very knowledgeable" "Telecommunication systems are going to grow" "Very helpful to rural patients" "Overall I think that it telehealth/telemedicine can be very useful" "Never heard of project ECHO" "It's difficult for me to attend Project ECHO at these times because I am usually seeing patients" "I just do not have the time to attend Project ECHO sessions" "I don't like me Web Cam being projected to others" I am not sure when or how to jump in" " I would prefer just attending a lecture" "I really don't have much FACILITATING PROJECT ECHO of telemedicine/telecommunication Frequency of contact with specialists Collaboration with specialists is important Motivated by patient population 33 that demonstrates a negative or neutral perception of telemedicine and/or telecommunication. Basically, anything but a positive perception of telemedicine/telecommunication. The number of times on average that the provider reports a collaboration with specialists either in writing or direct communication Participants feel that it is important to collaborate with specialist Any statement or thought process that demonstrates that he or she is strongly motivated to help their patients. These providers may attend extra training or participate in Project ECHO for the purpose of helping their patients Negative perception of collaboration with specialists Any statement or thought process that demonstrates a negative experience or perception of collaborating with specialists Negative/neutral live online seminar experience. Refers to someone who has had problems or negative experiences attending live online seminars in the past or someone who is neutral towards the seminars due to lack of experience. experience with telemedicine" "Lots of barriers to using telemedicine" "I think there are some diagnostic errors with telemedicine" "I have daily contact with specialists" "I probably talk to a specialist directly like 1-2 times per week" "Collaboration with specialists is very important" "By attending Project ECHO I can save my patients a lot of time and money" "I really wanted to help this patient" "I learned a lot about managing my patients condition" "I really want to help my patient but needed some help" "This particular specialists was not very helpful" "It is very hard to collaborate with specialists" "Sometimes I feel dumb when collaborating with specialists" "I have never attended any live online seminars" "These events are hard to listen to" "There were a lot of technical issues with live seminars" FACILITATING PROJECT ECHO 34 Appendix B Pre-Implementation Interview Data Concept Map FACILITATING PROJECT ECHO 35 Appendix C Post-Implementation Interview Code Book Code/Category Perceived or actual time barriers Lack of clinic support Motivated by patient population Project ECHO improves access and/or quality care Project ECHO can/may lead to professional development Confident in ability to use Project ECHO Project ECHO topic must be relevant/useful for patient population. Definition Refers to any perception that a provider/participant may have that they do not have enough time or actual time barriers to attending or using Project ECHO. Example "It's difficult for me to attend Project ECHO at these times because I am usually seeing patients" "I just do not have the time to attend Project ECHO sessions" Any thought or statement that "I would attend more if we had a suggests there is an overall lack set allotted amount of time to do of clinic support to using Project it" ECHO. "We need more support on the administrative end" Any statement or thought process "By attending Project ECHO I that demonstrates that he or she can save my patients a lot of time is strongly motivated to help their and money" patients. These providers may "I really wanted to help this attend extra training or patient" participate in Project ECHO for "If it were not possible for them the purpose of helping their to see a specialist Project ECHO patients would be very helpful" The perception of belief that "My patients can gain access to a Project ECHO does or can specialists a lot faster through improve either access to Project ECHO" specialists and/or improve the "I think Project ECHO can grant overall quality of care. access to specialist for those who can't afford to see one" Any perception or belief that "Project ECHO has helped me Project ECHO can or does lead improve in areas where I feel to professional development such uncomfortable" as improving skills, knowledge, "Project ECHO has can lead to collaborative efforts with others, improvement of chronic care etc. management" Statements suggesting that the "We can figure out the provider is confident in their technology side of things if we ability to register, log in to, and just have the time to attend" use Project ECHO. "I feel that I can log on and use Project ECHO" Statements that suggest that the " I would be very interested if topic(s) offered Project ECHO they would offer a cardiac will influence their desire to use Project ECHO" Project ECHO. "I wish they still did the headache and Neuro Project ECHO as I have several headache patients that it could benefit" FACILITATING PROJECT ECHO 36 Appendix D Post-Implementation Interview Data Concept Map