Development and Implementation of a Referral Resource for a Multidisciplinary Team at an Urban Free Mental Health Clinic

Approximately 30 million individuals in the United States are uninsured, and 11.2 million individuals are struggling with Serious Mental Illness (SMI). The lack of comprehensive health resources impedes uninsured and low-income individuals in reaching recovery. Free mental health clinics were formed to act as a safety net for this population, yet a 2015 systematic review found that the majority of student providers working in a free clinic did not know how to provide continuity of care for uninsured and low-income patients, and they requested an orientation regarding ongoing comprehensive care and these issues. A quality improvement project was developed to help increase providers' knowledge and awareness of available resources. Providers in an urban free mental health clinic were surveyed to identify gaps in knowledge. A referral resource and quick reference list were developed and presented to providers. The presentation included current data on uninsured rates, needs of the uninsured, and highlighted each available category within the resource. This project also sought to assess the satisfaction of the resource among providers and volunteers within this clinic setting by implementing a post-survey. Out of 29 eligible staff members, 16 completed the pre-intervention survey, and 6 attended the intervention training and completed the post-intervention survey. Thirty-one percent of participants felt the clinic provided adequate resources prior to the intervention, and 83.3% following the intervention. Provider's perception of their referral rate appeared to increase with majority reported referrals 25-50% before the intervention and 50-100% of the time post-intervention. Providers lacked knowledge and awareness of referral resources within this urban free mental health clinic. Barriers to comprehensive health care include a lack of awareness of referral resources, medical insurance, and low-income status. The implementation of a referral resource appeared to improve the barrier to providing continuity of care.

Development and Implementation of a Referral Resource for a Multidisciplinary Team at an Urban Free Mental Health Clinic Tessa Hamilton The University of Utah College of Nursing 1 Abstract Background: Approximately 30 million individuals in the United States are uninsured, and 11.2 million individuals are struggling with Serious Mental Illness (SMI). The lack of comprehensive health resources impedes uninsured and low-income individuals in reaching recovery. Free mental health clinics were formed to act as a safety net for this population, yet a 2015 systematic review found that the majority of student providers working in a free clinic did not know how to provide continuity of care for uninsured and low-income patients, and they requested an orientation regarding ongoing comprehensive care and these issues. Methods: A quality improvement project was developed to help increase providers' knowledge and awareness of available resources. Providers in an urban free mental health clinic were surveyed to identify gaps in knowledge. A referral resource and quick reference list were developed and presented to providers. The presentation included current data on uninsured rates, needs of the uninsured, and highlighted each available category within the resource. This project also sought to assess the satisfaction of the resource among providers and volunteers within this clinic setting by implementing a post-survey. Results: Out of 29 eligible staff members, 16 completed the pre-intervention survey, and 6 attended the intervention training and completed the post-intervention survey. Thirty-one percent of participants felt the clinic provided adequate resources prior to the intervention, and 83.3% following the intervention. Provider's perception of their referral rate appeared to increase with majority reported referrals 25-50% before the intervention and 50-100% of the time post-intervention. 2 Conclusions: Providers lacked knowledge and awareness of referral resources within this urban free mental health clinic. Barriers to comprehensive health care include a lack of awareness of referral resources, medical insurance, and low-income status. The implementation of a referral resource appeared to improve the barrier to providing continuity of care. 3 Introduction Problem Description In 2019, a client presented to an urban free mental health clinic and disclosed that she was suffering from domestic violence. After completing a safety assessment, the student nurse practitioner was unsure to which community resources to refer this client. At the time, no referral resource was fully established, and the other staff present were unaware of what resources were appropriate. Free mental health clinics are primarily staffed by students and volunteers who provide care. A systematic review found that most students are unaware of referral options and continuity of care for their patients (Kilbourne, Switzer, Hyman, Crowley-Matoka, & Fine, 2006). Free mental health clinics provide services to the uninsured and low-income population. This population faces many challenges in receiving adequate care and wellness. Thirty million individuals in the United States are still uninsured, with as many as 11.8% uninsured in Utah (Cohen, Terlizzi, & Martinez, 2019). The healthcare needs of the uninsured and low-income are complex; this population has incredibly high comorbidity rates, meaning they have a physical illness at the same time as their mental illness (Notaro et al., 2012). Oftentimes, this population is left unsure where to receive care. It was estimated that between 2003-2006, health disparities cost approximately 229.4 billion in the United States, which included inappropriate mental health visits in the emergency room (LaVeist, Gaskin, & Richard, 2009). Available Knowledge 4 Psychiatric institutions. For many years the severely mentally ill were sent to psychiatric institutions. There was a push for deinstitutionalization, beginning in 1955 (Morrissey & Goldman, 1986). Deinstitutionalization took the care of the severely mentally ill out of the institutions and put their care into the community. It is argued that some of the consequences of deinstitutionalization were increases in homelessness, inappropriate emergency department visits and the criminalization of the mentally ill (Hudson, 2016). Formation of free clinics. Free clinics were first formed in the 1960s to provide care for those in need and worked as a safety net. These first free clinics began as a way to provide specific care for drug-treatment issues (Smith, 1971). It is now estimated that there are at least 1,114 free clinics spanning across the United States, and they range from a small basement clinic in a Presbyterian church in Michigan which runs one to three hours a day to a large family clinic in Los Angeles, California that staffs over 2,000 volunteers (Gertz, Frank, & Blixen, 2011). Free mental health care clinics provide behavioral care, which focuses on the mental and emotional well-being of the uninsured and low-income populations. However, it is unclear how many free clinics specialize in mental health care and with that, the management of serious mental illness (SMI). Vulnerable population. Thirty million individuals in the United States are still uninsured, with as many as 11.8% uninsured in Utah (Cohen et al., 2019). The definition of a vulnerable population includes populations who are less likely to receive care of "appropriate quality and quantity" (Pauly & Pagán, 2007). The healthcare needs of the uninsured and low-income are complex, including higher rates of SMI and physical 5 illnesses (Notaro et al., 2012). The majority of these illnesses and limitations on functioning are preventable, making these struggles disconcerting. Approximately one in five U.S. adults struggles with a mental illness with nearly 11.2 million meeting criteria for SMI (National Institute of Mental Health, 2019). SMI is defined as any mental, behavioral or emotional disorder which results in significant impairment with life activities and commonly includes diagnoses such as schizophrenia, depression and bipolar disorder. SMI is associated with barriers to employment resulting in a lowered-income, legal difficulties and inappropriate utilization of the health care system. According to the National Survey on Drug Use and Health, 70.7% of individuals with SMI reported that they could not afford treatment (Substance Abuse and Mental Health Services Administration, 2012). Nonelderly adults with mental illness often have co-morbid health conditions including; heart conditions, diabetes, hypertension, asthma and addictions. One study reports that those with SMI have high rates of physical illness which leads to a poorer response to mental health treatment (Pohle, Domschke, Roehrs, Arolt, & Baune, 2009). Adequate continuity of care. This population faces many barriers to receiving comprehensive and evidenced-based treatment. Continuity of care is defined as providing a coordination of services that fit the patient's current needs long-term (Weaver, Coffey, & Hewitt, 2017). As mental health services have moved to the community, services have become increasingly fragmented. Even though Maslow's hierarchy has limited research in the world of mental health, envisioning needs as a hierarchy gives a visual representation of how one's basic needs, such as shelter and pending homelessness, may 6 affect their recovery. For example, 51% of U.S. personal bankruptcies are due to medical bills (Hackney, Friesner, & Johnson, 2016). Often, the work being provided at a free mental health clinic is the last line keeping clients from homelessness. A literature review found that there were promising results in functioning and wellness when patients had good continuity of care (Weaver et al., 2017). Interventions should be tailored to the individual client, otherwise known as person-centered care; active engagement in planning can help someone develop the confidence and skills needed to reestablish his or her identity and regain control (Adams & Grieder, 2014). Rationale The Three Phases of Health Disparities is the framework for this quality improvement project and guides care for vulnerable populations who receive a lower quality of care than other populations (Kilbourne et al., 2006). This framework recommends a stepwise approach for the implementation of interventions to reduce health disparities and includes the following phases: identification of health disparities, understanding health disparities, and implementation of interventions that decrease health disparities. Providers' role includes providing evidence-based care which includes safe and comprehensive options. Treatment plans for those struggling with SMI are often extensive. When working with this population, improving continuity of care is one way to improve treatment and improve wellness. A systematic review found that most 7 students are unaware of referral options and continuity of care for their patients (Kilbourne et al., 2006). Providers are unable to provide comprehensive care if they are unaware of available resources. Specific Aims The purpose of this quality improvement project is to develop and implement a referral resource to increase the provider's knowledge and awareness of available resources for clients they serve in this local free mental health clinic. Prior to the implementation of the referral resource, a pre-intervention survey was taken. Then, upon conclusion of the training and use of the referral resource, a post-intervention survey assessing provider knowledge of available resources was administered, measuring providers' comfort level and liking. The referral resource is intended to improve continuity of care for the clinic's client population and enhance the ability of providers, students, and volunteers to provide improved treatment in the management of care for this vulnerable population. The hypothesis is that staff and students who utilize the referral resource will feel more knowledgeable about managing the care of clients with complex diagnoses and make better use of various referral options available to them. A short-term aim is to disseminate the findings from the pre- and postintervention surveys to the volunteer coordinator of this free mental health clinic to help improve the care of clients. Due to the limited data regarding referral resources in free mental health services, the long-term aim of this report is to contribute more data regarding supportive resources in free mental health clinics. Methods 8 Context This quality improvement project took place at a local urban free mental health clinic. The clinic serves the uninsured and those with a household income of under 150% of the Federal Poverty Level. The primary focus of the clinic is conducting psychiatric evaluations and psychiatric medication management. The clinic is open five days a week, three of these days being clinical days, and two being administrative only days. The clinic includes a paid medical director, executive director, volunteer coordinator, and front office coordinator. The staff includes two psychiatric mental health nurse practitioner (PMHNP) providers in partnership with the University of Utah, who provides oversite for Doctor of Nursing Practice (DNP) students specializing in Psychiatric Mental Health. Additional providers include five DNP students, one bachelor of nursing student, six psychiatry residents, eleven volunteers, and four medical translators. Currently, the clinic does not employ a social worker and primarily provides medication management. The clinic defines a provider as the clinical and non-clinical staff who provide direct patient care. For this project, the medical translators were excluded from this project as they do not provide direct patient care. The clinic uses a donated basic electronic medical record system for documentation and record storage. Intervention(s) This quality improvement project was implemented by a third-year Doctor of Nursing Practice (DNP) student in psychiatric mental health who has worked as an intern at the clinic for two years. An advanced practice registered nurse who specializes in psychiatric mental health and works at the clinic was recruited for guidance on this 9 project, as well as the volunteer coordinator at the clinic who provides onboarding and training for all providers of the clinic. The project was divided into four phases and are as follows: 1. Assess providers' knowledge gaps and awareness in referral resources for uninsured and low-income clients. 2. Develop a referral resource to be utilized by multidisciplinary providers within the clinic. 3. Provide training via Microsoft PowerPoint about the resource and implement the resource within the clinic. 4. Re-assess providers' knowledge and awareness of referral resources for uninsured and low-income clients. In the first stage, pre and post-surveys were created by utilizing a previous tool and previous experience working with uninsured and low-income patients. The presurvey was then sent to providers prior to the creation of the resource. For the purpose of this project, providers include all volunteers, staff, and providers who provide direct patient care within the clinic, excluding four Spanish translators. Secondly, consultation with the volunteer coordinator and the two PMHNP's on staff revealed that an electronic referral resource would be beneficial to the clinic, and this was developed based on best practice guidelines, collaboration with various community resources, and feedback from the volunteer coordinator. Calls to each community resource were made by the DNP student to assess current services being offered. Resources and guidelines by other mental health clinics and resources were used to direct the formatting of the referral resource. Finally, following the presentation, the referral resource was delivered to the providers of the clinic via email, downloaded onto all clinic devices, and added to the 10 clinic's universal Dropbox. Providers who attended the presentation completed the postsurvey, and data comparison between the pre-and post-surveys was assessed to evaluate whether the referral resource and training were useful. The findings were then presented to the volunteer coordinator within the clinic. Study of the Intervention(s) Surveying providers for changes via pre-/post-surveys assessed the impact of the intervention. Surveys were used to assess the primary outcome of the project, primarily their knowledge and perception related to referral resources. Surveys were administered via paper to ensure survey completion and due to the clinic's limited technological resources. Measures Data was collected via a pre- and post-survey, which measured the providers' knowledge of referral resources. All surveys were developed via Microsoft Word and approved for content by the content expert. The survey collected information such as age, role at the clinic, years of experience, and how long providers had worked at the clinic. Additionally, a Likert scale was used to determine knowledge and awareness regarding referral resources. Additional questions were included to identify the comfort level of providers in assessing, diagnosing, and treating patients with serious mental illness and their satisfaction with the amount and quality of resources available to them to help facilitate the care of this population. The DNP student invited, in person, and via email, the providers at the urban free mental health clinic to complete the pre-survey prior to presentation and implementation 11 of the referral resource. Of the 29 team members, 16 completed the pre-intervention survey. Following the training of the referral resource, a post-intervention survey was implemented. Of the 29 team members, 6 attended the training and completed the postsurvey. Analysis This study used descriptive statistics, via Microsoft Excel, and describe the demographics and knowledge of participating providers. Quantitative and qualitative data were collected using open-ended questions via the surveys. Knowledge and awareness of referral resources appeared to increase following training and implementation of the resources; however, this was not tested statistically as the results did not calculate due to the small sample size. Ethical Considerations This project fits the definition of health care quality improvement and the University of Utah Institutional Review Board determined IRB review was not required. This project does not fit the definition of research (Department of Health and Human Services, 2009). No Protected Heath Information (PHI) was utilized throughout this project and no conflicts of interest were identified during the completion of this project. Results Provider demographics In the first stage of the project, the pre-intervention survey was sent out to all 29 providers within the clinic. The pre-intervention survey completion rate was 55% (n=16). Providers demographics can be viewed in Table 1. As shown in the table, 50% (n=8) of 12 the providers were between the ages of 25-34, with the rest falling between the ages of 18-24 (25%), 35-44 (12.5%) and 45 or older (12.5%). There were 11 individuals (68.8%) who identified as female, with the remaining 5 (31.3%) identifying as male. More than half of the survey participants were comprised of non-paid staff including; volunteers (37.5%), DNP students (25.0%) and resident MD's (12.5%) and was followed by paid staff with 2 APRN's (12.5%) and paid administration (12.5%). The pre-intervention showed that 31.3% of participants had been at then clinic for less than a year and 25.0% at the clinic for less than 6 months, with 31.3% having been at the clinic for one-two years and 12.5% for more than two years. The average hours at the clinic per week showed that 31.3% were there for 2-4 hours, 25.0% spending 7-9 hour at the clinic, 37.5% spending more than 9 hours at the clinic with only one (6.0%) spending 5-7 hours. The majority of participants (50.0%) spend at least 2-4 hours with clients each week, with only 18.8% (n=3) spending 0-1 hours, 12.5% (n=2) spending 5-7 hours and 12.5% (n=2) spending more than 9 hours with clients. When asked if the clinic had provided adequate training and resources regarding referral resources the majority of participants answered no (43.8%), neither agree or disagree (25.0%) with the remaining answering yes (31.3%). In working with clients, providers estimated referring clients to outside resources with (43.8%) doing so 25-50% of the time, and (12.5%) 50-75% of the time, (37.5%) 1-25% of the time and one participant (6.0%) 0% of the time. In the same survey, Likert scale questions were used to assess providers' perception regarding referral resources. The categories included the options strongly 13 disagree, disagree, neutral, agree and strongly agree. Most participants agreed (31.3%) or strongly agreed (43.8%) that they asked their clients what they wanted out of treatment, with the remaining (18.8%) neutral, or disagree (6.3%). Most providers were neutral (43.8%) when asked if there were enough resources in the clinic to implement evidencebased treatment plans to fit their patient's needs, with (25.0%) agree and (12.5%) strongly agree, the remaining providers disagreed (12.5%) or strongly disagreed (6.3%). Most providers disagree (37.5%) or were neutral (25.0%) when asked if they knew where to find information about accessing outside resources available to their clients, with the remaining who agree (18.8%) or strongly agree (18.8%). When asked about awareness of the extent of services offered by outside resources providers answered neutral (37.5%), disagree (18.8%) or strongly disagree (18.8%) with the remaining agree (12.5%) or strongly agree (12.5%). During the third stage, the referral resource was implemented by way of PowerPoint presentation with delivery of the resource via Microsoft Excel and Microsoft Word to attending providers. 21% (n=6) providers attended the resource training and completed the post-intervention survey. The demographics of the providers who attended the training were primarily 35 years or older (67%), female (66.7%), DNP students (50.0%), who had been at the clinic for at least one-two years (66.7%), who were physically at the clinic 9+ hours a week (66.7%), and average contact time with clients was 9+ hours (33.3%). When asked if the clinic had provided adequate training on referral resources, (83.3%) responded yes, and (16.6%) responded neither agree or disagree with no one responding no (0.0%). When asked how often they estimate they 14 refer clients to outside resources, providers answered evenly between 25-50% of the time (33.3%), 50-75% of the time (33.3%) and 75-100% of the time (33.3%). Within the Likert scale, questions (83.3%) of providers answered strongly agree that they asked their clients what they want out of treatment. Of providers, (66.7%) answered neutral to whether the clinic has enough resources to implement evidence-based treatment plants to fit their patient's needs. Participants either agree (50.0%) or strongly agree (50.0%) that they know where to find information about accessing outside resources that are available to their client and that they are aware of services offered by outside resources. Mann-Whitney U Test: When attempting to compare pre and post-intervention data, the Mann Whitney U Test was attempted, but failed to calculate due to the limited number of participants who completed the training and the post-intervention survey. It appears that there might have been an increase in knowledge and awareness of resources but it was not tested statistically. Qualitative: Within the pre- and post-intervention surveys there were two free text sections for questions or comments. Responses were limited. Within the post-intervention survey, one participant stated: "I will use this all the time" and another stating "I wish I would have had this sooner." Contextual elements: Pre and post-intervention sample sizes varied greatly, with only 6 participants completing the training and filling out the post-survey. Throughout the study, there was 15 high turnover and onboarding of volunteers due to school schedules. Another aspect that interfered with implementation was the executive director resigned, with other staff members left to cover this significant clinic role and the executive director position remaining unfilled. This was disruptive to the typical workflow of the staff, including the volunteer coordinator, and content expert of this project. Most providers are at the clinic one day a week. Discussion Summary With the majority of the participants of the training and post-survey being DNP students, one strength of this project is that a fellow DNP student created the referral resource. As a result of this project, the clinic is considering applying for funding to bring on board a full-time case manager. The role of the case manager will be to connect clients to outside resources in addition to the management of medications being done at the clinic. Interpretation Descriptive statistics of the pre- and post-intervention indicate the referral resource is useful, enhancing providers' awareness and use of outside resources. However, this was unable to be tested statistically due to the small sample size. After the implementation of the referral resource, it appears there was an improvement in 16 provider's asking their clients what they wanted out of treatment. The providers indicated improvement in knowing where to find information about accessing outside resources and being aware of the extent of services offered by outside resources (refer to Table 3). The post-intervention responses, though limited, suggest that providers will utilize the referral resource. Time spent developing the referral resource was minimal in contrast with the time clients spend trying to navigate a fragmented health system, especially if dealing with SMI. One of the benefits to the electronic referral resources as it can be further developed by the clinic and perhaps future students. Limitations Limits to the generalizability of this project include the small sample size. There was a smaller number of providers that participated in the training and post-survey than was initially estimated. Some factors that may have possibly caused this were inconsistent schedules of providers and the volunteer basis of the clinic. The clinic does not hold regular training, and while providers were encouraged to attend, there was no incentive given for attendance. Another limitation of this project is the higher rate of provider turnover as most providers are students and work other jobs. With no participant identifier, the project was unable to assess changes in perception and knowledge specific to each provider. Another limitation of this project is selection bias, namely which providers were available at the clinic more often. Although all providers were encouraged to fill out surveys and participate in the training via email, providers who were at the clinic more 17 often and on the same days as the DNP student implementing this project were encouraged in-person and more frequently. There is an ever-growing demand for services with the increasing burden of chronic care, such as experienced by the uninsured and low-income. What services are available can change. Once a patient gets referred, it is dependent on whether the patient can afford services rendered if there is a fee. Conclusions The findings of this project support staff training and providing referral resource training for free mental health clinics. The training PowerPoint presentation, referral list, and referral resource can be utilized to train future providers of this clinic, which has limited funding and no case manager on staff. A survey with a larger sample size, perhaps including past providers, could guide the clinic as to what steps could be taken in the future to increase providers' knowledge and awareness of resources and their compliance in working towards continuity of care. Acknowledgments This project was developed following the breakdown of previous scholarly project attempts, and could not have been completed without the encouragement and support from many individuals. I want to thank my project chair, Dr. Pamela Hardin, for her gentle support and guidance. I also would like to thank my content experts, Sara Webb and Mary-Grace Heustess, for your willingness to assist me along this journey despite your overwhelmingly busy schedules. Nicholas Baggett, your gentle natural confidence and 18 constant words of encouragement provided me with needed stamina to keep going. Sheila Deyette, your genuine concern and investment have kept me afloat throughout the entire graduate school experience, including this scholarly project. To Randy Bullock, my small group leader, your guidance and genuine concern have buoyed my burnt-out spirit. Amy Morton-Miller, you were the first person that came to mind when considering a potential chair for my Jonas scholarship; due to your collaborative nature and willingness to support us as your students. All of you are the definition of teamwork. Lastly, I would like to thank all of my clients; all of your stories have inspired me along my journey to becoming a future provider and caretaker. 19 References Cohen, R. A., Terlizzi, E. P., & Martinez, M. E. (2019). Health Insurance Coverage: Early Release of Estimates From the National Health Interview Survey, 2018. In U. S. D. o. H. a. H. Services (Ed.). Department of Health and Human Services. (2009). Protection Of Human Subjects. Gertz, A. M., Frank, S., & Blixen, C. E. (2011). A Survey of Patients and Providers at Free Clinics Across the United States. Journal of Community Health, 36(1), 83-93. doi:10.1007/s10900-010-9286-x Hackney, D. D., Friesner, D., & Johnson, E. H. (2016). What is the actual prevalence of medical bankruptcies? International Journal of Social Economics, 43(12), 1284-1299. Hudson, C. G. (2016). A Model of Deinstitutionalization of Psychiatric Care across 161 Nations: 2001-2014. International Journal of Mental Health, 45(2), 135-153. doi:10.1080/00207411.2016.1167489 Kilbourne, A. M., Switzer, G., Hyman, K., Crowley-Matoka, M., & Fine, M. J. (2006). Advancing Health Disparities Research Within the Health Care System: A Conceptual Framework. American Journal of Public Health, 96(12), 2113-2121. doi:10.2105/AJPH.2005.077628 LaVeist, T. A., Gaskin, D. J., & Richard, P. (2009). The economic burden of health inequalities in the United States. Morrissey, J. P., & Goldman, H. H. (1986). Care and Treatment of the Mentally Ill in the 20 United States: Historical Developments and Reforms. The ANNALS of the American Academy of Political and Social Science, 484(1), 12-27. doi:10.1177/0002716286484001002 National Institute of Mental Health. (2019). Mental Illness. Retrieved from https://www.nimh.nih.gov/health/statistics/mental-illness.shtml Notaro, S. J., Khan, M., Bryan, N., Kim, C., Osunero, T., Senseng, M. G., . . . Nasaruddin, M. (2012). Analysis of the Demographic Characteristics and Medical Conditions of the Uninsured Utilizing a Free Clinic. Journal of Community Health, 37(2), 501-506. doi:10.1007/s10900-011-9470-7 Pauly, M. V., & Pagán, J. A. (2007). Spillovers and vulnerability: the case of community uninsurance. Health affairs (Project Hope), 26(5), 1304. doi:10.1377/hlthaff.26.5.1304 Pohle, K., Domschke, K., Roehrs, T., Arolt, V., & Baune, B. T. (2009). Medical Comorbidity Affects Antidepressant Treatment Response in Patients with Melancholic Depression. Psychotherapy and Psychosomatics, 78(6), 359-363. doi:10.1159/000235975 Smith, D. E. (1971). Love needs care ; a history of San Francisco's Haight-Ashbury Free Medical Clinic and its pioneer role in treating drug-abuse problems ([1st ed.].. ed.): Boston : Little, Brown. Substance Abuse and Mental Health Services Administration. (2012). Results From the 2012 National Survey on Drug Use and Health: Mental Health Detailed Tables. Retrieved from 21 http://www.samhsa.gov/data/NSDUH/2k12MH_FindingsandDetTables/MHDT/N SDUHWeaver, N., Coffey, M., & Hewitt, J. (2017). Concepts, models and measurement of continuity of care in mental health services: A systematic appraisal of the literature. Journal Of Psychiatric And Mental Health Nursing, 24(6), 431-450. doi:10.1111/jpm.12387 Table 1 Participants' Demographics Variable Age 18-24 25-34 35-44 45 or older Gender Male Female Role Student, DNP Resident, MD APRN Volunteer Paid administration and support staff Duration at clinic less than 6 months less than a year one-two years more than two years Pre- N (%) Post- N (%) 4 8 2 2 1 1 2 2 (25.0%) (50.0%) (12.5%) (12.5%) 5 (31.3%) 11 (68.8%) 4 2 2 6 22 (25.0%) (12.5%) (12.5%) (37.5%) (16.7%) (16.7%) (33.3%) (33.3%) 2 (33.3%) 4 (66.7%) 3 0 1 0 (50.0%) (0.0%) (16.7%) (0.0%0 2 (12.5%) 2 (33.3%) 4 5 5 2 0 2 4 0 (25.0%) (31.3%) (31.3%) (12.5%) (0.0%) (33.3%) (66.7%) (0.0%) Avg. # of hours per week at clinic 0-1 2--4 5--7 7--9 9+ Avg. contact time with clients while at clinic 0-1 2--4 5--7 7--9 9+ 0 5 1 4 6 (0.0%) (31.3%) (0.1%) (25.0%) (37.5%) 0 0 0 2 4 (0.0%) (0.0%) (0.0%) (33.3%) (66.7%) 3 8 2 1 2 (18.8%) (50.0%) (12.5%) (0.1%) (12.5%) 1 1 1 1 2 (16.7%) (16.7%) (16.7%) (16.7%) (33.3%) Table 2 Resources Variable Pre Post The clinic has provided adequate training and resources regarding referral resources Yes 5 (31.3%) 5 (83.3%) No 7 (43.8%) 0 (0.0%) Neither agree or disagree 4 (25.0%) 1 (16.6%) How often do you estimate you refer your clients to outside resources 0% 1 (6.0%) 0 (0.0%) 1-25% 6 (37.5%) 0 (0.0%) 25-50% 7 (43.8%) 2 (33.3%) 50-75% 2 (12.5%) 2 (33.3%) 75-100% 0 (0.0%) 2 (33.3%) 23 Table 3 Likert scale survey questions Referral resource experience at clinic with Likert Scale survey questions. Pre- & Post Strongly Disagree, Neutral, Agree, intervention disagree, No. (%) No. (%) No. (%) No. (%) 0 (0%) 1 (6.3%) 3 (18.8%) 5 (31.3) I ask my clients what they want out of Pre 0 (0%) 0 (0%) 0 (0%) 1 (16.7%) treatment. Post There are enough resources in this Pre 1 (6.3%) 2 (12.5%) 7 (43.8%) 4 (25.0%) clinic to implement an evidence-based 0 (0%) 0 (0%) 4 (66.7%) 2 (12.5%) treatment plan to fit the patient's needs. Post I know where to find information about Pre 0 (0%) 6 (37.5%) 4 (25.0%) 3 (18.8%) accessing outside resources that are 0 (0%) 0 (0%) 0 (0%) 3 (50.0%) Post available to my clients. I am aware of the extent of services Pre 3 (18.8%) 3 (18.8%) 6 (37.5%) 2 (12.5%) offered by outside resources 0 (0%) 0 (0%) 0 (0%) 3 (50.0%) Post 24 Strong agree, No. (% 7 (43.8% 5 (83.3% 2 (12.5% 0 (0%) 3 (18.8% 3 (50.0% 2 (12.5% 3 (50.0% Appendix A Referral Resource Training 25 26 27 28 29 30 Appendix B Pre‐/Post‐ Survey 31