Towards automated tooling for disorders of consciousness: considerations and perspectives

This work explores the specialized application of a smart patient room, aimed at defining the use case, potential value, and important considerations for automating aspects of workflows in Disorders of Consciousness (DoC) treatment. Designing technology for patients experiencing DoC is littered with ethical and technical challenges, but previous advancements in hospital automation and other use cases of Internet-of- Things based technology in a hospital suggest that pushing towards the automation of Sensory Stimulation (SS) aspects may improve efficiency for hospital workers and increase the standard of care for DoC patients. We take a qualitative human-centered approach, using semi-structured interviews and thematic analysis, to understand key stakeholder perspectives, ultimately developing themes to define design opportunities and important considerations.

TOWARDS AUTOMATED TOOLING FOR DISORDERS OF CONSCIOUSNESS: CONSIDERATIONS AND PERSPECTIVES by Thomas Kauffman A Senior Honors Thesis Submitted to the Faculty of The University of Utah In Partial Fulfillment of the Requirements for the Honors Degree in Bachelor of Science In Computer Science Approved: _________ Jason Wiese Thesis Faculty Supervisor _ Thomas C. Henderson Honors Faculty Advisor ____ ____________ Mary Hall Director, Kahlert School of Computing _____________________________ Monisha Pasupathi, PhD Dean, Honors College April 2025 Copyright © 2025 All Rights Reserved ABSTRACT This work explores the specialized application of a smart patient room, aimed at defining the use case, potential value, and important considerations for automating aspects of workflows in Disorders of Consciousness (DoC) treatment. Designing technology for patients experiencing DoC is littered with ethical and technical challenges, but previous advancements in hospital automation and other use cases of Internet-ofThings based technology in a hospital suggest that pushing towards the automation of Sensory Stimulation (SS) aspects may improve efficiency for hospital workers and increase the standard of care for DoC patients. We take a qualitative human-centered approach, using semi-structured interviews and thematic analysis, to understand key stakeholder perspectives, ultimately developing themes to define design opportunities and important considerations. ii TABLE OF CONTENTS ABSTRACT ii INTRODUCTION 1 BACKGROUND AND RELATED WORK 3 METHODS 8 RESULTS 11 DISCUSSION 45 CONCLUSION 49 REFERENCES 50 iii 1 INTRODUCTION As smart hospitals continue to evolve, leveraging the technological advancements in location recognition and tracking sensors, Internet of Things, wearables, AI, and extended reality [21] to improve patient care and promote efficiency [10], specialized use-cases emerge—particularly in Disorders of Consciousness treatment. Along with the adoption of Smart Hospitals, Technological and Scientific advancement is occurring in Disorders of Consciousness (DoC) in areas such as diagnosis and prognosis with BrainComputer Interfaces, modeling, and neuroimaging [7,17,22,23,25]; Some are also exploring assistive robotic technology for DoC treatment [13] and the use of diagnostic biomarkers [9]. Designing technology for patients experiencing DoC, however, is littered with ethical and technical challenges such as the inherit heterogeneity in this population [35], legal and ethical issues with consent [25], and the open questions and controversies about consciousness in general [36]. With that being said, previous advancements in hospital automation such as automatic bed turn reminders [30], real time location services and process tracking [6,33], and other use cases of Internet-of-Things based technology in a hospital suggest that pushing towards the automation of Sensory Stimulation (SS) aspects may improve efficiency for hospital workers and increase the standard of care for DoC patients [21]. To navigate these challenges, we explore the development of technology through a different lens than most recent literature surrounding DoC technology, taking a humancentered approach to define the use-case, value, and subsequent design considerations for a system which assists hospital employees by automating aspects of their current 2 workflows surrounding Sensory Stimulation in a smart patient room. Our investigation is outlined by the following questions: RQ1) What aspects of Sensory Stimulation do stakeholders think would and would not be useful to automate or supplement technology? RQ2) How do relevant hospital employees currently use Smart Patient Rooms for DoC care? What design opportunities exist in this space? RQ3) How do stakeholders envision leveraging the data captured from patient rooms? What is the work involved in these opportunities and is it feasible? RQ4) What are the risks involved in creating an automated system and what steps can be taken to mitigate them? To address these questions, we conducted semi-structured interviews with providers who interact with Disorders of Consciousness protocols at the Craig H. Neilsen Rehabilitation Hospital, which features 75 Smart Patient Rooms. We then used Thematic Analysis to analyze the data, ultimately developing themes to encapsulate the key stakeholder perspectives on technological innovation in the space. 3 BACKGROUND AND RELATED WORK The following section gives relevant background information on DoC, SS, and Smart Patient Rooms (SPRs), and a brief overview of recent technological advancements in the field. We use this background information to ground our research questions. DOC OVERVIEW Disorders of Consciousness is a spectrum of varying levels of arousal and awareness that may result from a traumatic or nontraumatic brain injury. Patients with DoC exhibit different characteristics depending on their current state. Patients can be in comatose state, or coma, with the complete absence of arousal and awareness, vegetative state (VS) or recently renamed Unresponsive Wakefulness Syndrome (UWS), exhibiting periods of arousal without awareness of themselves or their surroundings, or Minimally Conscious State (MCS), with arousal and reproducible but inconsistent awareness [14,20]. Coma is an acute condition lasting up to one month [17], at which point patients transition into a prolonged DoC, typically entering into UWS or MCS, which can be either permanent or transient [17,18]. There are over 300,000 confirmed cases of UWS and MCS together annually in the United States in Inpatient Rehabilitation Facilities (IRF), with about 30%-40% of these patients partially recovering to the point of being able to consistently follow commands and communicate, and at least 20% further recover towards independence within five years [20]. Restoring consciousness in patients experiencing DoC therefore remains a tremendous clinical and scientific challenge. In fact, there is only one treatment that has obtained sufficient efficacy from randomized placebo-controlled trials in promoting 4 recovery to be recommended by clinical DoC guidelines from the American Academy of Neurology: pharmacological treatment with Amantadine [14,18]. THERAPEUTIC INTERVENTION SS entails stimulating the senses whether tactile, visual, auditory, olfactory, or gustatory to ultimately evaluate and restore cognitive function [29]. SS is primarily built on the hypothesis that it can promote brain plasticity, the capacity of a mature brain to reorganize itself to compensate for affected regions, and the evidence that sensory deprivation can slow recovery [8,14,27]. The efficacy of SS has not been defined by well-controlled trials, but several studies have shown promising results from SS in promoting responses that are indicative of arousal and selective attention [13,26]. While administering SS, there has also been promising results while using personally relevant stimuli in the process [26,27,29]. Contributing to the lack of well-controlled trials is the creation of a control group. Denying treatment to this vulnerable population faces tremendous ethical issues and is therefore often unacceptable to caregivers [25]. With that being said, SS has been practiced in hospitals for decades [15,20,35] and is currently practiced at the Nielsen Rehabilitation Hospital (NRH) where we collected our data. With the lack of well-controlled trials and definitive guidelines, it is unclear what might be beneficial to automate and what risks an automated system may have. Therefore, to understand the perspectives of people who interact with these protocols, we ask RQ1: What aspects of Sensory Stimulation do stakeholders think would and would not be useful to automate or supplement technology? Furthermore, it is unclear what risks an automated system may have, so we ask RQ4: What are the risks involved in creating an automated system and what steps can be taken to mitigate them? 5 DOC DIAGNOSIS AND PROGNOSIS The current gold standard method of DoC diagnosis, prognosis, and evaluation is behavioral observation [19]. However, behavioral observation has inherent issues as it often results in misdiagnosis [1]. Misdiagnosis rates of Locked-In Syndrome, which describes patients who remain awake and conscious without motoric output [14], for UWS and MCS consistently reach 40% and misdiagnosing between UWS and MCS at 41% [25]. Proper diagnosis is incredibly important for not just treatment decisions, but also to inform the family members of the level of consciousness and function the patient is experiencing [18]. Therefore, quantitative measurement of consciousness is a goal in recent literature. Gathering and operating on neurological data using Brain-Computer Interfaces (BCI) for DoC diagnosis has gained much traction [7,17,22,25]. Other work investigates the use of vital signs and facial expression, showing that vital signs are not generally reliable to indicate of the level of consciousness, but body temperature and facial flushing specifically may be [31]. Some work examines blood biomarkers and cerebrospinal fluid to support diagnosis [2,39]. Recent work also explores and discusses utilizing the advancements in Extended Reality namely augmented and virtual reality paired with EEG BCIs, SS, and behavioral assessment. However, progress here is hindered by the problems with EEG BCIs described above [34], the fact that most DoC patients lack oculomotor control [17], and sample sizes remain small [24,32,38]. While much work is being done to model consciousness and DoC diagnoses [7,17,22,24,25,31,32,34,38], to the best of the author's knowledge, modelling effective SS is an unsolved problem. The SPRs at NRH contain a variety of sensors that collect 6 biophysical data on the patients and a camera in each room that is used for monitoring high risk patients [10]. Understanding the available data, what the specialists believe to be the most feasible and helpful data, and how they would like to interact with this data lies at the heart of the current study. Therefore, we investigate RQ3: How do stakeholders envision leveraging the data captured from patient rooms? What is the work involved in these opportunities and is it feasible? INPATIENT REHABILITATION FACILITIES During the acute stage of DoC, which is operationally defined as the first 28 days after sustaining the incident, patients are treated in the emergency department and intensive care unit [14]. Then, once medical stability is achieved, the patient ideally enters an inpatient rehabilitation facility where a specialized team can administer optimal treatment and diagnoses [20]. However, this is often not the case. In fact, admission to acute inpatient rehabilitation facilities is infrequent for DoC patients due to admission policies and cost-control measures despite the evidence that recovery is highly contingent on high-quality specialized treatment [20]. The University of Utah Health Craig H. Neilsen Rehabilitation Hospital (NRH), where we conducted this study, is a public US university hospital that opened in 2020 which offers specialized DoC treatment [40]. NRH operates as a typical rehabilitation hospital; however, all 75 patient rooms incorporate smart home technology, containing a smart TV and soundbar, with cable, Apple TV, AirPlay, and bedside HDMI input for entertainment options. Smart lights, blinds, and a thermostat allow for adjusting the patient room environment. 7 Previous work [10–12], has explored the effects of smart home technology deployed in a hospital setting on all stakeholders involved. From this work, it is clear that SPRs offer opportunities to support healing and therapy, but it requires a massive usercentered research effort to properly support all stakeholders involved. We aim to further the understanding of how hospital employees can leverage the technology deployed in the room to support this healing and therapy by investigating a specific use case of the SPR by examining RQ2: How do relevant hospital employees currently use Smart Patient Rooms for DoC care? What design opportunities exist in this space? 9 (2) Able to tolerate a 30 to 60-minute engagement for an interview session (3) Able to communicate in English (4) Capable of making informed consent We obtained written consent from each participant before interviews. The interviews were recorded, but only the separated audio and resulting anonymized transcript was stored and used for analysis. The Utah Institutional Review Board approved this procedure. DATA COLLECTION The interviews were conducted and recorded on Microsoft Teams. Following the interviews, the audio was separated with Adobe Premiere Pro, saved, and the video files were destroyed. The auto-generated transcript was downloaded from Teams, and uploaded to Dovetail alongside the audio file where the analysis took place. Important passages in the transcripts were corrected if a mistake was found by listening to the audio. The interviews lasted between 47 and 74 minutes with an average of 58 minutes and 31 seconds. One researcher conducted the interview while one or two others observed, took notes, and asked follow-up questions if they deemed it necessary, primarily to capture nuances or resolve confusion. DATA ANALYSIS After interviewing each participant who wished to participate, we used Dovetail for data analysis using thematic analysis [4,5]. The interviews were listened to and read, and important passages were highlighted and given a code. The first interview was partially coded collaboratively, with the rest of the tags being almost exclusively coded 10 by the lead researcher. For particularly ambiguous or otherwise confusing quotes, the research team discussed and coded collaboratively to ensure accurate interpretation. Some content from the interviews was not coded if the scope of the discussion was not focused on DoC specifically or indirectly. After each interview was coded, three to five quotes were grouped into low level themes, with some outliers due to the intricately detailed nature of DoC and the specialties of the providers that were interviewed. From there, the titles of the low-level themes are used as the lowest level of abstraction and iteratively grouped into mid-level themes, which consisted of two to nine low-level themes. Finally, groups of four to six mid-level themes were used to generate five sharable, high-level themes encapsulating both the overarching ideas and nuanced provider perspectives. 11 RESULTS Through our analysis of the data, we generated five high-level themes which encapsulate the challenges currently faced with DoC workflows, as well as the providers’ perspectives and considerations regarding the facilitation of technology-based improvements. INCREASING THE FREQUENCY OF SS To develop a shared understanding to build on, we asked each provider about SS in general, how they administer it, how they determine the optimal frequency, and what changes they would like to see in how it is administered at NRH. We found that the administration of SS at NRH differs based on the patient’s abilities as well as the provider’s role. In-room stimulation (IRS) is typically carried out by the nursing team and aides. It entails stimulating the patient in the SPR either in bed or in a chair if their abilities allow for it, using a mixture of modalities, such as music, slideshows, and environmental modification such as opening the blinds and adjusting the lighting. IRS is distinguished from therapy sessions, which utilize SS techniques, among other domainspecific strategies, to facilitate interactions according to each therapy discipline. Providers generally expressed a desire for more frequent SS, and specifically for increased IRS, built on the possible benefit to the patient’s recovery based on the general motivation for SS and the idea that increasing the frequency of stimulation can increase the likelihood of observing desired behaviors. so having it [SS] throughout the day is just more likely to catch these behaviors too. (P2) 12 Similarly, P3 emphasized that more frequent IRS could maximize the utility of therapy sessions. A patient's whole 24 hour day. I'm with them for an hour and a half, so in the full scheme of what input their brain is receiving, it's very little. So what would be so helpful for my workflow, as if some of this was going on without me standing in the room, right? So if I had the ability. To start the patients day off with some meaningful sense of sensory stimulation an hour or two before my session. So they're kind of like primed and ready for my session. And then that makes my stuff more effective. (P3) Despite these aspirations, Increasing the frequency of SS in general—and IRS in particular—entails significant logistical, technical, and emotional complexities. SS RISK CONSIDERATIONS We directly asked about the risks involved with SS. Providers explained that there are generally minimal risks to the patient involved with sensory stimulation, yet there are still risks. When directly asked about the risks involved with SS, P5 highlights that while the in-room stimulation that occurs while the patient is in bed has near-zero risk to the provider or the patient, there are indirect risks inherent. So I guess that that depends on what you mean, like by a risk like-I don't think that the sensory stimulation can necessarily, in most cases, hurt them. I don't know how a slideshow is gonna hurt a patient. Unless they get, you know, they start to wake up and they 13 get agitated and they fall out of their chair. Which, but that-that's kind of an extreme scenario. (P5) With minimal risk in mind, there is a strong desire to deploy as much stimulation as possible. However, there are important considerations to ensure that the patient can plausibly engage with the stimulation and to avoid overstimulating the patient to the point of agitation. P6 describes how fatigue can decrease the likelihood of observing the behaviors indicative of awareness. Or often I will see with DoC like in the morning time when they rest it when their brain is like fully fully rested then they can like generate like “Hi” or an “OK”, but once their brain is, their brain is so tired, that speech portion is like the first thing to go, like verbalization. (P6) P5 emphasized that excessive stimulation can provoke agitation, ultimately working against therapeutic goals. While this is a known risk, participants acknowledged that guidelines such as those from Medicare—as well as best practices for monitoring biometric data—help mitigate it. So the frequency we still have to we still have to. We still strive to meet the Medicare guideline of like three to four hours and some type of therapy, right? But. It's also kind of like how much the patient can tolerate. […] The most common way that we assess their tolerance are like vital signs, right? So is their heart beating too fast? Or is their blood pressure go up? Do they? Do they start 14 to show signs of like psychomotor agitation? Like that's those are clinical indicators that it's time to back off. (P5) In addition to the guidelines and indicators, P1 highlighted the importance of proper training and supervision taking place to adequately adhere to these principles. So with the risk, you know, it's really important for people to be appropriately trained. And so that's why like [name omitted] has been identified as one of the leads for being able to do this because he does have a lot of trainees that he's able to integrate, but he provides that supervision level. (P1) To our surprise, when asked about the risks involved with SS, many risks discussed were not patient-centered, but provider-centered. For example, the interviews showed that SS can induce physical harm to providers as the patients become more alert, though this is a rare occurrence. Very rarely. [Yeah] Like I said, I, I had been doing that for weeks and then finally he was alert enough to like open his eyes and go try to hit me and he was really very deconditioned. Like he couldn't really hurt me, but he was gonna try. So, you know, it's possible but very rare. Yeah. (P2) Furthermore, while not a direct risk, providers can get emotionally invested in the heavily personalized treatment, which can be difficult. P1 sheds light on this aspect. that's another thing that people don't typically consider is the distress that it can like implicate the individual that's trying to do the stimulation because this patient population, you know, it's can 15 be quite a lot to process for individuals within that are practitioners. (P1) PERSONNEL AND LOGISTICAL BARRIERS TO OPTIMAL SS FREQUENCY The providers explained that the main obstacle towards administering meaningful IRS as frequently as wanted is the lack of available personnel. They elaborated further, explaining that high nursing and aid turnover contributes heavily to the challenges faced. When asked about the barriers to more frequent SS, P6 described the friction that arises with high turnover. So, like I always think the barrier for us is always nursing because nursing staff changes so much and we're just on such different schedules with nursing and therapy and stuff, so. The aids are not consistent, and the nurses are not consistent, but the therapy staff is like we try to have the continuity of care from a therapy staff like standpoint. So that is hard to carry out that like that sensory plan that I'm talking about because new aids come in and they just they just don't know. They're not familiar with DoC. (P6) Educating new providers on DoC is challenging because the condition is relatively uncommon, and thus staff are not likely to have been trained on these treatment protocols. Furthermore, this treatment often doesn't result in observable patient engagement, which can result in discouragement and inconsistences in practice. P4 explained the discouraging aspects of the often-limited visual results of SS: 16 Umm but yeah, I think there's just been a lot of education with [nursing and aide] staff because it doesn't feel that anything's happening in the moment. (P4) Beyond staffing concerns and the all-encompassing views of SS at NRH, we also investigated the difficulty of their personal workflows. Some providers explained that due to the infrequency of DoC admissions, caring for these patients can be perceived as additional workload, which becomes increasingly salient as the number of DoC patients admitted to NRH increases, We don't have these patients very often. So that's one challenge is we kind of, it's not like a normal part of our day and then we have one, it's suddenly all this extra work basically that we have to do and we're not really in the habit of doing. (P2) We also found that some stimulation modalities are easier to administer than others. For example, maximizing the amount of vestibular and proprioceptive stimulation by getting the patient out of bed is a primary goal at NRH, as this can effectively simulate aspects of their life before their injury beyond digital media, creating a meaningful experience. So that's a big one is getting, keeping them out of bed more ideally, they would be out of bed all day. Like what better way to have some kind of sensory stimulation than to not be lying in bed. Like you have for the last six weeks, like to be upright in a chair facing the window. (P2) 17 However, this provider also explained that stimulating the patient outside of their bed in a chair can be logistically difficult for the providers and physically difficult for the patients due to fall-risk during the transitions, and unanticipated bowel movements, Because those are pretty hard to, hard to do. Everyone has to be in the same place, the right place at the right time and the patient has to be ready, you know, like you can spend 30 minutes getting out of bed and then they have a massive bowel movement and it's all over. (P2) While dynamic and physical stimulation techniques—such as in-chair stimulation—is clearly logistically complex, providers also conveyed that the more accessible forms of SS, such as IRS with digital materials (e.g., music and slideshows), suffers from similar issues, ultimately compromising the continuity and quality of treatment these patients receive. But again, the-the problem is it all has to be activated by a very busy nurse who's taking care of six other patients. So, it-it-it's been, it's been put in place, it’s been effectuated very inconsistently. To where we in some cases you could anecdotally say, yeah, that's probably helping. But most of the time it's like, well, we hope it is 'cause we're doing it the best we can, but it's not consistent. (P5) The tension between increased SS and the challenges inherent in its administration—including staffing shortages and turnover, training barriers, and inconsistencies—directly led to many of the providers expressing interest in integrating 18 technology-based solutions. These ideas, along with the subsequent hypothetical risks and considerations they raise, are discussed in depth in the following section. OPPORTUNITIES AND PRECAUTIONS FOR AUTOMATED SS Building on the challenges involved with providing optimal SS, we asked each provider about how they use technology, with a particular focus on smart home technology incorporated into the patient rooms. After establishing this foundation, we investigated their perspectives, concerns, and ideas for incorporating automation into SS. P5 describes that incorporating automation into IRS is desirable, going as far as to say that the lack of automation is detrimental: So there's the ideal, something I call sunup to sundown. Right, where you have your standard therapy disciplines of physical, occupational, speech therapy. But in the-in the interim, in between those sessions, there are things you can do with a room or with additional staff members that could provide that sensory stimulation: turning on music, showing a slideshow of pictures, putting a taste on someone's tongue, wafting a smell under their nose. Like, and that's something that we have tried to incorporate, but due to kinda human error and it's just it's not as tight as I would like it. But that that's what we try and do, and we try and incorporate. But due to the- the fact that it's not automated it, it becomes problematic in standardizing across all patients. (P5) 19 WHAT MODALITIES SHOULD AND SHOULD NOT BE AUTOMATED This line of questioning led to specific ideas regarding what each provider deemed to be the best—and least—suited for technological innovation particularly. Many ideas discussed have great potential, but the technological and logistical complexity differs greatly in both implementation and potential use-case. For instance, the primary use of technology, outside typical hospital sensors and devices, is using the iPad in the patient rooms to play music and mirror slideshows onto the television. I'm using the music a lot. There's the sound bar on each of the TV's, or just playing directly from like the bedside iPad. Kind of closer to their face, or putting on some like, you know, YouTube via the Apple TV in their room. (P3) This leads to a natural extension to streamline the process of IRS with digital materials in an automated fashion. The providers had different perspectives about the extent to which automation should be applied, though all wanted some way to streamline the process of IRS. I'm totally open to whatever the solution is. But, but some way to get those personal meaningful things in that patient's room. And so as a nurse with almost no training, could just push a button and get it started basically. (P2) On the other hand, it is less obvious how smart home technology—or technology in general—could be applied to proprioceptive, vestibular, or otherwise dynamic and physical stimulation. While some ideas that may indirectly benefit these modalities are described in the next section, providers at NRH currently don’t leverage technology for 20 these modalities and highlight the reasoning as well as the limitations of technology in future applications. When asked if there were any aspects of SS that they would like to incorporate that aren’t already incorporated at NRH, P4 highlighted these limitations. Yeah, um, through smart home technology, if we could get a lot of these things automated, it would definitely help our-the-the workflow and the patient to ensure that these things are done exactly when they need to be. So, a lot of this, I feel like it could be automated, of course, except the shower. But, umm, maybe pictures can start showing or music can start playing at certain times that it needs to be done. (P4) With the technological limitations in mind, providers did express interest in using technology to aid in physical stimulation, particularly in tactile settings. P7 explains that vibrational stimulation is incredibly arousing, and has some ideas to incorporate technology to streamline the process: Yeah, like if their bed was like a massage chair that could turn on and like, start like moving their muscles around. […] it's super arousing like you like. You put vibration on your on your feet, especially if you're ticklish like you're like jumping out of your skin. (P3) P7 sheds light on this aspect as well with another application. Maybe having a weighted or heated or cooled blanket that activates to try to incorporate those senses as well. (P7) 21 Following these device-centered innovation ideas, we asked providers follow-up questions regarding what sort of extra precautions and risks may arise. We found that if physical technology is used on the patient for tactile stimulation, there are concerns about agitation and skin wounds. It's a cool-it's a cool thing that, you know, I do theorize about. Of course, there's like a lot of like lines and medical, you know, just like wounds and pain and how to manage all of that is very complicated, but would definitely be a an interesting application of the technology for sure. (P3) Providers also offered insight into the stark technological limitations and practical complexities involved in streamlining olfactory and gustatory stimulation. While providers did not propose any technological solutions for gustatory stimulation, they did suggest several for olfactory. Currently, providers target this sense by applying products such as colognes, perfumes, and soaps, or presenting foods and drinks that have been identified as meaningful to the patient. When asked to retrospectively walk us through how personally relevant stimulation materials were used, P2 explained the process. We found out some foods or smells he likes. So we like we got his, he liked mint. His mom brought in like an infuser and we would play, put like some just like mint infusion because he kind of liked those smells. (P2) While the implementation details are not immediately obvious, providers expressed interest in using technology to make the olfactory stimulation process easier. 22 When asked a follow up question regarding the ideal stimulation sequence triggering, such as pushing a button and having a stimulation sequence play out, P4 said, Yeah, playing music. Let's see, pictures, of the-the sense-or the-if there's something that we can have for their sense of smell, if that would be possible. I don't know if it would be, but that would be my other ask. (P4) However, in practice, there are complications with hospital regulations, agitating smells, and potential allergies, creating a significant implementation barrier. Others highlighted the possibility of pungent aromas becoming agitating, while P1 focused on more logistical concerns. but I believe that it's still against hospital policy to have anything for olfactory. Because patients are allergic, staff members can be allergic. What have you. So I think looking into the regulations pertaining to any type of olfactory stimulation is gonna be critical for moving forward to thinking about that piece.(P1) While many ideas were discussed to automate aspects of SS across modalities, providers ultimately saw automating aspects of digital IRS—particularly the ability to program a sequence of music and slideshows—appears to be the most feasible, attractive, and potentially impactful area for technological development. P5 summarized this concisely, saying, “That’s the initial big step, is turning the room into the therapist that we can program.” 23 CONCERNS ARISE FROM TECHNOLOGICAL INNOVATION While the providers think that technology has potential to enhance the patient experience and ease burden on themselves and their colleagues, they have different perspectives on what the role of technology in DoC care is and should be, and each provider explained complications that arise when technology in general and automation in particular are introduced. Firstly, the providers explained that technological solutions have unpredictable outcomes and learning curves and these factors must be accounted for to ensure both enhanced patient care and streamlined provider workflows. When asked about possible detriments of an automated system, P7 explains these issues. I think I just gotta kind of try and see, you know, things can look great on paper and then you put real people in the mix and it can get really weird and messy very quickly. (P7) While there is a general consensus that there will be inherent unforeseen detriments in an automated system, providers explain that there are certain steps that can be taken to mitigate potential risk. The general risk of overstimulating patients experiencing DoC in section 4.1 applies here as well. Therefore, if SS is automatically administered, there must be safeguards in place to make sure that the patient is not overstimulated to a point of agitation. Providers further define these safeguards in settings where a provider is present and not. If a provider is present while the stimulation is occurring, they need the ability to halt the process. Hmm. Yeah, go do it. I think you'd have to be able to shut it off pretty quick and easily. Because I want to see signs of arousal. But I don't wanna see signs of agitation. (P3) 24 If a provider is not present, there were ideas about how to quantitatively model agitation with biometric thresholds revolving around perspiration, movement, and respiration, such that a system could latch on to the occurrence and alert a provider or automatically dampen the stimulation. P5 explained these concepts when asked how flexible or dynamic an automated SS system would have to be. Umm. I I think the flexibility has to. So just using an example here, going back to a slide show, right, if images are displayed on a slide show that proved to be some type agitating. I don't. I don't if there is a mechanism by which staffing can be alerted or the room can be shut down or the whatever it be manually or automatically terminated. Then that would suffice. That would be fine. And. How to do that I I don't know, I just. See that? The patient's getting a little more agitated just to alert the bedside nurse, and they can go in and turn it off. That might be the easiest thing, right? (P5) P5 goes on to state that a clinical filter is often necessary for these patients. But you also have to remember well, and I guess another point is. Biometric data. Such as heart rate could also mean something like excitement too. Doesn't have to mean agitation. So. The biometric data itself is helpful, but in this patient population it probably has to pass through. A-A clinical filter like. Is this patient enjoying this? Is this patient distressed? (P5) Following the direct risks to workflows and patients, the providers explained that DoC treatment in general and sensory stimulation in particular, cannot and should not be 25 100% automated. P2 and P6 both explained, while noting that this is not well-studied, that the human element involved in sensory stimulation is perceived as an incredibly important aspect of DoC treatment. I think there like could be. You know, like setting up the room if you know the TV or something was on a schedule. And I'm sure like, that stuff could be automated. In a sense, but I also think like so much of what we like, for people who have-are disorders of consciousness like, they need that they need, but like they benefit so much from, like, humanness of it all. You know, the more like kind of automated or techy, it is, to me, it feels a little bit less natural and I definitely don't think there's enough research on, so I shouldn't, you know, like I can't say for certain. This is just anecdotally. That, like I would think there would be like a little bit better response when it's like a human providing the stim versus a robot type of thing you know. (P6) The providers explained that automating aspects of SS contains risks, though manageable, and may inherently demonstrate less efficacy due to the robotic nature. With that in mind, the providers generally think that leveraging the smart home technology in the patient rooms can help deliver high quality treatment to patients experiencing DoC by facilitating optimal therapeutic interventions. P2 explains this relationship when asked if they think any aspects of their workflow can be automated, saying. “I think with this situation it’s, it’s all just to help the person-to-person interaction.” P5 had similar 26 thoughts when asked a follow-up question about the human-element, saying, “This is more of a partnership than the room taking over.” We also investigated the details of how the providers would like to interact with this automated tooling. We found that the providers like the idea of a centralized system to schedule and automate stimulation on the iPad in the patient rooms. A subprocess involved in this strategy, which is explained in the next section in depth, is centralizing the personally relevant digital stimulation materials. PRM OPPORTUNITIES AND CHALLENGES While investigating the desire for streamlined SS, both in automated or manual settings, we found that another significant barrier lies in gathering and accessing the stimulation materials. Using PRM in SS has promising results [26,27,29] and at NRH, maximizing the meaning, collecting, and using these materials in SS is one of the primary objectives for DoC treatment. However, both families and providers face significant challenges that can impede frequent, maximally meaningful, and timely stimulation upon admission. Providers explain that in an ideal system, this subprocess will seamlessly integrate into an automated IRS system. In the interim, however, issues with gathering and accessing PRM in manual workflows may benefit from lightweight technological solutions, while still facilitating optimal treatment. This section explores the provider’s perspectives, ideas, and motivation to facilitate streamlined PRM collection and deployment with technology, as well as the logistical, emotional, and privacy-related challenges that arise. 27 COLLECTING AND DEPLOYING PRM The providers explained that DoC is a complex and nascent field of study with few controlled treatment trials and therefore limited treatment options for patients. They expanded on these topics, describing that despite the lack of research, intervention rooted in sensory stimulation remains the cornerstone of DoC care at NRH because it is the only option outside of pharmacology. Well, it's not. There's actually no strong evidence in the literature that-There's no strong evidence in the literature that supports, uum, the use of it. It's all anecdotal. And it's experiential. So it's it essentially if if you're looking at it from a pure research perspective, it's. It it doesn't hold weight but it's, but it's all we have really. (P5) Providers built on this idea, explaining that patients experiencing DoC are hospitalized for a very long time compared to other patient populations leading to habituation and desensitization. Combating these aspects by appealing to the patient’s memories and emotions is therefore of the utmost importance with maximally meaningful, sensory stimulation. Oftentimes after their coma, it's already been many weeks. So they're totally desensitized to alarms, you know, that they hear in the hospital to being rolled and turned and changed and cleaned. Like all those regular nursing cares are totally there's no sensory response, right? Like when we look at their, when we study their, their brains, like there's no, they don't react to those things 28 because they've just become yeah immune to those. So we try to find, we're trying to create things that are meaningful and have some kind of familiar routines and rhythms to, to access those brain areas (P2) While explaining this process, some offered powerful anecdotes of evoking behaviors indicative of awareness while utilizing maximally meaningful stimuli. But if we, sometimes they look like that but then we'll notice like, oh when they played, when I played this song, they started crying or they mostly their eyes just like wander . But then I pulled out a picture of their wedding and they focused on it and they looked at that photo. So then we know that they're recognizing something, something is like coming through their senses, they're remembering or thinking, understanding and some kind of purposeful, meaningful response is coming out.(P2) To understand what opportunities to maximize meaningful SS a given patient has, there is a large collaborative effort with the patient’s family. Upon admission to the DoC program at NRH, an intake form is distributed to the patient’s loved ones to learn what their likes and dislikes are with the goal of creating a maximally personalized treatment plan. At admission, there is a-The equivalent of essentially an intake form that characterizes the patient's who they were before their injury, their likes or dislikes, music they listen to, foods they enjoyed, cologne or perfume they wore, their education level. Just 29 pretty much a 360 degree kind of assessment who they were and that's obviously done by a loved one upon admission. (P5) Providers further explained that the intake form is used to inform all aspects of SS, showing that ‘personally relevant stimulation materials’ encompasses more than just materials. P6—among others—offers an example of personally relevant stimulation outside the digital domain, highlighting the importance of using an all-encompassing stimulation protocol. In this case, it was explained that modifying the patient’s environment can be a meaningful form of SS. I think the sensory stim comes a lot from that sensory profile, like, um, if someone is, you know if this person like works outside all day, like trying to get like out like them outside during the day. You know, when we're done with therapy, like having the AIDS be like, OK, he has like 2 hours of outside time 'cause he usually is, you know, works outside or something like that. So trying to yeah match what is typical for them, in the hospital environment. (P6) Obtaining this information as early as possible upon admission is one of the primary goals, though it is not always an efficient process. P2 explains through a retrospective account that when the patient’s loved ones are not in-person, logistical issues arise. That's another thing we need to make easier because if they're in person and we have them right there and we have them fill it out. That's great. But with her being in another state emailing this form 30 and her like trying to fill it out and send it back. Like I said, that took like two weeks to just even get the information about him. (P2) Along with the intake form, the patient’s loved ones are also tasked with distributing photos and music to be used directly in IRS. While there is a section on the intake form for the patient’s loved ones to list specific songs and music the patient likes, the process of collecting photos is less formalized than the intake form, leading to frustration. While collecting the completed intake form is primarily difficult when the patient’s loved ones are not in-person, the process of distributing digital photos remains constant regardless of the patient’s loved ones’ physical proximity, and thus the issues are prevalent within many patients. P3 emphasizes and explains this difficulty, saying, “It seems like it would be so easy, but it’s so unbelievably hard.” Providers explained that they have tried to incorporate some solutions to centralize the process, but due to technical difficulties, they use personal email accounts and phone numbers to get these digital photos on to the iPad in the patient rooms, which itself is not difficulty prone. It was really, really hard because then we had to, well, we created a DoC Google account and then we sent, had her try to send the photos to that, that Google Drive account, but then it didn't work. So then I had to send them to me and then I sent them to the Google Drive and then we logged into the Google Drive on the ipad and then we mirrored with the TV and then we started the slide show and then it just wouldn't even work sometimes. I have no idea why. It just, nothing would even show up on the TV. Or 31 half the time. It just didn't even work. So, it was like all that work and then it still didn't work. So it's frustrating. (P2) P2 also explains that using personal accounts is highly undesirable. There's gotta be a better way to do that than what we've been doing, which is just sometimes just here's my number, send them to me. I don't love that. That's kind of weird. Now, I have all these my patient's family in my phone. Like that's not ideal. But it’s just literally the only way is to send it to my email or whatever (P2) Difficulties that the patient’s family faces during PRM distribution were also highlighted. Alongside distributing the photos, the family is somewhat tasked with sifting through them to ensure that the patient does not encounter emotionally triggering photos and videos that may have some relationship to the injury-inducing accident or anything else that may be emotionally disturbing. P3 explains the possible emotional impact of this process on the family members. They're strapped, financially, emotionally, like all of it. So it's really hard like say, like your son is in the hospital and then, like, you're asking for photos and then all week people are just sending you a million photos of them. Like, that's really emotionally distressing. (P3) After sorting, they then must distribute these materials back to the care team. This process relies heavily on their technological skills, which may lead to a delay in the stimulation pipeline. 32 depending on the family member, right. Like this, young girl's fiance was like, super tech savvy and he figured it out right away. But then this other the the boy who we watched World Cup with, his mother was in Mexico, and I was having her, I think email like she was taking pictures of family photos to email to Tyler and I that then we then like email to another email to put on the iPad because we don't wanna use our personal stuff but like it's was really hard really hard. (P3) Further family-facing logistical difficulties arise. Knowing the nuanced details about what someone likes and dislikes is a challenge, even between family members, and having access to their specific tastes and preferences around music and photos that may be behind a privacy wall is uncommon. When asked a follow-up question about exactly how music is played with the iPad, P3 explained these nuances. And that's if I have access to their Spotify, which is actually kind of rare. I'd say what I do the most is I just see if friends or family can give me any artist name, which is harder and harder these days. Like all of our music is so just like eclectic and we'll have playlists and stuff that we just listen to. Right? Random, but if you were to like, say about your loved one, could you pick like 3 bands or artists that they honestly like is kind of hard.(P3) Family members can also serve as PRM themselves as the emotional connection between family members can be an intensely meaningful modality. Ideally, this is utilized 33 by either face-timing if they are remotely located, or interacting with the patient in person. Whether in-person or not, P3 explains the complexity of this method. It's really tough when they don't have family, either locally or who who can come in because of, you know, very understandably, like you have a job, you have life. It's been months of this. (P3) Once obtained from the family, accessing the materials presents similar challenges. Sometimes technical difficulties hinder the process as explained by P2: Ok, so we need, we got his mom, we emailed her, she eventually emailed us back with some pictures and a list of songs he likes and I made a playlist on youtube and I hooked up the, the play the playlist to, with the room ipad to the TV. But it wouldn't always mirror correctly. And it seemed like the easiest way to transfer was Google Photos. But actually Apple Photos because it's an Apple TV, would have worked better. But she had, she didn't have an Apple account and all these, you know, so we're transferring between things and so it was just basically two weeks and a lot of hours just to get a music playlist and a slide show set up in his room. (P2) POSSIBLE TECHNOLOGICAL SOLUTIONS When it’s time to deploy digital stimulation materials, the nursing team gets instructions from therapy on what stimulation materials to use for IRS, as well as where to locate them, but due to the technical difficulties described above, there is inconsistency which sometimes leads to difficulty. Therefore, there is a desire to use technology to 34 centralize both PRM collection and deployment with the hope of streamlining the processes. These ideas help to define automated IRS but also may help streamline manual processes in the interim. Specifically, following from the general goal and subsequent challenges of getting the music, photos, and videos onto the iPad, providers expressed interest in having the capability to handle the collection and deployment of materials completely on the iPad. P2, P4, and P7 each defined this even further, explicitly mentioning that the ideal workflow for deploying PRM is the touch of a button. P4 highlights this desire. Really any of them are pretty simple, but if we could even simplify it further with just a touch of a button rather than having to like go in and find the information, I think that's the hardest part. (P4) While there is general interest in centralizing these processes on the iPad, noteworthy privacy and security concerns arise. Similar to the access issues discussed in section 4.2, there are concerns and open questions as to who exactly should have access to the controls and media. I mean, it's, it's definitely, you know, PHI. like it's gonna have their photos and their history and memories and audio and personal things like that. And it's gonna be like a, a health care intervention that says this is how you can help the patient be more awake, do these things. So, I mean, definitely like, but for anyone who's in their room, I don't know, I guess, yeah, we would just need to have some kind of, I guess if they had random visitors, we might not want any random person who comes in their room to be able to 35 access all that. That's true. So, yeah, maybe it needs some kind of log in or credential or something. (P2) Aside from streamlining IRS administration, some providers offered other ideas to maximize the meaning and efficacy of IRS with technology while decreasing burden on family members. The interviews showed that when facetime or in-person familial interaction is not possible or to increase the frequency of this modality, providers complement or substitute it with technology in the form of slideshows of familymembers. One provider in particular showed interest in taking it a step farther by creating and deploying deepfakes of loved ones to the patient, justifying it further with the potential decrease in familial effort. Yeah. Yeah. No, that would be very interesting if, like if I could like. Program mom in the video to say the commands for my test would be very interesting because it's so hard when somebody else besides a trained person is doing this, cause again, they're offering. They're like doing that. They're giving too much, so it would be very interesting if I could be like, yeah, it could be like their moms voice giving that exact prompt in the way that I can control it and see if there's a different response there could be, could be pretty interesting. (P3) As mentioned above, another area that can be challenging for the family is the challenging judgement calls about whether or not to use certain stimulation materials that may be emotionally distressing for the patient. Providers suggest that family is necessary for these calls. P5 explains that this could also be partially automated to reduce burden. 36 The type of the intent of this stimulation is right positive stimulation, right. We're not showing the car accidents or slashing movies or, you know, things of that nature. It's. It and that therein lies kind of the individualization, and this is what you can automate is the judgment that comes from, well, what do we incorporate into these stimulation protocols? Like how do we personalize this in an effective-in an effective way? (P5) DATA COLLECTION CHALLENGES AND OPPORTUNITIES With the goal of building automated tooling in mind, we asked the providers many questions regarding current data utilization and challenges, what additional data may be useful in the future, as well as how they would like to interact with this data. We found that enhanced data collection may help facilitate further cross-team communication and could help cover gaps in provider’s attention—such as at night, during treatment sessions, or when family is interacting with the patient. This may support many areas of DoC care such as intervention scheduling and validating experiences, ultimately resulting in scientific advancement. Providers offered insightful descriptions of the importance of proper decision making. They explained that patients with DoC must have a positive prognosis to get selected for rehab and maintain a positive prognosis to continue rehab. If they are not identified for rehab, they typically enter a long-term facility, where therapeutic intervention that is offered in an Inpatient Rehabilitation Facility like NRH is not 37 delivered to the same extent. P6 explains this through an account of a previous position at a long-term facility. Some patients, like I can think of maybe two or three, but I wasn't following any sort of protocol and I wasn't working as collaboratively with PTS and OTS as I do here. I definitely remember using some sensory. Input and like trying to arouse people 'cause they were so. Unconscious, but I definitely didn't like follow a protocol or it was kind of more nonsensical and like guessing because we didn't really know a lot about it. (P6) This becomes complicated by signs of consciousness having been historically missed, leading to many patients being misdiagnosed and not receiving proper care. So it's just this whole Pandora's box of like, oh my gosh, all these people that we thought couldn't do anything. Maybe we just didn't know how to assess them. We didn't know how to access their abilities. We've been sending all these people, you know, to hospice that maybe we should have sent to rehab. So it's been a whole upheaval, you know, but then all the details of like all the care that that person requires and it's a really ethical conversation, of course. So it's a whole new complicated field. (P2) Given these challenges, the remainder of this section explores provider perspectives on improving documentation and enhancing observation capabilities that may facilitate more data-driven treatment decisions. 38 DOCUMENTATION CHALLENGES The interviews showed that treatment decision making is based on both formal assessments and less robust, sometimes informally documented and communicated arousal and awareness events. The JFK Coma Recovery Scale-revised is the assessment that providers use to formally establish prognoses and diagnoses. In therapy sessions and IRS, specific desired behaviors are captured by different disciplines to inform their treatment plans. Providers elaborated that there is a wide range of actions and responses to stimuli that each specific discipline specifically tries to foster, and often these overlap, necessitating cross-team communication as highlighted by P2: So just talking about those details like, oh actually it's like some patients, the-their first word is in physical therapy and the first time they give a thumbs up or move their arm is in speech therapy. (P2) Despite its importance, cross-team communication is not seamless, mainly occurring during in-person meetings or through email. Contributing to this issue is the fact that DoC is a new and evolving field with less infrastructure in their electronic health records (EHR) systems. Yeah, most of it is in person and luckily, we-we have the same therapist working with these patients. So they'll touch base with nursing staff to get updates. There's not a great place for us to document epic currently, so we haven't been doing that. It's been more of um in person updates. We also have a team conference for these patients, umm, that they do a few times a week and so that's another good time to discuss your-give any updates on the 39 progress with the patient at that time when you have nursing, therapy, provider, everyone together. (P4) Along with cross-discipline collaboration, many behaviors that may indicate awareness are subtle and easy-to miss. This has been historically—and still can be—an extreme challenge as different patients may have different abil ities or lack thereof, and behaviors are inconsistent. […] and that's one of the biggest barriers is actually capturing these behaviorsbecause they can be so intermittent. So a lot of times, we don't get a -an accurate-an accurate assessment about their level of cognition is. (P5) ENHANCING, COLLECTING, AND VALIDATING DATA WITH TECHNOLOGY These issues lead to a desire for more robust charting and documentation. Providers had different ideas and specifications for exactly how this should be done. Some specify additions to the current EHR system they would like to see. Umm, if we were gonna collect data, preferably to have it added to epic, our charting system if possible. Umm, I think that would be the best way. (P4) Others focused more on settings where a provider is not present, thus wanting a more custom, automated solution to record these behaviors. It's like the music wasn't even on, but around 9:00. O'clock every day they open their eyes and look around for a few minutes, and they're just happens to not be a provider in the room at that time and we're all missing it, you know, so kind of that, like, 40 spontaneous. Signs of arousal that don't even have to do with the stimulation I'm providing would just be really interesting to know, because then the next day I would be there at 9:00 AM. Like, look at the fork like didn't trying to do all my assessments. So that would be really helpful. (P3) Some providers took this idea further, describing a solution that could integrate with the iPad in the patient rooms. I so ideally like pie in the sky where I-I call it room as the therapist. And what I mean by room as the therapist is not just something that is doing something to the patient, but actually being able to help us capture Cognitive improvements in levels of consciousness that then we can kind of latch onto. (P5) Providers offered different descriptions of behaviors and data points that they collect as well as future data they would like to capture and track. Eye-tracking, purposeful bodily movement, and vocalization were noted by providers as the main behaviors they use to determine a patient’s ability to engage with their environment. Some want to use technology to document and leverage these behaviors. Yeah. I would like to be able to collect. Eyes data on their eyes, like the pictures and the songs. Like, did they open their eyes and look at the target, or did their eyes just drift to their resting position where they always are to their eyes, even open at all, but like just watching eye gaze is a really big-ah-starting point. Umm, 41 I from there would be curious what other kind of sounds- um signs of arousal they show.(P3) Ideas about enhancing data-collection led some providers to a few different applications. Firstly, there were ideas about how to schedule therapeutic interventions with a quantitative or otherwise data-driven approach. The providers explained that the optimal time to do therapy sessions is when the patient is already in a heightened state of arousal. However, arousal is difficult to predict, so it is typically estimated manually by trial and error. we definitely try to try to schedule it around their maximal arousal. Maybe they're the patients, like a teenager who never woke up before 10:00 AM before and would be pissed if they were waking up. OK, I'll show up 11:15, like I'll try to go later. Or maybe we find, you know they're most awake first thing in the morning because they just got their meds. So let's go in then. So we try to, if we see a potential to optimize, we take it for sure. But if there isn't, isn't like anything really indicating, we just kind of play around and see what happens. And from there we usually start to get some feeling about what it's better. So maybe like the morning sessions, they're really good. And that's when I want to do my assessment and the afternoon sessions. They're less aroused, and that's when I'm gonna do more sensory stim. Just see if I can get them back aroused. P3 42 However, there are hurdles present in optimizing this scheduling as little data is gathered outside of therapy sessions, and biometric data such as heart rate can be unreliable and noisy in general. Given these hurdles, the providers explained that technology-based behavioral indicators of arousal and awareness could support datadriven decision-making. P7 expressed interest in having a system automatically administer stimulation when a behavioral event occurs, though recognizing the technological barriers explains the process described above. Because even if we just have a data set of a particular patient and maybe their cycle of consciousness or or how that looks for them, that maybe at certain points in the day they're they're more likely or they've shown that they're have occurred more arousal occurrences have happen. Then maybe we can schedule therapeutic interventions for those times and try to extend. That state of consciousness. And so that could be helpful as well, if it, even if it doesn't have an automated response, but at least it's giving us a set of data to to analyze and to determine how to proceed therapeutically. (P7) Another potential application is tracking and aiding in establishing sleep cycles in patients. Establishing circadian sleep-cycles specifically can help the patient orient themselves, as well as increase the likelihood of observing behaviors indicative of arousal and awareness by aligning times of wakefulness with the providers’ work schedules. Currently, sleep cycles are tracked manually, and some want a way to automate this. 43 If there's an order for a sleep log, we have an aide who would check on the patient and chart every hour if they're awake or asleep. So if there were a way to have, I don't know how you would, but that would be automated, that'd be pretty cool. (P4) At the same time, providers also already enforce circadian rhythm by using the smart home technology in patient rooms to augment the environment to promote wakefulness and restfulness. Some want to further automate this workflow. We talked about just the the room, you know, just these patients should have the blinds go all the way up as soon as the sun rises. I mean, they should just have all the sun and light in the day as possible and all the quiet and dark at night as possible. Like they just need to really, so that could be automated like as soon as it's morning, like it's just bright in the room, you know, that that would be fine. I think whatever helps make it that day and night really clear to them. (P2) Lastly, while executing the Coma Recovery Scale assessment, there seems to be a significant benefit from cross-validating claims between providers. So we're trying to have some validity with, with the testing of like at least two people have their eyes on him at the same time and are seeing that behavior and would agree that, like, yeah, most people would say that he moved his thumb. It was pretty clear because otherwise, you know, that's, we just try to be somewhat valid, reliable. (P2) 44 However, one provider explained that this is also a possible source of inefficiency as it takes time from the providers’ schedules, which they could plausibly spend on intervention. There is a desire, though directly limited to one participant, to introduce automation to increase the efficiency of the assessment. As an aside, I'm looking to hopefully code and build, um, AI assisted scoring for video, not just audio and a lot of this is video, would be necessary, as well as audio, of course […]. I think it could do the whole JFK coma recovery scale, like the whole score, like just go crank through it. (P1) While the development effort, technological limitations, and possible value-add differ through different ideas expressed, it is clear that providers believe enhancing the data collection, documentation, and communication capabilities can improve the DoC program at NRH. 45 DISCUSSION The results encapsulate the providers’ perspectives on current challenges faced and how technology might address them, along with the potential risks and considerations that such innovation may facilitate. This section applies these results to answer our research questions. DESIGN OPPORTUNITIES AND DATA-DRIVEN INNOVATION The most immediately actionable technological innovation in DoC care appears to be streamlining in-room sensory stimulation. Specific implementation details were given throughout the interviews, which guide answers to research questions one, two, and three. While investigating RQ2, (How do relevant hospital employees currently use Smart Patient Rooms for DoC care? What design opportunities exist in this space?), we found that if the provider uses the technology, the methodology in treatment is quite consistent. If it is used, providers play music, display slideshows or relevant media on the television, and modify the environment to stimulate the patient. Furthermore, many improvement ideas can be summarized as natural extensions of current workflows, primarily focused on streamlining what is already being done. Answers to RQ2 are closely related to RQ1 (What aspects of Sensory Stimulation do stakeholders think would and would not be useful to automate or supplement technology?). Many ideas incorporated an AI-based aspect, though barriers towards these longer-term, AI-based solutions were recognized. The ability to streamline the in-room stimulation by programming stimulation sequences to play automatically at the push of a button appears to be the most wanted, feasible, and helpful innovation at this point, which could integrate with AI in the future. Despite its relative feasibility, this still demands 46 significant development effort. Maximizing the usability and utility of this system requires streamlined personally relevant stimulation material collection and deployment, which requires a way to integrate seamlessly with existing processes, and involves multiple user-groups who may all need unique interaction considerations. Alongside the ability to program stimulation sequences, the providers also expressed much interest in enhancing data collection, helping to answer RQ3 (How do stakeholders envision leveraging the data captured from patient rooms? What is the work involved in these opportunities and is it feasible?). Whether while an automated stimulation sequence is being administered or in other settings where a provider’s attention cannot be focused, there is a desire to leverage the cameras in the patient rooms to capture behaviors that indicate arousal and awareness. Studies have showcased the capabilities and potential of AI-based patient monitoring and arousal [16] in hospital settings, though not with patients experiencing DoC. There is also work, though limited to a preprint, regarding camera-based arousal detection [3]. While potentially further in the future, the idea of autonomously capturing desired behaviors is exciting and should be explored in more depth. We also found that building this technology may not only help patients with DoC, but all patients with brain injuries. Most notably and immediately, it may help patients with Post-Traumatic Amnesia (PTA). PTA encapsulates a state after traumatic brain injuries and is distinguished from coma, UWS, and MCS by the presence of intact— though limited—consciousness. Furthermore, while PTA may be the direct outcome of emerging from MCS, it is not necessary [37]. Though they both revolve around stimulation at NRH, DoC treatment has stark differences from PTA treatment. DoC 47 treatment revolves around frequent, sometimes intense, sensory stimulation. In PTA treatment, stimulation is limited to reduce agitation as these patients can be very sensitive leading to agitation and various other implications [28]. Enabling providers more finegrained control of stimuli in the patient rooms, some providers think may be useful for PTA and DoC treatment alike. Notably, however, PTA applications were not studied to the same extent as their DoC counterparts. Therefore, these ideas remain unclear. AUTOMATION RISKS AND CONSIDERATIONS The providers presented a range of perspectives and considerations regarding the potential risks of automating aspects of SS, answering RQ4 (What are the risks involved in creating an automated system and what steps can be taken to mitigate them?). We found that the risk involved with SS is minimal, which led to many indicating or stating directly that there would be minimal risks in an automated system, given that necessary precautions are considered. There was agreement amongst providers that some form of safeguard to ensure overstimulation does not occur is needed. However, there was no consensus regarding the specific implementation of a safeguard system. Some providers expressed interest in developing algorithmic safeguards, while some focused on the manual ability to halt a sequence. While these ideas are not mutually exclusive, and may both contribute towards an optimal policy, the specific implementation details remain unclear and thus should be investigated further. LIMITATIONS This work was limited to a single institution which has many tech-driven workflows in place already, possibly leading to biased data. Therefore, future work 48 should obtain the perspectives of friends and family members of patients with DoC as well as healthcare providers from different hospitals. 49 CONCLUSION We found that while providers at NRH have many ideas for innovation, the most immediately actionable and helpful technological opportunities lie in In-Room Sensory Stimulation, collectively forming a specialized use-case of the smart patient rooms. These opportunities include streamlining the collection and distribution of personally relevant stimulation materials, facilitating the administration of these materials within the smart patient rooms, and leveraging the technology in the rooms to capture desired behaviors. In the short term, such a system would likely benefit patients, their loved ones, and providers by reducing burden, thereby enabling more frequent and meaningful SS. In the long term, the implementation of this system may help advance the scientific investigation of therapeutic intervention for patients with DoC by standardizing care. 50 REFERENCES 1. Jitka Annen, Maddalena M. Filippini, Estelle Bonin, Helena Cassol, Charlène Aubinet, Manon Carrière, Olivia Gosseries, Aurore Thibaut, Alice Barra, Audrey Wolff, Leandro R. D. Sanz, Charlotte Martial, Steven Laureys, and Camille Chatelle. 2019. Diagnostic accuracy of the CRS-R index in patients with disorders of consciousness. Brain injury: [BI] 33, 11: 1409–1412. 2. Sergio Bagnato and Cristina Boccagni. 2023. Cerebrospinal Fluid and Blood Biomarkers in Patients with Post-Traumatic Disorders of Consciousness: A Scoping Review. Brain sciences 13, 2. https://doi.org/10.3390/brainsci13020364 3. Björn Braun, Daniel McDuff, Tadas Baltrusaitis, Paul Streli, Max Moebus, and Christian Holz. 2024. SympCam: Remote optical measurement of sympathetic arousal. arXiv [cs.CV]. Retrieved April 14, 2025 from http://arxiv.org/abs/2410.20552 4. Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2: 77–101. 5. Virginia Braun and Victoria Clarke. 2012. Thematic analysis. In APA handbook of research methods in psychology, Vol 2: Research designs: Quantitative, qualitative, neuropsychological, and biological. American Psychological Association, Washington, 57–71. 6. Ann-Marie Cannaby, Vanda Carter, Stephenson Strobel, and Elena Ashtari Tafti. 2022. Measuring nursing movements using real time tracking data at the Royal Wolverhampton Hospital Trust. Spatial and spatio-temporal epidemiology 43: 100543. 7. Camille Chatelle, Srivas Chennu, Quentin Noirhomme, Damian Cruse, Adrian M. Owen, and Steven Laureys. 2012. Brain-computer interfacing in disorders of consciousness. Brain injury: [BI] 26, 12: 1510–1522. 8. Lijuan Cheng, Daniela Cortese, Martin M. Monti, Fuyan Wang, Francesco Riganello, Francesco Arcuri, Haibo Di, and Caroline Schnakers. 2018. Do Sensory Stimulation Programs Have an Impact on Consciousness Recovery? Frontiers in neurology 9. https://doi.org/10.3389/fneur.2018.00826 9. L. Coppola, P. Mirabelli, D. Baldi, G. Smaldone, A. Estraneo, A. Soddu, A. M. Grimaldi, G. Mele, Marco Salvatore, and Carlo Cavaliere. 2022. An innovative approach for the evaluation of prolonged disorders of consciousness using NF-L and GFAP biomarkers: a pivotal study. Scientific reports 12, 1: 18446. 10. Joshua Dawson, Eden Fisher, and Jason Wiese. 2024. Hospital Employee Experiences Caring for Patients in Smart Patient Rooms. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI ’24), 1–16. 11. Joshua Dawson, Thomas Kauffman, and Jason Wiese. 2023. It Made Me Feel So Much More at Home Here: Patient Perspectives on Smart Home Technology Deployed at Scale in a Rehabilitation Hospital. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23), 1–15. 12. Joshua Dawson, K. Jens Phanich, and Jason Wiese. 2024. Reenvisioning Patient Education with Smart Hospital Patient Rooms. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 7, 4: 1–23. 51 13. Rosaria De Luca, Mirjam Bonanno, Giuliana Vermiglio, Giovanni Trombetta, Ersilia Andidero, Angelo Caminiti, Patrizia Pollicino, Carmela Rifici, and Rocco Salvatore Calabrò. 2022. Robotic Verticalization plus Music Therapy in Chronic Disorders of Consciousness: Promising Results from a Pilot Study. Brain sciences 12, 8. https://doi.org/10.3390/brainsci12081045 14. Brian L. Edlow, Jan Claassen, Nicholas D. Schiff, and David M. Greer. 2021. Recovery from disorders of consciousness: mechanisms, prognosis and emerging therapies. Nature reviews. Neurology 17, 3: 135–156. 15. Brian L. Edlow, Leandro R. D. Sanz, Len Polizzotto, Nader Pouratian, John D. Rolston, Samuel B. Snider, Aurore Thibaut, Robert D. Stevens, Olivia Gosseries, and Curing Coma Campaign and its contributing members. 2021. Therapies to Restore Consciousness in Patients with Severe Brain Injuries: A Gap Analysis and Future Directions. Neurocritical care 35, Suppl 1: 68–85. 16. Paolo Gabriel, Peter Rehani, Tyler Troy, Tiffany Wyatt, Michael Choma, and Narinder Singh. 2025. Continuous patient monitoring with AI: real-time analysis of video in hospital care settings. Frontiers in Imaging 4: 1547166. 17. Valentina Galiotta, Ilaria Quattrociocchi, Mariagrazia D’Ippolito, Francesca Schettini, Pietro Aricò, Stefano Sdoia, Rita Formisano, Febo Cincotti, Donatella Mattia, and Angela Riccio. 2022. EEG-based Brain-Computer Interfaces for people with Disorders of Consciousness: Features and applications. A systematic review. Frontiers in human neuroscience 16: 1040816. 18. Joseph T. Giacino, Douglas I. Katz, Nicholas D. Schiff, John Whyte, Eric J. Ashman, Stephen Ashwal, Richard Barbano, Flora M. Hammond, Steven Laureys, Geoffrey S. F. Ling, Risa Nakase-Richardson, Ronald T. Seel, Stuart Yablon, Thomas S. D. Getchius, Gary S. Gronseth, and Melissa J. Armstrong. 2018. Practice guideline update recommendations summary: Disorders of consciousness: Report of the Guideline Development, Dissemination, and Implementation Subcommittee of the American Academy of Neurology; the American Congress of Rehabilitation Medicine; and the National Institute on Disability, Independent Living, and Rehabilitation Research. Neurology 91, 10: 450–460. 19. Joseph T. Giacino, Caroline Schnakers, Diana Rodriguez-Moreno, Kathy Kalmar, Nicholas Schiff, and Joy Hirsch. 2009. Behavioral assessment in patients with disorders of consciousness: gold standard or fool’s gold? In Progress in Brain Research, Steven Laureys, Nicholas D. Schiff and Adrian M. Owen (eds.). Elsevier, 33–48. 20. Joseph T. Giacino, John Whyte, Risa Nakase-Richardson, Douglas I. Katz, David B. Arciniegas, Sonja Blum, Kristin Day, Brian D. Greenwald, Flora M. Hammond, Theresa Bender Pape, Amy Rosenbaum, Ronald T. Seel, Alan Weintraub, Stuart Yablon, Ross D. Zafonte, and Nathan Zasler. 2020. Minimum Competency Recommendations for Programs That Provide Rehabilitation Services for Persons With Disorders of Consciousness: A Position Statement of the American Congress of Rehabilitation Medicine and the National Institute on Disability, Independent Living and Rehabilitation Research Traumatic Brain Injury Model Systems. Archives of physical medicine and rehabilitation 101, 6: 1072–1089. 52 21. Hyuktae Kwon, Sunhee An, Ho-Young Lee, Won Chul Cha, Sungwan Kim, Minwoo Cho, and Hyoun-Joong Kong. 2022. Review of Smart Hospital Services in Real Healthcare Environments. Healthcare informatics research 28, 1: 3–15. 22. Yuanqing Li, Jiahui Pan, Jinyi Long, Tianyou Yu, Fei Wang, Zhuliang Yu, and Wei Wu. 2016. Multimodal BCIs: Target Detection, Multidimensional Control, and Awareness Evaluation in Patients With Disorder of Consciousness. Proceedings of the IEEE 104, 2: 332–352. 23. Andrea I. Luppi, Joana Cabral, Rodrigo Cofre, Pedro A. M. Mediano, Fernando E. Rosas, Abid Y. Qureshi, Amy Kuceyeski, Enzo Tagliazucchi, Federico Raimondo, Gustavo Deco, James M. Shine, Morten L. Kringelbach, Patricio Orio, Shinung Ching, Yonatan Sanz Perl, Michael N. Diringer, Robert D. Stevens, and Jacobo Diego Sitt. 2023. Computational modelling in disorders of consciousness: Closing the gap towards personalised models for restoring consciousness. NeuroImage 275: 120162. 24. Maria Grazia Maggio, Antonino Naro, Gianluca La Rosa, Alice Cambria, Paola Lauria, Luana Billeri, Desiree Latella, Alfredo Manuli, and Rocco Salvatore Calabrò. 2020. Virtual Reality Based Cognitive Rehabilitation in Minimally Conscious State: A Case Report with EEG Findings and Systematic Literature Review. Brain sciences 10, 7. https://doi.org/10.3390/brainsci10070414 25. John C. O’Donnell, Kevin D. Browne, Todd J. Kilbaugh, H. Isaac Chen, John Whyte, and D. Kacy Cullen. 2019. Challenges and demand for modeling disorders of consciousness following traumatic brain injury. Neuroscience and biobehavioral reviews 98: 336–346. 26. Julian O’Kelly, L. James, R. Palaniappan, J. Taborin, J. Fachner, and W. L. Magee. 2013. Neurophysiological and behavioral responses to music therapy in vegetative and minimally conscious States. Frontiers in human neuroscience 7: 884. 27. René Padilla and Anna Domina. May/Jun 2016. Effectiveness of Sensory Stimulation to Improve Arousal and Alertness of People in a Coma or Persistent Vegetative State After Traumatic Brain Injury: A Systematic Review. The American journal of occupational therapy: official publication of the American Occupational Therapy Association 70, 3: 1-5A. 28. Thomas D. Parker, Richard Rees, Sangeerthana Rajagopal, Colette Griffin, Luke Goodliffe, Michael Dilley, and Peter Owen Jenkins. 2022. Post-traumatic amnesia. Practical neurology 22, 2: 129–137. 29. Fabien Perrin, Maïté Castro, Barbara Tillmann, and Jacques Luauté. 2015. Promoting the use of personally relevant stimuli for investigating patients with disorders of consciousness. Frontiers in psychology 6: 1102. 30. David Pickham, Nic Berte, Mike Pihulic, Andre Valdez, Barbara Mayer, and Manisha Desai. 2018. Effect of a wearable patient sensor on care delivery for preventing pressure injuries in acutely ill adults: A pragmatic randomized clinical trial (LS-HAPI study). International journal of nursing studies 80: 12–19. 31. Ana Claudia Giesbrecht Puggina and Maria Julia Paes da Silva. 2015. Patients with disorders of consciousness: vital, facial and muscular responses to music or messages. Revista brasileira de enfermagem 68: 102–110. 53 32. Giuseppe Reale, Augusto Fusco, Rossella Calciano, Noemi Vallario, Gabriele Vagnarelli, Pietro Caliandro, Letizia Castelli, Marco Moci, Gaetano Tieri, Luigi Iasevoli, and Luca Padua. 2023. The Immediate Effects of Immersive Virtual Reality on Autonomic Nervous System Function in Patients with Disorders of Consciousness after Severe Acquired Brain Injury: A Pilot Study. Journal of clinical medicine research 12, 24. https://doi.org/10.3390/jcm12247639 33. Eric Rojas, Jorge Munoz-Gama, Marcos Sepúlveda, and Daniel Capurro. 2016. Process mining in healthcare: A literature review. Journal of biomedical informatics 61: 224–236. 34. Mehar Sahu, Rohan Gupta, Rashmi K. Ambasta, and Pravir Kumar. 2024. IoTdriven augmented reality and virtual reality systems in neurological sciences. Internet of Things 25: 101098. 35. Caroline Schnakers, Wendy L. Magee, and Brian Harris. 2016. Sensory Stimulation and Music Therapy Programs for Treating Disorders of Consciousness. Frontiers in psychology 7: 297. 36. Anil K. Seth and Tim Bayne. 2022. Theories of consciousness. Nature reviews. Neuroscience 23, 7: 439–452. 37. Mark Sherer, Douglas I. Katz, Yelena G. Bodien, David B. Arciniegas, Cady Block, Sonja Blum, Matt Doiron, Kim Frey, Joseph T. Giacino, Min Jeong P. Graf, Brian Greenwald, Flora M. Hammond, Kathleen Kalmar, Jacob Kean, Marilyn F. Kraus, Risa Nakase-Richardson, Shital Pavawalla, Amy Rosenbaum, Donald T. Stuss, and Stuart A. Yablon. 2020. Post-traumatic confusional state: A case definition and diagnostic criteria. Archives of physical medicine and rehabilitation 101, 11: 2041– 2050. 38. Fabrizio Stasolla, Leonarda Anna Vinci, and Maria Cusano. 2022. The Integration of Assistive Technology and Virtual Reality for Assessment and Recovery of Postcoma Patients With Disorders of Consciousness: A New Hypothesis. Frontiers in psychology 13: 905811. 39. Jie Yu, Fanxia Meng, Fangping He, Fei Chen, Wangxiao Bao, Yamei Yu, Jintao Zhou, Jian Gao, Jingqi Li, Yao Yao, Woo-Ping Ge, and Benyan Luo. 2021. Metabolic Abnormalities in Patients with Chronic Disorders of Consciousness. Aging and disease 12, 2: 386–403. 40. 2021. Brain Injury Treatment & Recovery. University of Utah Health | University of Utah Health. Retrieved June 28, 2024 from https://healthcare.utah.edu/neilsenphysical-rehab-hospital/specialties/brain-injury 54 Name of Candidate: Thomas Kauffman Date of Submission: April 22, 2025