Dexterous control of a hand prosthesis using neuromuscular signals from implanted or surface electrodes

Amputation of all or part of the upper limb, whether by trauma or by disease, is a particularly debilitating condition that greatly limits function and degrades quality of life. Commercially available prosthetic arms restore some basic functions but are far from restoring full limb functionality, such as dexterous and intuitive hand motions and fingers that provide sensory feedback. More functional, multiarticulated, sensorized robotic prosthetic hands are being developed. However, the interface from the human to the robotic prosthesis is lacking. The primary goal of the work presented herein is to develop a decoding strategy for a bidirectional human-machine prosthetic arm interface. The decoding system described here is capable of deciphering intended hand movements from the user's residual nerves and/or muscles in a bidirectional setting. This work details the evolution and improvement of a decoding strategy over time as more numbers and types of electrophysiological signal sources are used as inputs. Due to its versatility, the decode strategy described here has the potential to be used widely by amputees, and may be applicable to a number of other adaptive technologies beyond prosthetic arms. I begin by showing that peripheral nerve signals from human upper-limb amputees can be measured with implanted 100-microelectrode arrays and decoded in realtime to control 5 degrees-of-freedom (DOF) of a virtual prosthetic hand. Additionally, these implanted neural electrodes were used to provide a large variety of sensations by electrical stimulation of the nerve. In the next iteration, I expand the decoding strategy to include iv signals measured from implanted electrodes in the residual muscles of human amputees. Using these electromyographic signals, I show that amputees can control 10 DOF of a virtual or robotic prosthetic hand. In the next iteration, I expand the decoding system to include electromyographic signals measured from surface electrodes. Despite the lessspecific nature of the surface electromyographic signals, I showed that 10 DOFs could still be decoded in realtime. Finally, I used the decode strategy developed herein to decode movements from the arm stump of a human congenital upper-limb amputee (3 DOFs), and from the neck and shoulder muscles of a quadriplegic human (10 DOFs) using surface electromyographic electrodes.

DEXTEROUS CONTROL OF A HAND PROSTHESIS USING NEUROMUSCULAR SIGNALS FROM IMPLANTED OR SURFACE ELECTRODES by Suzanne Marie Wendelken A dissertation submitted to the faculty of The University of Utah in partial fulfillment of the requirements for the degree of Doctor of Philosophy Department of Bioengineering The University of Utah May 2018 Copyright © Suzanne Marie Wendelken 2018 All Rights Reserved The University of Utah Graduate School STATEMENT OF DISSERTATION APPROVAL The dissertation of Suzanne Marie Wendelken has been approved by the following supervisory committee members: and by Gregory A. Clark , Chair May 31, 2017 Christopher R. Butson , Member May 31, 2017 Alan Dale Dorval II , Member May 31, 2017 Douglas T. Hutchinson , Member May 31, 2017 David J. Warren , Member May 31, 2017 and by David B. Kieda, Dean of The Graduate School. Date Approved Date Approved Date Approved Date Approved , Chair/Dean of David W. Grainger the Department/College/School of Date Approved Bioengineering ABSTRACT Amputation of all or part of the upper limb, whether by trauma or by disease, is a particularly debilitating condition that greatly limits function and degrades quality of life. Commercially available prosthetic arms restore some basic functions but are far from restoring full limb functionality, such as dexterous and intuitive hand motions and fingers that provide sensory feedback. More functional, multiarticulated, sensorized robotic prosthetic hands are being developed. However, the interface from the human to the robotic prosthesis is lacking. The primary goal of the work presented herein is to develop a decoding strategy for a bidirectional human-machine prosthetic arm interface. The decoding system described here is capable of deciphering intended hand movements from the user’s residual nerves and/or muscles in a bidirectional setting. This work details the evolution and improvement of a decoding strategy over time as more numbers and types of electrophysiological signal sources are used as inputs. Due to its versatility, the decode strategy described here has the potential to be used widely by amputees, and may be applicable to a number of other adaptive technologies beyond prosthetic arms. I begin by showing that peripheral nerve signals from human upper-limb amputees can be measured with implanted 100-microelectrode arrays and decoded in realtime to control 5 degrees-of-freedom (DOF) of a virtual prosthetic hand. Additionally, these implanted neural electrodes were used to provide a large variety of sensations by electrical stimulation of the nerve. In the next iteration, I expand the decoding strategy to include signals measured from implanted electrodes in the residual muscles of human amputees. Using these electromyographic signals, I show that amputees can control 10 DOF of a virtual or robotic prosthetic hand. In the next iteration, I expand the decoding system to include electromyographic signals measured from surface electrodes. Despite the lessspecific nature of the surface electromyographic signals, I showed that 10 DOFs could still be decoded in realtime. Finally, I used the decode strategy developed herein to decode movements from the arm stump of a human congenital upper-limb amputee (3 DOFs), and from the neck and shoulder muscles of a quadriplegic human (10 DOFs) using surface electromyographic electrodes. iv This work is dedicated to the volunteers of the studies described here in. These individuals, whose names I withhold for privacy, gave freely of themselves to advance the field for others. They are the true modern day heroes. TABLE OF CONTENTS ABSTRACT....................................................................................................................... iii LIST OF TABLES .............................................................................................................. x LIST OF FIGURES ........................................................................................................... xi ACKNOWLEDGEMENTS .............................................................................................. xv Chapters 1. INTRODUCTION .......................................................................................................... 1 1.1 Significance .............................................................................................................. 1 1.1.1 Commercially available myoelectric prostheses and their limitations............... 3 1.1.2 Bidirectional interfaces ...................................................................................... 3 1.1.3 Decode approaches ............................................................................................ 5 1.1.4 Utah neuromuscular decode approach ............................................................... 7 1.1.5 Preliminary studies............................................................................................. 7 1.1.6 Peripheral nerve implants .................................................................................. 9 1.1.7 Muscle implants ................................................................................................. 9 1.2 Aims of this research .............................................................................................. 10 1.2.1 Aim 1: Restore more than 2-DOF motor control in a realtime, bidirectional interface..................................................................................................................... 10 1.2.2 Aim 2: Use a hybrid neuromuscular implant system (USEAs and iEMG) to provide high-DOF control of a bidirectional virtual or robotic prosthetic hand... 11 1.2.3 Aim 3: Demonstrate that a high-DOF control of a prosthetic hand can be achieved using surface EMG .................................................................................... 11 1.3 Organization............................................................................................................ 12 1.4 References ............................................................................................................... 14 2. RESTORATION OF MOTOR CONTROL AND PROPRIOCEPTIVE AND CUTANEOUS SENSATION IN HUMANS WITH PRIOR UPPER-LIMB AMPUTATION VIA MULTIPLE UTAH SLANTED ELECTRODE ARRAYS (USEAS) IMPLANTED IN RESIDUAL PERIPHERAL ARM NERVES ..................... 23 2.1 Abstract ................................................................................................................... 23 2.1.1 Background ...................................................................................................... 23 2.1.2 Methods............................................................................................................ 24 2.1.3 Results .............................................................................................................. 24 2.1.4 Conclusions ...................................................................................................... 25 2.1.5 Keywords ......................................................................................................... 25 2.2 Background ............................................................................................................. 25 2.3 Methods .................................................................................................................. 28 2.3.1 Study volunteers............................................................................................... 28 2.3.2 Device .............................................................................................................. 29 2.3.3 Surgical procedures .......................................................................................... 29 2.3.4 Experiment setup ............................................................................................. 31 2.4 Results ..................................................................................................................... 39 2.4.1 Electrode impedances ...................................................................................... 39 2.4.2 Decoding USEA recordings allowed intuitive control of many movements ... 40 2.4.3 Offline decode performance ............................................................................ 42 2.4.4 USEA microstimulation produced numerous sensations spanning the hand... 43 2.4.5 USEA-evoked sensations can be used for closed-loop control ....................... 45 2.4.6 Subjects describe their experience in a positive manner.................................. 46 2.4.7 Limited adverse effects .................................................................................... 46 2.5 Discussion ............................................................................................................... 47 2.5.1 Impedance ........................................................................................................ 47 2.5.2 Decode ............................................................................................................. 48 2.5.3 Stimulation ....................................................................................................... 50 2.5.4 Closed-loop control .......................................................................................... 52 2.5.5 Study limitations .............................................................................................. 54 2.6 Conclusions ............................................................................................................. 54 2.7 List of abbreviations ............................................................................................... 55 2.8 Declarations ............................................................................................................ 56 2.8.1 Ethics approval and consent to participate....................................................... 56 2.8.2 Funding ............................................................................................................ 56 2.8.3 Authors’ contributions ..................................................................................... 56 2.9 References ............................................................................................................... 57 2.10 Additional files ..................................................................................................... 77 3. DECODING MOTOR INTENT FROM HUMAN TRANSRADIAL AMPUTEES USING IMPLANTED NEURAL AND INTRAMUSCULAR ELECTRODE ARRAYS .................................................................................................. 80 3.1 Abstract ................................................................................................................... 80 3.2 Background ............................................................................................................. 81 3.2.1 Bidirectional interfaces .................................................................................... 81 3.2.2 Decoding approaches ....................................................................................... 84 3.2.3 Utah neuromuscular decode approach ............................................................. 86 3.2.4 Previous studies ............................................................................................... 86 3.2.5 Functional considerations ................................................................................ 87 3.3 Methods .................................................................................................................. 88 3.3.1 Study population .............................................................................................. 88 3.3.2 Neural implants ................................................................................................ 88 3.3.3 Muscle implants ............................................................................................... 89 3.3.4 Surgical procedure ........................................................................................... 89 vii 3.3.5 Experimental methods and metrics .................................................................. 91 3.3.6 Data collection ................................................................................................. 91 3.3.7 Feature extraction............................................................................................. 92 3.3.8 Decode calibration (training phase) ................................................................. 92 3.3.9 Data alignment ................................................................................................. 93 3.3.10 Feature selection ............................................................................................ 93 3.3.11 Realtime decoding ......................................................................................... 94 3.3.12 Online decode performance assessment ........................................................ 99 3.3.13 Post-hoc decode performance assessment ................................................... 101 3.3.14 Signal quality and device stability assessment ............................................ 101 3.4 Results ................................................................................................................... 102 3.4.1 Subjects can perform high-DOF decodes using iEMG and USEA implants implants ................................................................................................................... 102 3.4.2 iEMG is the primary contributor to decoder information .............................. 102 3.4.3 iEMG decode calibration can be stable for up to 14 d ................................... 104 3.4.4 Offline decode stability testing ...................................................................... 104 3.4.5 Arm position effects on decode performance ................................................ 105 3.4.6 Attempts to “rescue” the decode by subtracting out positional baseline were unsuccessful ............................................................................................................ 106 3.4.7 Decoding can be robust during muscle activity inadvertently induced by stimulation............................................................................................................... 106 3.4.8 iEMG signal quality does not degrade over 3 to 12 months .......................... 107 3.4.9 The ability of USEA to detect motor units in the peripheral nerve declines over 3 months .......................................................................................................... 107 3.4.10 Number of working electrodes decreases substantially over 3 months ....... 108 3.5 Discussion ............................................................................................................. 108 3.6 Conclusion ............................................................................................................ 113 3.7 Acknowledgements ............................................................................................... 114 3.8 Additional video files............................................................................................ 144 3.9 List of abbreviations ............................................................................................. 145 3.10 References ........................................................................................................... 145 4. 10 DEGREE-OF-FREEDOM DECODES FROM FOREARM MUSCLES OF INTACT AND TRANSRADIAL HUMAN AMPUTEES USING SURFACE ELECTROMYOGRAM SIGNALS ............................................................................... 151 4.1 Abstract ................................................................................................................. 151 4.2 Introduction ........................................................................................................... 152 4.3 Background ........................................................................................................... 152 4.4 Methods ................................................................................................................ 155 4.4.1 Study population ............................................................................................ 155 4.4.2 Experimental methods ................................................................................... 156 4.4.3 Data analysis .................................................................................................. 162 4.5 Results ................................................................................................................... 164 4.5.1 Acute studies .................................................................................................. 164 4.5.2 Stability studies .............................................................................................. 165 viii 4.5.3 Realtime comparison sEMG and iEMG ........................................................ 166 4.5.4 Post-hoc comparison of sEMG with iEMG ................................................... 166 4.5.5 “Button” electrodes work as well as commercial “snap” electrodes ............. 167 4.5.6 Unconstrained software-differenced pairs of monopolar sEMG signal improves offline performance compared to traditional forced-bipolar-pairs ......... 167 4.5.7 Addition of thresholds to decode output improves individual DOF independence and reduces crosstalk ....................................................................... 168 4.6 Discussion ............................................................................................................. 169 4.6.1 Intrinsic hand motions.................................................................................... 170 4.6.2 Advantages and disadvantages of sEMG compared with iEMG ................... 170 4.6.3 Signal processing ........................................................................ ................... 173 4.7 Conclusion ............................................................................................................ 174 4.8 Acknowledgements ............................................................................................... 174 4.9 List of abbreviations ............................................................................................. 196 4.10 References ........................................................................................................... 196 5. FUTURE DIRECTIONS ............................................................................................ 199 5.1 Impacts on the field of neuromuscular decoding interfaces ................................. 199 5.2 Future sEMG studies and prototypes .................................................................... 199 5.2.1 An affordable, high-DOF sEMG prosthetic arm ........................................... 200 5.2.2 Study limitations to be addressed in future studies ........................................ 200 5.2.3 Possibilities for transhumeral amputees......................................................... 201 5.2.4 Possibilities for congenital amputees ............................................................. 201 5.2.5 Possibilities for spinal cord injuries ............................................................... 202 5.3 Acknowledgements ............................................................................................... 204 5.4 References ............................................................................................................. 209 ix LIST OF TABLES Tables 3.1 Online decode results for HS1 and HS2 comparing USEA+iEMG, iEMG, and USEA data .............................................................................................................. 121 3.2 Offline decode analysis of data from 5 training sets from HS1 comparing neural (USEA) and iEMG signal sources. ......................................................................... 122 4.1 List and descriptions of DOFs. ............................................................................... 178 4.2 Best performances during online decodes for target touching tasks for a) intact and b) amputee subjects .......................................................................................... 179 4.3 Decode stability for each subject .. .......................................................................... 181 4.4 iEMG vs. sEMG online results. .............................................................................. 187 LIST OF FIGURES Figures 1.1 Schematic diagram of our decode approach to decoding neuromuscular signals. ... 19 1.2 Top panel: Implant locations for S3-S6. ................................................................... 20 1.3 Schematic diagram of the USEA implanted into a peripheral nerve ....................... 21 1.4 Depiction of the 32-channel IEMG array. ............................................................... 22 2.1 USEAs implanted in human peripheral arm nerves were used to provide amputees with multi-DOF control of virtual prosthetic hand movement and restore numerous hand sensations. ........................................................................................................ 62 2.2 Number of working electrodes (impedance < 500 kΩ) ............................................ 63 2.3 USEA recordings were collected during a training session ..................................... 64 2.4 Distinct patterns of USEA electrodes with firing rates correlated to movement ...... 65 2.5 Number of electrodes with driven units for S4. ........................................................ 66 2.6 Decode output and raster plot during a 5-DOF target-touching task for S4. ............ 67 2.7 S4 tracked the position of three different moving virtual targets with the thumb, index, and middle fingers.......................................................................................... 68 2.8 “Best case scenario” for multi-DOF offline decode.. ............................................... 69 2.9 USEA microstimulation provided a rich selection of percepts of various qualities and locations spanning the phantom hand ................................................................ 70 2.10 Percepts evoked by median and ulnar nerve USEAs are generally within the established intact-hand innervation regions for each nerve. ..................................... 71 2.11 Number of sensory percept-evoking electrodes by week. ........................................ 72 2.12 Confusion matrix for a 1-DOF closed-loop experiment performed by S3.. ............. 73 2.13 Example of decode output (dashed line) and target distance (solid line) during two sequential trials of a 1-DOF closed-loop session for S3.. ......................................... 74 3.1 Diagram of hybrid neuromuscular prosthetic interface concept ............................. 115 3.2 Photographic and X-ray images of implanted USEAs. .......................................... 116 3.3 Depiction of the 32-channel iEMG array (Ripple, LLC). ....................................... 117 3.4 Schematic diagram for the hybrid neuromuscular decoding system. ..................... 118 3.5 Hand matching task in the Mujoco VRE ................................................................ 119 3.6 Data display of a 6 s window of USEA and iEMG during an omnibus trial from HS2 ......................................................................................................................... 120 3.7 Boxplots of MSE and correlation coefficients from offline analysis using 5 training sets from HS1. ......................................................................................................... 123 3.8 HS2 3-DOF hand matching task performance improves over time. ...................... 124 3.9 Three decode calibrations were tested 4 to 9 d post training for HS1 .................... 125 3.10 3-DOF hand-matching task performance for HS2 using day 0 decode coefficients.............................................................................................................. 126 3.11 RMSE and correlation coefficients from offline analysis of 8-DOF training data for HS1. ........................................................................................................................ 127 3.12 Correlation coefficient (left) and RMSE (right) of 8-DOF offline decodes from HS1 ......................................................................................................................... 128 3.13 RMSE and correlation coefficients from offline analysis of 8-DOF training data for HS2. ........................................................................................................................ 129 3.14 Correlation coefficient r (left) and RMSE (right) of 8-DOF offline decodes from HS2. ........................................................................................................................ 130 3.15 Offline decode analysis of 8-DOF training sets performed by HS2 in different arm positions. ................................................................................................................. 131 3.16 Offline decode examples of HS2 training data while arm was in position 1..........132 3.17 Using the “standard” position decoder (first bar in each set), HS2 was asked to subjectively rate his sense of control over each DOF ............................................ 133 xii 3.18 Attempts to “rescue” the decode performance by subtracting out baseline............ 134 3.19 Example of stimulation artifact in the decode output. ............................................ 135 3.20 Hand-matching task results for HS2 in the presence or absence of MCAP-inducing stimulation............................................................................................................... 136 3.21 Right panel shows “working electrodes” (with z < 500 kOhms) for the ulnar USEA for HS1 through post-implant day 87. .................................................................... 137 3.22 Top panel shows “working electrodes” (with z < 500 kOhms) for the median USEA for HS1 through post-implant day 87. .................................................................... 138 3.23 Number of “working” USEA electrodes (z < 500 kOhms) declines for both median (blue trace) and ulnar (red trace) arrays for HS2 over the first 3 months. .............. 139 3.24 Impedances of “working” electrodes (z < 500 kOhms) for the median (blue trace) and ulnar (red trace) USEA electrodes over the first 3 months of the implant. ...... 140 3.25 Top figure shows driven motor units from median and ulnar USEAs from HS1 over time. ............................................................................................... 141 3.26 EMG signal amplitude (top) and SNR (bottom) during omnibus sessions for HS1 .................................................................................................................... 142 3.27 EMG signal amplitude (top) and SNR (bottom) during omnibus sessions for HS2 ..........................................................................................................................143 4.1 Electrode configurations. ........................................................................................ 175 4.2 Schematic diagram of EMG data collection and decoding signal processing steps......................................................................................................................... 176 4.3 Example depiction of target-touching task. ............................................................ 177 4.4 Example of a 10-DOF online decode target-touching task for Intact_S3. The black lines and blue lines show the target position and decode output for each DOF, respectively. ............................................................................................................ 185 4.5 Example of a 10-DOF online decode target-touching task for Transradial_S1...... 186 4.6 Top row: Offline analysis of simultaneously collected iEMG (red) and sEMG (green) during flexion movement cues (blue)......................................................... 188 4.7 Top row: Offline analysis of simultaneously collected iEMG (red) and sEMG (green) during extension movement cues (blue)..................................................... 189 xiii 4.8 Boxplots of offline decode performance metrics of 6 training sets (RMSE, top, and Correlation, bottom) for offline decodes computed using forced-differential pairs ........................................................................................................................ 190 4.9 Boxplot of median EMG amplitude (32 features) during training session using snap or button electrodes in the same positions on neoprene sleeve for Intact_S1. ....... 191 4.10 Boxplots of offline metrics commanded movement correlation and RMSE for “snap” electrodes vs. “button” electrodes for Intact_S1 ......................................... 192 4.11 Surface electrode position shifts relative to underlying muscle during wrist rotation .................................................................................................................... 193 4.12 Boxplot of RMSE during nonmovement time period (“X-talk RMSE”) for 7 10DOF training sets (4 from intact subjects, and 3 from transradial subjects)........... 194 4.13 Example offline decode output from a 10-DOF training set from Intact_S3, with threshold (red trace) and without threshold (dashed-blue trace) applied (black trace represents movement cue)....................................................................................... 195 5.1 Schematic example of the sEMG technology applied in two different ways .......... 205 5.2 sEMG configuration used for decoding 3 DOF from a congenital transradial amputee. .................................................................................................................. 206 5.3 Image showing the placement of 32 sEMG electrodes on spinal cord injured volunteer. ................................................................................................................ 207 5.4 Decode output (blue trace) and target distance (black trace) for a 10-DOF online target-touching-task, performed by a C2 level quadriplegic individual. ................ 208 xiv ACKNOWLEDGEMENTS Isaac Newton is credited with the saying, “If I have seen further it is because I am standing on the shoulders of giants.” In neural engineering, it is because we stand on the nerves of giants. I thank my colleagues and mentors at the University of Utah, who played an integral role in this work. Specifically, I thank Tyler Davis Ph.D., who mentored me from my first neural engineering project at the U of U in cortical visual prostheses while at the Greger Lab (2011), all the way through the present neuromuscular prostheses project in the Clark Lab. I owe much of my learning of experimental methods and realtime software development to him. His insight and dedication to functional and practical solutions played a pivotal role in this research and have shaped the methods we developed together. Additionally, I thank my committee members, and student colleagues in the Clark Lab including David Page Ph.D., David Kluger, and Jake George, as well as colleagues in other labs and departments, including Jacob Nieveen, Zach Kegan, and Christopher Duncan M.D., for their help conducting experiments and their insight for analyses. I thank my advisor Gregory Clark Ph.D. for his leadership, insight, vast scientific and philosophical knowledge, sense of humor, and tireless attention to detail. I thank David Warren Ph.D. for his technical expertise and signal processing insight. Thanks also to so many past mentors and teachers, Lewis Duncan Ph.D., Jerry McCoy, Sujeet Shenoi Ph.D., Metin Akay Ph.D., George Blike M.D., Kirk Shelly M.D. Ph.D., and Henry Friedman M.D., who encouraged me to explore anything and everything in science, medicine, and engineering, and inspired me to achieve. I thank Janet Basset, the “mom” of the MD-PhD program at the U of U, who helped smooth the way for me on many, many occasions, and was always a friendly face to turn to. Also I thank the local giant in the field, Dick Normann Ph.D., for blazing a trail in neural engineering, and providing insight and inspiration, Finally, I thank my family, friends, hockey and swimming teammates, and adventure buddies, without whom I would not have been able to complete this work and maintain sanity. Much of the work in this dissertation was sponsored by the Defense Advanced Research Projects Agency (DARPA) Biological Technologies Office (BTO) Hand Proprioception and Touch Interfaces (HAPTIX) program under the auspices of Dr. Doug Weber, as well as the DARPA Microsystems Technology Office (MTO) under the auspices of Dr. Jack Judy through the Space and Naval Warfare Systems Center, Pacific Grant/Contract Nos. N66001-15-C-4017 and N66001-12-C-4042. Additional funding was also provided via the National Institutes of Health (NIH NCATS Award No. 1ULTR001067) and the National Science Foundation (NSF ECCS-1533649). xvi CHAPTER 1 INTRODUCTION 1.1 Significance Amputation of all or part of the upper limb, whether by trauma or by disease, is a particularly debilitating condition that greatly limits function and degrades quality of life. It is estimated that 1 in every 200 Americans suffers from limb loss [1], [2], and that there are 2 million hand amputees worldwide [3]–[5]. Although there are a variety of hand prosthetics that restore basic function, such as open-close grip, current prosthetic options lack the dexterity of control and the sensory feedback needed to fully replace hand function. Because of the limited function and other factors (such as comfort and aesthetics), at least 20% of upper limb amputees abandon their prostheses [6]. Multiarticulated robotic prosthetic hands are being developed. These prosthetics are capable of dexterous finger motions and providing environmental feedback through pressure sensors on fingertips and hand surfaces, and joint positions from the motors. However, the interface from the human to the robotic prosthesis is lacking. Dexterous ability together with naturalistic sensory feedback in a bidirectional control system is of great importance and would represent the next major advancement in prostheses. A major focus of this work was on advancing motor decode strategies for such bidirectional neuromuscular interfaces which utilize implanted neural and muscular 2 electrodes for recording and decoding signals from the residual limb, and provide sensory feedback through neural electrical stimulation. State-of-the-art decoding strategies use pattern classification, which constrains the user to a limited number of preprogram patterned hand motions and grasps. The user must train the decoder to perform each desired grip pattern. The greater the number of grips desired, the greater the amount of and more burdensome the training required. In contrast, the decoding strategy described herein uses a type of predictive estimator (a modified Kalman Filter) to predict, in realtime, individual motions of the digits and wrist. The user only needs to train the decoder on a small number of individual hand motions, such as “thumb flex.” As a result, the user is able to intuitively combine individual hand motions into any desired grip pattern without having specifically trained the decoder on that pattern. The decoding strategy described herein is capable of using several types and combinations of motor-driven electrophysiological signals from residual arm limb nerves and muscles, such as neural firing rate or electromyogram (EMG) power, to decode motor intent. Additionally, this method is configurable to any number of signal sources. The number of signal sources used in this method is limited only by computational power of the decoding machine, and physical constraints of electrode size and residual limb volume or surface area. The outputs of the decode algorithm do not necessarily need to be mapped to fingers and digits of a prosthetic wrist. The decode outputs can be used to control any system where multiple, simultaneous and individual degree-of-freedom input is needed (such as a motorized wheelchair, computer console, or game controller). Because of this high configurability, the decode method presented here is applicable to many styles of prosthetic interfaces and adaptive technologies. Thus, the methodologies described herein 3 have the potential to improve function and quality of life of not only amputees, but many types of injured or disabled individuals. 1.1.1 Commercially available myoelectric prostheses and their limitations Examples of commercially available myoelectric arms include the “Utah Arm” (Motion Control, Salt Lake City, UT, USA [7]), the “Steeper Myoelectric Hand” (SteeperUSA, San Antonio, TX, USA [8]), “Bebionic hand” (Ottobock, Duderstadt, Germany [9]), and the “i-limb” (Touch Bionics, Mansfield, MA, USA [10]). Most commercially available myoelectric prosthetic arms utilize a simple, low degree-offreedom (DOF) decode algorithm, termed “direct control,” which uses a limited number of precisely placed surface EMG (sEMG) electrodes (4-10) whose signals are directly assigned to individual DOFs or grip patterns. As a result, the user can only control a few (1-2) DOFs or patterns of control of the prosthetic hand (typically hand open/close and wrist rotation), and often needs to manually switch, or “clutch,” between different movement modes or grasp categories, often by co-contracting two of the control muscle groups simultaneously which may be cumbersome and fatiguing. Additionally, graded movements can be difficult for the user, and initiation of movement requires a high level of muscle contraction, both frustrating and fatiguing the user. Finally, recalibration often must be done by a prosthetist which is burdensome and expensive. 1.1.2 Bidirectional interfaces To date, a number of approaches to interface with the nerve and muscle have been proposed and attempted, with promising results. Interfaces require two major components: 4 1) the ability to record and decode electrical activity of residual muscles and/or nerves; 2) the ability to provide sensory feedback via electrical of the nerve or sensory substitution. Early bidirectional interfaces included single Longitudinal Intrafascicular Electrodes (LIFE electrodes) implanted into residual arm nerves to provide limited, single-DOF control and provide sparse localized sensations on the phantom hand [11]–[13]. More recent interfaces utilize a combination of implanted neural and sEMG electrodes to provide up to 4 movements (by a classification-based decode) and 2 regions of sensation in closedloop operation [14], [15]. Because providing sensory feedback is important for embodiment of the prosthesis [16]–[18], several groups have focused primarily on restoring stable sensory feedback via neural stimulation or sensory substitution. In sensory substitution, feedback is provided by mechanical pressure or vibrations applied to the end of the stump. This noninvasive method is a viable approach for providing sensory feedback [17]. However, truly naturalistic sensations cannot be provided in this manner due to the inherent mismatch between the applied stimulus location/modality and environmental stimulus. Thus, restoration of naturalistic sensation through direct neural stimulation is being explored. Clark, Page et al. have demonstrated restoration of up to 131 highly localized cutaneous and proprioceptive sensations spanning the phantom hand of amputees using Utah Slanted Electrode Arrays (USEAs) implanted in residual peripheral arm nerves [16], [19], [20]. Additionally Tan, Tyler et al. have demonstrated restoration of naturalistic pressure sensations on up to 20 regions on the phantom hand in 2 subjects which have been achieved using cuff electrodes, which wrap around the nerve, with stability of up to 2 years [18]. A growing number of advanced robotic prostheses capable of supporting a closed- 5 loop control interface are currently undergoing development. Examples include the Modular Prosthetic Limb (Johns Hopkins Applied Physics Laboratory, Baltimore, MD, USA) and “Luke” arm (DEKA, Manchester, NH, USA) [21], [22]. These prostheses have multiarticulated joints, capable of complex movements and dexterous, high-DOF control. Additionally, pressure and torque sensors positioned on the fingers, joints, and palm of the prosthesis can provide information about the environment and position of the robotic digits and wrist. This sensory information can then be translated into neural stimulation parameters (frequency, amplitude, etc.) [16], [18]. A bidirectional interface for such a prosthesis is presented in this work. 1.1.3 Decode approaches A number of groups have demonstrated the feasibility of using peripheral nerve signals, iEMG, and/or sEMG for decoding intended movement in both animals and humans [12], [15], [23]–[32]. The most common control strategy for commercially available myoelectric prostheses is the “direct control” method. In this method, the power of EMG signals from specific electrodes are mapped “directly” to a single direction of one DOF. For example, EMG power from surface electrodes placed over forearm flexor muscles can be mapped to the “hand close” command, and electrodes placed over the forearm extensor muscles can be mapped to the “hand open” command. Using such methods, typically 1 or 2 DOFs such as hand open/close, and/or wrist pronation supination are employed in commercially available devices. Advanced prostheses such as the “i-limb” hand allow the user to select from a number of hand grasp such as “tool grip” or “pencil grip.” However, in these 6 scenarios, the grasp open/close commands are still controlled by only a single DOF (e.g., flexor surface electrodes control the fingers closing together in the specific grip pattern and extensor surface electrodes control the fingers opening together in the specific grip pattern). A limitation of this approach is the small number of DOFs that can be simultaneously controlled. In direct control, crosstalk between the DOFs is also problematic. Because surface electrodes pick up signals from adjacent muscles beneath the electrodes, it is difficult to isolate signal sources. The separability of DOFs in the direct control depends on isolation of the signal sources. Weir et al. have made efforts to isolate signals by using transcutaneous, fine-wire electrodes inserted directly into specific muscle bellies. Four DOF control has been achieved with more specific fine wire EMG [29]. Pattern recognition strategies are used to decode grasps, as opposed to individual DOFs [33]. A classification decoder is trained using categorical grips or movements. Classification techniques such as support vector machines, artificial neural networks, linear discriminant analysis, hidden Markov models, Gaussian mixture models, and fuzzy logic classifiers are among those used [34]–[38]. Zhou, Kuiken et al. have demonstrated up to 16 grip classifications using high-count sEMG recorded from a transhumeral amputee after targeted muscle reinnervation (TMR) procedure [23]. Although classification approaches have proven to be highly accurate (approaching 100% accuracy) [31], they offer a limited number of grips the user can control, and sequential in nature (only one movement can be performed at a time). Additionally, users must train on all categorical grips desired for the classification “vocabulary,” which may be burdensome. Realtime regression algorithms to predict individual DOFs are also used [28]. In 7 contrast to classification methods, regression strategies provide realtime, intuitive proportional control of individual degrees of freedom, allowing for an unlimited number of grip possibilities and arm positions. Smith, Hargrove et al. demonstrated 3-DOF proportional control using fine wire EMG implanted in the forearm of intact subjects [28]. Additionally, a commercial prosthetic arm, the Coapt Complete Control (Coapt LLC, Chicago, IL, USA), uses 9 surface EMG electrodes and a proprietary predictive pattern recognition algorithm to decode 3 DOFs (hand open/close, wrist pronate/supinate, and elbow flex/extend) [39]. 1.1.4 Utah neuromuscular decode approach Our approach involves using a modified Kalman filter (described in Chapter 3) to estimate the position of individual virtual fingers from the firing rate and/or EMG power from the residual arm. This method is a type of Bayesian estimator that has previously been used to successfully decode intended movements from firing rates of a population of neurons in cortex [40]–[44]. Our current decode approach, capable of using USEA and/or EMG signals for motor decodes, is outlined in Figure 1.1. 1.1.5 Preliminary studies To date, as part of an ongoing effort through the DARPA RE-NET and HAPTIX programs, our group has implanted 6 human transradial amputees with USEAs (subjects S1-S6) and iEMG (subjects S5-S6) in residual median and/or ulnar nerves (see Figure 1.2). Early subjects (S1-S3) had one or two USEAs implanted in a distal location near the end of the residual limb, due to safety precautions. In these subjects, there was limited 1-2- 8 DOF online control [45] for several reasons: 1) The implant location was distal to many motor branch points controlling hand movements in these nerves; 2) because EMG spikes, common across many electrodes, were also present, which often obscured smaller neural spikes. In later subjects (S4-S6), USEAs were implanted at an above-elbow location, proximal to most motor neuron branches for the forearm in attempt at getting a better control signal. S5 and S6 (also referred to as HS1 and HS2, respectively) were additionally implanted with a 32-channel IEMG array in residual forearm muscles. The location and types of implants are shown in Figure 1.2. In the work reported herein, data from S3-S6 are used, in addition to data from intact subjects. In preliminary studies involving S1-S2, Davis, Wark et al. [45] used neural signals from USEAs implanted into the peripheral arm nerves of a human amputee to control up to 2 movements virtual prosthetic hand (VPH) in realtime. In these studies, one USEA was implanted into either the distal residual median (S1) or ulnar (S2) arm nerves for 4 weeks. An online Kalman filter was used to estimate the position of individual virtual fingers from the neural firing rate from specific electrodes which were manually selected in realtime. Proportional online control of 2 movements was demonstrated for both subjects. Subjects were trained on up to 8 DOF, 13 movements, where each movement represents either flexion or extension of a particular DOF. Offline, these training sessions were decoded and the output compared to the movements trained on. Correlations between decode output and training movement ranged from approximately 0.7 – 0.9 for 2 movements to approximately 0.5 for 13 movements. 9 1.1.6 Peripheral nerve implants All transradial amputee subjects discussed in this work were implanted with two, 100-electrode Utah Slanted Electrode Arrays [46] (USEAs, Blackrock Microsystems, Salt Lake City, UT, USA). Ninety-six of one hundred electrodes were used for recording and stimulation, and 4 electrodes were used as an on-array reference. Each array measured 4x4 mm, with electrodes arranged in a 10x10 grid, spaced at 400 um, and containing electrodes of 0.7-1.5 mm lengths (see Figure 1.3). The electrodes were wire-bonded to an 11-14 cm silicon-potted platinum wire bundle which was soldered to a custom 100-channel Gator connector (Ripple LLC, Salt Lake City, UT, USA) or 96 pin omnetics-ZIF adapter (TDT Microsystems, Alachua, FL, USA). 1.1.7 Muscle implants The intramuscular implant prototype used in these experiments consisted of a 32channel, 9-lead array of flexible, silicon-potted coiled wires (Ripple, LLC, Salt Lake City, UT, USA). Exact specifications are not currently available from the manufacturer. An illustration of a similar prototype is found in Figure 1.4. Leads 1-8 were approximately.20 cm long, and each contained 4 cylindrical electrodes approximately 2 mm wide, spaced at approximately 2 mm, at the distal end. Lead 9 was approximately 5 cm long, contained 2 cylindrical electrodes (same size and spacing of leads 1-8) that served as the reference and ground. Additionally, a small flexible plastic barb was attached to the end of each lead for anchoring the leads into the tissue. All leads were tethered together into a bundle at the proximal end and wire-bonded to a custom 32-channel Gator connector (Ripple LLC). The wire bundle passed through a percutaneous opening where the Gator board was connected 10 to the data acquisition system. 1.2 Aims of this research The overarching aim of this research is to advance and implement realtime motor decoding strategies for upper limb prostheses. The first aim focuses on using the USEAs to decode more than 2 DOF in realtime. The second aim focuses on expanding aim 1 and utilizing implanted EMG (iEMG) signals in addition to USEAs for decoding motor intent. In the final aim, the methodologies developed in aims 1 and 2 are implemented with noninvasive surface EMG electrodes. 1.2.1 Aim 1: Restore more than 2-DOF motor control in a realtime, bidirectional interface Expanding on work performed by Davis, Wark et al. [45], where signals from one USEA were used to decode 2 DOFs in realtime and restore sensations in a single nerve distribution, the goal of this aim was to decode more than 2 DOFs decoded in realtime and restore sensations across the entire palmar surface of the phantom hand. In this study, two USEAs were implanted in the median and ulnar nerves of two human transradial amputees (one USEA each in the ulnar and median nerves for both subjects). Doubling the number of electrodes from 100 (in S1 and S2) to 200 (in S3 and S4) provided more signal sources, which allowed a higher DOF decode. In these studies, 5 distinctive DOFs were decoded in realtime. In addition, because USEAs were implanted in both the median and ulnar nerves, sensations spanning the palmer surface of the phantom hand were demonstrated. 11 1.2.2 Aim 2: Use a hybrid neuromuscular implant system (USEAs and iEMG) to provide high-DOF control of a bidirectional virtual or robotic prosthetic hand In this aim, the methods developed in Aim 1 were modified to incorporate signals from both USEAs and iEMGs in order decode more than 5 DOFs in realtime. In these studies, 2 USEAs and a 32-electrode iEMG array were implanted in two transradial amputees for ≥ 3 months. Ten simultaneous DOFs were decoded in realtime and used to control a virtual or robotic prosthetic hand. A number of functional topics were also addressed in this aim including: the information value of neural signals compared and contrasted to muscular signals; the stability of decode calibration in both realtime and offline scenarios; the effects of various arm reach positions on decoder performance; device signal quality over time; and the effects of neural stimulation on decode performance in a bidirectional setting. 1.2.3 Aim 3: Demonstrate that a high-DOF control of a prosthetic hand can be achieved using surface EMG In this aim, methods developed in Aim 2 are used in the setting of high-count (≥18) surface EMG electrodes. In these studies, sEMG from transradial amputees and intact subjects are decoded, and 10 DOF realtime control of a virtual hand and wrist were demonstrated. Other topics were addressed including: calibration stability over the course of a session and between donning and doffing electrodes; and the ability of subjects to control DOFs that are naturally controlled by muscles residing in the hand (i.e., thumb adduction, index-pinky ab/adduction) using sEMG from the forearm. 12 1.3 Organization In this work, I present the evolution of decoding strategies which use implanted neural electrodes, implanted muscular electrodes, and surface electrode electromyographic (EMG) signals. In Chapter 2, decoding motor intent from the peripheral nerve using signals recorded from Utah Slanted Electrode Arrays (USEAs) is described. In Chapter 3, this approach is expanded to incorporate implantable EMG electrodes, which greatly improves decode performance. In Chapter 4, techniques developed in Chapters 2 and 3 are expanded to include surface EMG signals. In Chapter 5, implications and future directions of these decode strategies are presented. Major results from these chapters are described briefly here. In Chapter 2, the ability of neural recording and stimulation from Utah Slanted Electrode Arrays (USEAs) implanted in the peripheral arm nerves of human amputees could provide subjects with both 1) simultaneous proportional movement control of the digits and wrist of a virtual prosthesis; and 2) a rich selection of proprioceptive and cutaneous sensations spanning the phantom hand in both median and ulnar sensory distributions. A 5-DOF decode (movements included flexion/extension of the thumb, index, middle, and ring fingers, and the wrist) and 131 USEA-evoked cutaneous and proprioceptive percepts exceeds what has previously been accomplished with neural implants in the peripheral nerves of transradial amputees. One subject used a USEAevoked hand sensation as feedback to complete a 1-DOF closed-loop virtual-hand movement task. In Chapter 3, I demonstrate the feasibility of using IEMG signals from the residual forearm of a transradial amputee to decode up to 10 individual and simultaneous DOFs. In 13 these studies, two subjects were implanted with 2 USEAs and a 32-channel IEMG array for ≥ 3 months. The decode calibrations were highly stable for up to a week, reducing the need for frequent and burdensome recalibration by the user. The technique developed here was robust in the presence of muscle activity that was inadvertently caused by stimulation in a closed-loop scenario by the addition of configurable thresholds applied to the decode output. IEMG was the primary contributor to decoder information in both online and offline testing. The motor-driven neural signals from USEAs were not as plentiful or specific as the IEMG signals, and quickly deteriorated over a month. However, IEMG signal amplitude and quality were stable throughout the entire duration of the implants. In Chapter 4, the ability to decode up to 10 DOFs (up to 20 movements) in three transradial amputee subjects and three intact subjects, using one of several types (ECGstyle or dry metal discs) of high-channel-count (14-32 channels) monopolar, surface EMG electrodes and a modified Kalman filter decoding algorithm was demonstrated. I discuss the use of nonstandard signal processing techniques, such as software-differenced EMG channel pairs (as opposed to hard-ware forced-pair differences), and post-hoc application user-configurable gains and thresholds to the decode output for improving decode accuracy. I also compare the relative merits of implanted and surface EMG electrodes. These studies supported the use of sEMG signals as a potential substitute for high-channelcount iEMGs for decoding motor intent from transradial amputees. Finally, in Chapter 5, I discuss implications and future directions of this work in the context of myoelectric control of adaptive devices such as prosthetic limbs, motorized wheelchairs, and computer controls. I also present results from pilot work in which surface EMG signals are used to decode 3 DOFs from residual forearm muscles of a human 14 congenital transradial amputee and 10 DOFs from shoulder and neck muscles of a human quadriplegic individual. 1.4 References [1] K. Ziegler-Graham, E. J. MacKenzie, P. L. Ephraim, T. G. Travison, and R. Brookmeyer, “Estimating the prevalence of limb loss in the United States: 2005 to 2050,” Arch. Phys. Med. Rehabil., vol. 89, no. 3, pp. 422–429, Mar. 2008. [2] G. McGimpsey and T. C. Bradford, “Limb prosthetics services and devices,” Bioeng. Inst. Cent. Neuroprosthetics Worcest. Polytech. Inst., 2008. [3] “Open Bionics,” Open Bionics. [Online]. https://www.openbionics.com/about/. [Accessed: 16-Nov-2016]. Available: [4] “Statistics on hand and arm loss.” [Online]. http://www.ishn.com/articles/97844-statistics-on-hand-and-arm-loss. 13-Sep-2016]. Available: [Accessed: [5] “Bio 108 - Organ Replacement - Hand Prosthetics - Statistics.” [Online]. Available: http://biomed.brown.edu/Courses/BI108/BI108_2003_Groups/Hand_Prosthetics/st ats.html. [Accessed: 13-Sep-2016]. [6] E. A. Biddiss and T. T. Chau, “Upper limb prosthesis use and abandonment: A survey of the last 25 years,” Prosthet. Orthot. Int., vol. 31, no. 3, pp. 236–257, Sep. 2007. [7] “Motion Control, Inc. U3 Arm Myoelectric Prosthetic.” [Online]. Available: http://www.utaharm.com/ua3-myoelectric-arm.php. [Accessed: 17-Jul-2017]. [8] “- RSLSteeper.” [Online]. Available: http://rslsteeper.com/products/prosthetics/products/upper_limb/electric/select_myo _electric_hands. [Accessed: 17-Jul-2017]. [9] “bebionic hand — Ottobock.” [Online]. Available: http://www.ottobockus.com/prosthetics/upper-limb-prosthetics/solutionoverview/bebionichand/?utm_source=google&utm_medium=cpc&utm_term=bebionic%20hand&ut m_campaign=US%20|%20Upper%20Limb%20Myoelectric%20Prosthetics%20|% 20Brand. [Accessed: 17-Jul-2017]. [10] “i-limb ultra | Touch Bionics.” [Online]. http://www.touchbionics.com/products/active-prostheses/i-limb-ultra. Available: [Accessed: 15 17-Jul-2017]. [11] G. S. Dhillon and K. W. Horch, “Direct neural sensory feedback and control of a prosthetic arm,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 13, no. 4, pp. 468– 472, 2005. [12] G. S. Dhillon, S. M. Lawrence, D. T. Hutchinson, and K. W. Horch, “Residual function in peripheral nerve stumps of amputees: Implications for neural control of artificial limbs,” J. Hand Surg., vol. 29, no. 4, pp. 605–615, Jul. 2004. [13] G. S. Dhillon, T. B. Krüger, J. S. Sandhu, and K. W. Horch, “Effects of short-term training on sensory and motor function in severed nerves of long-term human amputees,” J. Neurophysiol., vol. 93, no. 5, pp. 2625–2633, May 2005. [14] S. Raspopovic et al., “Restoring natural sensory feedback in realtime bidirectional hand prostheses,” Sci. Transl. Med., vol. 6, no. 222, p. 222ra19–222ra19, 2014. [15] P. M. Rossini et al., “Double nerve intraneural interface implant on a human amputee for robotic hand control,” Clin. Neurophysiol., vol. 121, no. 5, pp. 777– 783, May 2010. [16] D. M. Page, “Restored hand sensation in human amputees via utah slanted electrode array stimulation enables performance of functional tasks and meaningful prosthesis embodiment,” Ph.D. Thesis, University of Utah, 2016. [17] C. Antfolk, M. D’Alonzo, B. Rosén, G. Lundborg, F. Sebelius, and C. Cipriani, “Sensory feedback in upper limb prosthetics,” Expert Rev. Med. Devices, vol. 10, no. 1, pp. 45–54, Jan. 2013. [18] D. W. Tan, M. A. Schiefer, M. W. Keith, J. R. Anderson, J. Tyler, and D. J. Tyler, “A neural interface provides long-term stable natural touch perception,” Sci. Transl. Med., vol. 6, no. 257, p. 257ra138-257ra138, Oct. 2014. [19] G. A. Clark et al., “Using multiple high-count electrode arrays in human median and ulnar nerves to restore sensorimotor function after previous transradial amputation of the hand,” in 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2014, pp. 1977–1980. [20] D. M. Page, “Restoration of sensory and motor hand function via two Utah Slanted Electrode Arrays (USEAs) in residual arm nerves after prior hand amputation,” in Neuroscience Meeting Planner, Washington, DC, 2014, p. 636.19. [21] M. S. Johannes, J. D. Bigelow, J. M. Burck, S. D. Harshbarger, M. V. Kozlowski, and T. Van Doren, “An overview of the developmental process for the modular prosthetic limb,” Johns Hopkins APL Tech. Dig., vol. 30, no. 3, pp. 207–216, 2011. 16 [22] L. Resnik, S. L. Klinger, and K. Etter, “The DEKA Arm: Its features, functionality, and evolution during the Veterans Affairs Study to optimize the DEKA Arm,” Prosthet. Orthot. Int., vol. 38, no. 6, pp. 492–504, 2014. [23] P. Zhou et al., “Decoding a new neural–machine interface for control of artificial limbs,” J. Neurophysiol., vol. 98, no. 5, pp. 2974–2982, Nov. 2007. [24] J. J. Baker et al., “Decoding individuated finger flexions with Implantable MyoElectric Sensors,” Conf. Proc. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. IEEE Eng. Med. Biol. Soc. Conf., vol. 2008, pp. 193–196, 2008. [25] N. G. Hatsopoulos and J. P. Donoghue, “The science of neural interface systems,” Annu. Rev. Neurosci., vol. 32, pp. 249–266, 2009. [26] S. Micera et al., “Decoding of grasping information from neural signals recorded using peripheral intrafascicular interfaces,” J. NeuroEngineering Rehabil., vol. 8:53, 2011. [27] L. R. Hochberg and D. M. Taylor, “Intuitive prosthetic limb control,” The Lancet, vol. 369, no. 9559, pp. 345–346, Feb. 2007. [28] L. H. Smith, T. A. Kuiken, and L. J. Hargrove, “Evaluation of linear regression simultaneous myoelectric control using intramuscular EMG,” IEEE Trans. Biomed. Eng., vol. 63, no. 4, pp. 737–746, Apr. 2016. [29] C. Cipriani, J. L. Segil, J. A. Birdwell, and R. F. Weir, “Dexterous control of a prosthetic hand using fine-wire intramuscular electrodes in targeted extrinsic muscles,” IEEE Trans. Neural Systems Rehab. Eng., vol. 99, pp. 1–1, 2014. [30] L. J. Hargrove, K. Englehart, and B. Hudgins, “A comparison of surface and intramuscular myoelectric signal classification,” IEEE Trans. Biomed. Eng., vol. 54, no. 5, pp. 847–853, May 2007. [31] E. N. Kamavuako, E. J. Scheme, and K. B. Englehart, “On the usability of intramuscular EMG for prosthetic control: A Fitts’ Law approach,” J. Electromyogr. Kinesiol., vol. 24, no. 5, pp. 770–777, Oct. 2014. [32] Warwick K, Gasson M, Hutt B, and et al, “The application of implant technology for cybernetic systems,” Arch. Neurol., vol. 60, no. 10, pp. 1369–1373, Oct. 2003. [33] L. J. Hargrove, G. Li, K. B. Englehart, and B. S. Hudgins, “Principal components analysis preprocessing for improved classification accuracies in pattern-recognitionbased myoelectric control,” IEEE Trans. Biomed. Eng., vol. 56, no. 5, pp. 1407– 1414, May 2009. [34] G. Purushothaman and K. K. Ray, “EMG based man–machine interaction—A 17 pattern recognition research platform,” Robot. Auton. Syst., vol. 62, no. 6, pp. 864– 870, Jun. 2014. [35] J.-U. Chu, I. Moon, and M.-S. Mun, “A real-time EMG pattern recognition system based on linear-nonlinear feature projection for a multifunction myoelectric hand,” IEEE Trans. Biomed. Eng., vol. 53, no. 11, pp. 2232–2239, Nov. 2006. [36] L. Hargrove, Y. Losier, B. Lock, K. Englehart, and B. Hudgins, “A real-time pattern recognition based myoelectric control usability study implemented in a virtual environment,” in Engineering in Medicine and Biology Society, 2007. EMBS 2007. 29th Annual International Conference of the IEEE, 2007, pp. 4842–4845. [37] K. Englehart, B. Hudgins, P. A. Parker, and M. Stevenson, “Classification of the myoelectric signal using time-frequency based representations,” Med. Eng. Phys., vol. 21, no. 6–7, pp. 431–438, Jul. 1999. [38] Y. Huang, K. B. Englehart, B. Hudgins, and A. D. Chan, “A Gaussian mixture model based classification scheme for myoelectric control of powered upper limb prostheses,” IEEE Trans. Biomed. Eng., vol. 52, no. 11, pp. 1801–1811, 2005. [39] “Coapt,” Coapt. [Online]. [Accessed: 07-Aug-2017]. Available: http://www.coaptengineering.com/. [40] W. Wu, M. J. Black, D. Mumford, Y. Gao, E. Bienenstock, and J. P. Donoghue, “Modeling and decoding motor cortical activity using a switching Kalman filter,” IEEE Trans. Biomed. Eng., vol. 51, no. 6, pp. 933–942, Jun. 2004. [41] W. Wu, Y. Gao, E. Bienenstock, J. P. Donoghue, and M. J. Black, “Bayesian population decoding of motor cortical activity using a Kalman filter,” Neural Comput., vol. 18, no. 1, pp. 80–118, 2006. [42] J. Egan, J. Baker, P. A. House, and B. Greger, “Decoding dexterous finger movements in a neural prosthesis model approaching real-world conditions,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 20, no. 6, pp. 836–844, Nov. 2012. [43] D. J. Warren et al., “Recording and decoding for neural prostheses,” Proc. IEEE, vol. 104, no. 2, pp. 374–391, Feb. 2016. [44] W. Q. Malik, W. Truccolo, E. N. Brown, and L. R. Hochberg, “Efficient decoding with steady-state kalman filter in neural interface systems,” IEEE Trans. Neural Syst. Rehabil. Eng. Publ. IEEE Eng. Med. Biol. Soc., vol. 19, no. 1, pp. 25–34, Feb. 2011. [45] T. S. Davis et al., “Restoring motor control and sensory feedback in people with upper extremity amputations using arrays of 96 microelectrodes implanted in the median and ulnar nerves,” J. Neural Eng., vol. 13, no. 3, p. 36001, Jun. 2016. 18 [46] A. Branner and R. A. Normann, “A multielectrode array for intrafascicular recording and stimulation in sciatic nerve of cats,” Brain Res. Bull., vol. 51, no. 4, pp. 293– 306, Mar. 2000. 19 Figure 1.1 Schematic diagram of our decode approach to decoding neuromuscular signals. Starting in the top left and proceeding counter-clock-wise, 1) Data are collected from USEAs and iEMGs implanted in the subject’s residual nerves and muscles while the subject imitated single-DOF virtual hand movements with their phantom hand. 2) Spike times are recorded and are converted into firing rates. Power is calculated from EMG signals. 3) Firing rates and EMG power for each channel are correlated with the intended movement amplitude. 4) Channels with high correlation between firing rate and intended movement are chosen as inputs to the decode algorithm. 5) Data from several trials of selected movements and electrodes are used to train the decode algorithm. 6) Control of the selected movements of the virtual hand is turned over to the subject. 20 Figure 1.2 Top panel: Implant locations for S3-S6. USEAs were implanted into distal median and ulnar nerves, near the neuroma at the end of the stump. USEAs were implanted proximally in the upper arm in median and ulnar nerves for S4-S6. S5 and S6 were additionally implanted with an 8-lead, 32-channel IEMG in the residual forearm muscle. Lower left panel shows arrays implanted in median and ulnar and covered with a protective wrap. Lower right panel shows an X-ray of the IEMG leads in the residual forearm of S5. 21 Figure 1.3 Schematic diagram of the USEA implanted into a peripheral nerve where electrode tips penetrate multiple fascicles and varying depths [46]. Left image shows a longitudinal view. Right image shows the cross section. Extrafascicular tissue is represented in yellow, and fascicular tissue in grey. Electrode tip positions are represented as dots. 22 Figure 1.4 Depiction of the 32-channel IEMG array (Ripple, LLC). In this prototype (not used in these experiments but similar in design), there are 8, variable-length leads with 4 cylindrical electrodes (light grey in color) at the distal end of the wires. Each lead is independent and flexible. At the far distal end of each lead, flexible plastic barbs (shown in blue) help to anchor the leads in place for the duration of the implant. At the proximal end of the prototype shown here, a wireless signal amplifier/digitizer is shown as the white “puck.” In contrast, the prototype used in our experiments contained leads which were bundled together and attached to a “Gator” board. The wire bundle passed through a percutaneous opening in the user’s arm. The board was connected to the Ripple Grapevine™ data acquisition system via either “Active Gators” or “Micro2+Stim” front ends (with a “Passive Gator” adaptor). Gator board and connectors are not shown. CHAPTER 2 RESTORATION OF MOTOR CONTROL AND PROPRIOCEPTIVE AND CUTANEOUS SENSATION IN HUMANS WITH PRIOR UPPER-LIMB AMPUTATION VIA MULTIPLE UTAH SLANTED ELECTRODE ARRAYS (USEAS) IMPLANTED IN RESIDUAL PERIPHERAL ARM NERVES This manuscript was submitted to the Journal of Neural Engineering and Rehabilitation (JNER) in May 2017. Work relating to neural stimulation, including Figures 2.1, 2.9, 2.10, Table 2.2, and several sections of text are also present in Dave Page’s PhD. dissertation [1]. Dave Page and I are co-first authors on this manuscript, contributed equally to the work, and thus, much of our work overlapped. Contributing Authors: Suzanne Marie Wendelken, David M. Page, Tyler Davis, Heather A. C. Wark, David T. Kluger, Christopher Duncan, David J. Warren, Douglas T. Hutchinson, Gregory A. Clark 2.1 Abstract 2.1.1 Background Despite advances in sophisticated robotic hands, intuitive control of and sensory feedback from these prostheses has been limited to only 3-degrees-of-freedom (DOF) with 24 2 sensory percepts in closed-loop control. A Utah Slanted Electrode Array (USEA) has been used in the past to provide up to 81 sensory percepts for human amputees. Here, we report on the advanced capabilities of multiple USEAs implanted in the residual peripheral arm nerves of human amputees for restoring control of 5 DOF and sensation of up to 131 proprioceptive and cutaneous hand sensory percepts. We also demonstrate that USEArestored sensory percepts provide a useful source of feedback during closed-loop virtual prosthetic hand control. 2.1.2 Methods Two 100-channel USEAs were implanted for 4-5 weeks, one each in the median and ulnar arm nerves of two human subjects with prior long-duration upper-arm amputations. Intended finger and wrist positions were decoded from neuronal firing patterns via a modified Kalman filter, allowing subjects to control many movements of a virtual prosthetic hand. Additionally, USEA microstimulation was used to evoke numerous sensory percepts spanning the phantom hand. Closed-loop control was achieved by stimulating via an electrode of the ulnar-nerve USEA while recording and decoding movement via the median-nerve USEA. 2.1.3 Results Subjects controlled up to 12 degrees-of-freedom during informal, ‘freeform’ online movement decode sessions, and experienced up to 131 USEA-evoked proprioceptive and cutaneous sensations spanning the phantom hand. Independent control was achieved for a 5-DOF realtime decode that included flexion/extension of the thumb, index, middle, and 25 ring fingers, and the wrist. Proportional control was achieved for a 4-DOF realtime decode. One subject used a USEA-evoked hand sensation as feedback to complete a 1-DOF closedloop virtual-hand movement task. There were no observed long-term functional deficits due to the USEA implants. 2.1.4 Conclusions Implantation of high-channel-count USEAs enables multidegree-of-freedom control of virtual prosthetic hand movement and restoration of a rich selection of both proprioceptive and cutaneous sensory percepts spanning the hand during the short 4-5 week post-implant period. Future USEA use in longer-term implants and in closed-loop may enable restoration of many of the capabilities of an intact hand while contributing to a meaningful embodiment of the prosthesis. 2.1.5 Keywords prosthetic hand , neural interface, motor decode, nerve stimulation, sensory feedback , amputee, neural prosthesis, peripheral nerve, haptics, phantom-limb syndrome 2.2 Background Amputees using commercially-available mechanical or robotic prostheses do not currently receive cutaneous or proprioceptive sensory feedback from their prostheses, nor do they have simultaneous, independent, proportional control over all the digits of the prosthetic hand and the wrist. Sensory feedback from, and dexterous control of, a prosthetic robotic hand may assist upper-limb amputees in activities of daily living (ADL), restore a 26 sense of prosthesis embodiment, and alleviate phantom pain [1]–[7]. As early as 1974, amputees were instrumented with a single cuff-like electrode on their residual median nerve, which produced limited sensations in the phantom hand via electrical stimulation [4]. More recently, implanted longitudinal intrafascicular electrodes (LIFEs) were implanted into the peripheral arm nerves of several transradial amputees, and recordings from these electrodes provided subjects with one-degree-of-freedom (DOF) online control of a prosthesis [5]. Additionally, a limited number of sensations were evoked in the phantom hand by electrical stimulation via LIFE electrodes [5]–[7]. LIFE recordings were later used to achieve 3-DOF control of a prosthetic hand, including coordinated grips [8], [9], and basic object discrimination was enhanced by use of two sensory percepts elicited from electrical stimulation of the peripheral nerve via LIFEs [10]. Cuff electrodes (flat interface nerve electrodes, FINEs), implanted around each of the three major residual arm nerves of an amputee, have also been used to evoke 19 sensory percepts, and these percepts have been shown to be stable for up to two years [11]. The Utah Electrode Array (UEA) has been previously implanted in the distal median nerve of an intact individual and used to provide 1-DOF decode and limited sensory feedback in a closed-loop interface[12], [13]. However, it is unclear if such an approach would work on transradial amputees who have modified physiology in their residual arm. Finally, a recent closed-loop system has been demonstrated in which an amputee achieved 3-DOF control of a prosthetic hand using surface electromyography (sEMG) for motor control and transverse intrafascicular multichannel electrodes (TIMEs) implanted in residual arm nerves to provide sensory feedback in two phantom-hand locations [18]. Previously, we demonstrated that a single USEA implanted in a residual peripheral 27 arm nerve in human amputees can be used to evoke up to 81 different cutaneous percepts on the hand and provide proportional motor control of up to two DOFs [15]. These past subjects, referred to here as S1 and S2, were each instrumented with only one USEA, implanted at the terminal end of either the residual median or ulnar nerve, respectively. Preliminary results regarding multi-USEA instrumentation in two residual arm nerves of a third subject, S3, have also been presented [16]–[18], demonstrating cutaneous sensory percepts spanning the phantom hand, limited 2-DOF online motor control, and basic closed-loop control. In expansion of this work, we now present findings from two recent human subjects, S3 and S4. In addition to the use of two USEAs per subject (one in each of the median and ulnar arm nerves) for both S3 and S4, a notable improvement was achieved by implanting USEAs in S4 in the upper arm, proximal to extrinsic-hand-muscle nervebranches. This allowed for unprecedented dexterous hand control of up to 12 DOFs (informally quantified), and generation of numerous proprioceptive sensory percepts spanning the hand in addition to many USEA-evoked cutaneous percepts, totaling up to 131 percepts overall. We also report results regarding electrode and percept stability, successful automated electrode selection prior to motor decode, and a performance comparison of different decode algorithms. Preliminary reports of some of these findings have previously appeared [16], [19]–[21]. 28 2.3 Methods 2.3.1 Study volunteers Two transradial amputees, referred to here as subjects S3 and S4, were recruited in 2013 and 2014, respectively, and evaluated by a physician and psychologist for their willingness and ability to participate in the study (S1 and S2, published previously [15]). S3 was a 50-year-old left-dominant male, whose left arm had been amputated several centimeters proximal to the wrist 21 years prior, following a crush injury. S4 was a 36year-old ambidextrous male, with bilateral amputations several centimeters distal to the elbow 16 years prior, due to electrical injury. Baseline phantom limb surveys and medical histories were taken for each subject prior to the study. The surveys included assessment of the subjects’ perceived abilities to exert voluntary control over phantom movements, and perceive sensations (both painful and nonpainful) on their phantom limbs. Phantom pain was assessed on the basis of the duration, frequency, and intensity of pain episodes and this assessment continued during the duration of the implant period and for several months afterward. For the one-month period prior to the study, S3 was given a mirror box in order to practice the phantom-hand movements to be performed in the study [3]. Due to his being a bilateral amputee, S4 was unable use a mirror box and was instead given videos of hand movements to watch and imitate with his phantom hands. S3 continued his use of Gabapentin to relieve back pain throughout the study, which may affect peripheral-nerve activity. The study and consenting of human volunteers was approved by the University of Utah Institutional Review Board, the Salt Lake City Veterans Affairs Hospital Research and Development Service Center, and the Department of the Navy Human Research 29 Protection Program. 2.3.2 Device Two Utah Slanted Electrode Arrays (USEAs; Blackrock Microsystems, Salt Lake City, UT, USA) were implanted in each subject (one in the median nerve, one in the ulnar nerve). Each USEA consisted of 100 silicon microelectrodes arranged in a 10x10 grid on a 4x4 mm base, spaced at 400 µm, and varying in length from ~0.75 – 1.5 mm [22] (Figure 2.1A). Of the 100 electrodes on each USEA, 96 were used to record from and/or stimulate the nerve. Four electrodes near the corners of the USEA were used as an on-array electrical reference [23], and two separate looped platinum wires served as off-array electrical reference and ground leads. All implanted electrodes were wired via a percutaneous incision to a custom-developed printed circuit board designed to allow attachment to data acquisition and stimulation hardware via a ZIF-Clip-96 connector cable (Tucker-Davis Technologies Inc., Alachua, FL, USA). 2.3.3 Surgical procedures Prior to, and for several days following the implant procedure, subjects were given a prophylactic antibiotic (100 mg minocycline, 7 d b.i.d., starting the day before the implant surgery) which potentially improves the quality of chronic neuronal recordings [24]. Under general anesthesia, two USEAs were surgically implanted into each subject—one in the residual median nerve and one in the residual ulnar nerve (Figure 2.1B). In S3, both USEAs were placed in the lower arm, approximately 2 centimeters proximal to the amputation neuroma (Figure 2.1C). This distal location was used in S3 as an initial precautionary 30 measure, because nerves were not functionally attached at the distal implant locations. Hence, any nerve resection there would not compromise essential motor or sensory function. In S4, both USEAs were placed in the upper arm, approximately 2 centimeters proximal to the medial epicondyle. Importantly, the USEAs in S4 were proximal to many motor and sensory nerve-branch points, including branches to extrinsic hand muscles, thereby potentially providing a greater richness in motor and proprioceptive nerve fiber access. For S3, the surgical procedure involved the passage of the unprotected USEAs through a trocar from the percutaneous site to the implant site, which resulted in damage to four of the electrodes on the median nerve implant (and no documented damage to the ulnar nerve implant). A different USEA passage method was devised for S4, which involved securing the arrays inside a plastic tapered carrier for protection before passing them under the skin. There is no indication that any electrodes were damaged using this revised USEA passage method in S4. In both subjects, the epineurium was dissected from the surface of the nerves prior to pneumatic insertion of the USEAs [25]. The USEA wire bundle, ground, and reference wires were sutured to the epineurium (8-0 or 9-0 nylon suture), and a protective collagen wrap (AxoGen Inc., Alachua, FL, USA) was placed around the nerve, USEAs, and reference/ground wires. The wrap was secured with vascular clips and sutured to the epineurium for stability. After tourniquet removal, subjects were administered 0.1 mg/kg of dexamethasone intravenously to potentially mitigate the foreign body response and improve neural recording capability [26], [27]. Percutaneous wire-passage sites were re-dressed as needed throughout the study, 31 on at least a weekly basis. Antibiotic wound dressings (Biopatch, Ethicon US LLC, Somerville, NJ, USA) were placed directly over the percutaneous site throughout the study duration to reduce the risk of infection, although S4 did experience an infection from which he fully recovered (potentially due to an implant-related hematoma and/or via the percutaneous wire-passage site). After several weeks (4 weeks for S3, 5 weeks for S4), the USEAs were surgically explanted. In S3, the USEAs and neuromas were removed with the arrays still intact for histological analysis [28]. In S4, only the USEAs were removed due to their placement midway along the nerves in the upper arm. 2.3.4 Experiment setup Subjects returned for the first experimental session within 4 d of the USEA implant surgery. Experimental sessions were 1-6 h in duration, and were performed 3-5 d per week for 4 weeks for S3 and 5 weeks for S4. Experimental sessions typically included testing impedances of all USEA channels at the beginning of each session, followed by a recording/decoding session, a stimulation session, or both. 2.3.4.1 Impedance testing The impedance of each electrode on each USEA was measured in saline prior to implantation via one-week soak-testing using a custom-built impedance tester, at 1 kHz [29]. Impedances were also measured shortly before pre-implant sterilization using the NeuroPort System (Blackrock Microsystems, Salt Lake City, UT, USA) at 1 kHz. Impedance testing was subsequently performed in vivo at the beginning of each 32 experimental session using the NeuroPort System at 1 kHz. Impedance measurements were used to identify failed USEA electrodes/channels as well as to monitor the over time stability of working electrodes. We defined failed channels as those which had an impedance greater than or equal to 500 kΩ. Nonfailed channels were defined as channels that never had an impedance value above 500 kΩ across the implant duration. For each implanted USEA, we tested the null hypothesis that the number of failed USEA electrodes in a session does not change significantly across the implant duration, using a two-tailed Spearman’s rank correlation. Additionally, for each implanted USEA, we tested the null hypothesis that the impedance value for nonfailed electrodes does not change over time using a Friedman test followed by a post-hoc twotailed Wilcoxon’s signed-rank test between the first and last post-implant impedance testing sessions. The Friedman test served as a screening criteria; the Wilcoxon’s signedrank test was performed only if the Friedman test was statistically significant. Given the use of a cut-off value, and that impedance values vary on a nonnegative, logarithmic scale, we utilized nonparametric tests that did not assume normality. 2.3.4.2 Recording/decode Neural data collection was performed using the 128-channel NeuroPort System for S3 and either the NeuroPort System or the 512-channel Grapevine System (Ripple LLC, Salt Lake City, UT, USA) for S4. Continuous neural signals were band-pass filtered with cutoff frequencies of 0.3 Hz (1st-order high-pass Butterworth filter) and 7500 Hz (3rd-order low-pass Butterworth filter), and digitally sampled at 30 kHz. A digital high-pass filter was applied to sampled recordings (250 Hz, 4th-order Butterworth filter), and multiunit activity 33 was extracted by detecting threshold crossings of an adaptive, automated threshold, set to approximately negative 6 times the root mean square (RMS) of the signal. Spike-event times from each electrode were binned into 33.3 ms windows and converted into firing rates, which were then used as inputs to train and test a modified Kalman filter decode algorithm. In this application of the Kalman filter, we modified the decoder to impose a limit of -1 to 1 in order to prevent the decoder from exceeding the limits of the robotic or virtual hand (that is normalized from -1 to 1). Outputs of trained decode algorithms were used to provide the subjects with realtime control of the position of a simulated hand in a virtual environment [30] (Figure 2.1D). Here we define a degree-of-freedom (DOF) as the motion in a digit or the wrist in a single linear or rotational axis in either direction. Thus, a single DOF includes deviation from a rest position in both flexion and extension direction (e.g., a positive value indicates motion in the flexion direction and a negative value indicates motion in the extension direction). We define an individual ‘movement’ as a DOF including the directional component for each DOF (e.g., flexion and extension); consequently, there were twice as many possible individual ‘movements’ as possible individual DOFs. Although the virtual hand used in experiments had 24 actuating joints, interphalangeal joints were tied to the metacarpal phalangeal joints, giving the virtual hand a total of 12 DOF (flexion/extension of digits 1-5; adduction/abduction of digits 1, 2, 4, and 5; flexion/extension, ulnar/radial deviation, and pronation/supination of the wrist). The virtual model did not include adduction/abduction of digit 3. To train the decode algorithm, the subjects were instructed to imitate with their phantom hands a series of single-DOF virtual-hand movements shown on a computer 34 screen while USEA recordings were collected and saved. Training sets included 5 to 10 trials of each movement, with each movement trial lasting for 1 to 2 s (complete training session generally lasting 5-10 min.). The time from training-set completion to online decode testing was typically no longer than 5 to 10 min. Training was conducted at the start of a given day’s experimental recording session. During individual training motions, the experimenters manually selected a subset of electrode channels and movements by viewing electrode maps of spiking activity and selecting the electrode channels with greatest apparent correlation and specificity to a single movement. These electrodes were then used as inputs for training online decodes, whereas electrode channels with little or no firing that was correlated preferentially with single movements were excluded. On-line automated channel selection had not yet been implemented at the time of these experiments, but has been implemented since [19], [20]. Data from online training sessions were analyzed offline in order to compare performance of electrode selection methods, decode algorithms, and movement constraints. Electrode selection methods included manual and automated selection. Manual selection was performed during online decode experiments, and the same manually-selected electrodes were later used for offline decodes. Automated electrode selection was performed only for offline decodes, and involved selecting the electrodes that produced a correlation of at least 0.5 between firing rate and training movement cue position. For both online and offline decodes, at least five training trials, along with the associated firing rates of selected channels, were used to establish the coefficients of the decode algorithm. For offline decodes, the remaining five trials were used for validation testing of the algorithm. Offline decodes were performed using both automated electrode 35 selection and manual electrode selection for both a standard Kalman filter algorithm [31], or the ReFIT Kalman filter algorithm [32] (available for S4 only). These algorithms were chosen due to the stability of output in the presence of a noisy input signal. For each of up to 12 DOFs trained on, correlation coefficients between the decoded position and the intended position (half-cycle of a sine wave) were computed across all five trials. For each decode algorithm/configuration, the mean of the correlation coefficients across individual DOFs was computed to allow for comparison for S4. In the automated channel selection method, the algorithm selected only those electrodes that produced a correlation of at least 0.5 between firing rate and training movement cue position. Further details of the decoding algorithm are discussed elsewhere [20], [31]. Formal assessment of online decode performance was carried out via a virtual target-touching task. Specifically, one or more spherical virtual target(s) was positioned away from the resting position of one or more digit(s)/wrist along the arc of movement. The subject was then instructed to move the specified digit(s)/wrist inside the radius of the spherical target(s) for at least 250 ms while keeping the other DOFs in resting position. The virtual targets did not exclude the virtual fingers, and fingers could pass all the way through the spheres. Typically, a target diameter 15% of total range of motion was chosen during formal assessments. A trial was considered failed if the subject did not complete the task within a 30 s time-limit. In the case of closed-loop trials, a misclassification of the target’s distance by the subject was also considered a failure. After successful completion of a trial, virtual targets were automatically reset to their resting positions, and the subject was required to maintain all degrees-of-freedom in their resting positions for 1 s before another trial was presented. To verify proportional control while using a unchanging set of 36 decoding parameters, the subjects performed a similar task with targets located and held at several different positions along the trajectory of each DOF. Informal, “freeform” sessions in which intent was not objectively specified and hence errors were not directly measurable were also performed. RMS values of the decode output for all DOFs were calculated during a rest period. If the value of the decode output exceeded +/- 6*RMS during periods of intentional movement for a particular DOF, it was considered controlled by the subject. 2.3.4.3 Stimulation Electrical stimulation was performed using the IZ2-128 System (Tucker-Davis Technologies Inc., Alachua, FL, USA) for S3 and either the IZ2-128 System or the Grapevine System (Ripple LLC, Salt Lake City, UT, USA) for S4. For all USEA stimulation, biphasic, cathodic-first pulses were used (typically 200 µs width for each phase, 100 µs interphase interval). When a percept was evoked by USEA stimulation, subjects indicated the perceived location, quality, and intensity or size of the percept on an image of a hand using custom software (Figure 2.1E). Subjects were instructed to select the percept quality from a list of descriptors (e.g., ‘tingle’, ‘vibration’, ‘pressure’, ‘movement’, ‘hot’, ‘cold’) or to create and use their own descriptors as necessary. Full-USEA stimulation threshold maps were collected on weeks 1, 2, 3, and 4 for S3, and on weeks 2 and 5 for S4. For these maps, the threshold current (in µA) required to evoke a sensation via stimulation of each electrode was determined. Thresholds were defined as the minimum current level at which a subject repeatedly perceived stimulationevoked percepts. For these mappings, biphasic, 200-µs stimulus pulses (with a 100-µs 37 interphase interval) were delivered via single electrodes at 200 Hz for a 200-ms-duration train (the 200 Hz frequency was chosen empirically on the basis of the subjects’ ability to quickly reach threshold). The stimulation trains were initiated either by the experimenter or self-initiated by the subject via clicking a mouse button. Full-USEA threshold mapping sessions began by sequentially stimulating each electrode on the USEA individually with a low-amplitude stimulus (e.g., 2 µA), while documenting electrodes for which either a percept was evoked, or for which the voltage between the stimulating electrode and return electrode (looped platinum ground wire) did not return above the safety level of -0.6 V before the end of the interphase interval [33]. These electrodes were excluded from subsequent stimulation, whereas each of the remaining electrodes on the USEA was again sequentially stimulated at an incrementally higher current level. This pattern was repeated at increasing current levels until either there were no remaining unmapped electrodes, or the current reached a maximum threshold amplitude (varied between 35 µA and 120 µA depending on the subject and the session), at which point all remaining electrodes were excluded. For both subjects, full-USEA threshold mapping routines were performed at multiple times during the study, allowing for temporal stability analysis of the nature of percepts evoked by each electrode. Specifically, we quantified each USEA’s percept stability based on the percentage of electrodes on that USEA for which the evoked percept changed either location or quality between two consecutive full-USEA threshold-mapping sessions. For this analysis, a change in percept location was defined as a transition between any of 12 hand location categories (front/back of palm, and front/back of each of the 5 digits). A change in percept quality was defined as a transition between selected percept 38 quality descriptors. For S3, we computed the across-week mean of the number of electrodes that had a change in either percept quality or location from week to week. For S4, fullUSEA threshold maps were collected only on week 2 and week 5 due to time restrictions, and the percentage of electrodes which had a change in either location or quality between these two sessions was quantified. Additionally, we tested the null hypothesis that stimulation threshold currents for each electrode do not change significantly over time, using either a Friedman test with a post-hoc two-tailed Wilcoxon’s signed-rank test between the first and final threshold mapping sessions (for S3), or a two-tailed Wilcoxon’s signed-rank test (for S4, because there were only two full-USEA threshold mapping sessions). For S3, the Friedman test served as a screening criterion such that the Wilcoxon’s signed rank test was only performed if the Friedman test revealed a statistically significant temporal trend in the perceptual threshold. For each full-USEA threshold mapping session, we also calculated the percentage of median- and ulnar-nerve evoked percepts that were within the expected nerve-location distribution (based on muscular and cutaneous innervations documented in intact hands and arms [37], [38]). 2.3.4.4 Closed-loop control For S3, stimulation was delivered via a single electrode on the ulnar-nerve USEA during an online, one-DOF decode of simultaneous four-finger flexion produced via recordings on the median-nerve USEA. In a target acquisition task similar to others used for online decode testing, USEA-evoked sensory feedback was delivered whenever the virtual fingers were within virtual spherical targets, producing a basic sense of virtual- 39 object touch. Virtual targets were presented in a pseudorandom order in two different locations: ‘close’ or ‘far’, representing finger contact positions that were either close to, or far from, finger resting positions (equivalent to grasping a large-diameter or small-diameter object, respectively). For a successful trial, the subject was required to move the virtual fingers into the boundary of the virtual target and stay within the target zone for 250 ms and then correctly indicate whether the target was ‘close’ or ‘far’. Failed trials were those in which the subject either indicated the wrong distance to target, or failed to maintain 250 ms of consecutive contact with the virtual target before the 30 s time-limit. Importantly, these trials were performed in the absence of visual feedback from the computer monitor, presumably limiting feedback regarding contact with the virtual object to that evoked by USEA stimulation. 2.4 Results 2.4.1 Electrode impedances Implanted USEA electrodes had mixed resistance to failure over time. One array in each subject maintained a steady and high number (> 80) of working electrodes (impedances < 500 kΩ). The other array in both subjects showed a steady decline over the implant duration (Figure 2.2). The point of failure (e.g., electrode metallization, electrode insulation, array wiring, connector pin, etc.) for a given electrode was not determined. For each of the four USEAs, impedances on nonfailed channels (i.e., with impedance never > 500 kΩ) changed significantly over time (p < 0.0001, Friedman test, 2 dof (S4), 3 dof(S3); see Table 2.1). Post-hoc testing between the first and final post-implant sessions revealed a significant pairwise drop in impedance for electrodes on S4’s median- 40 nerve USEA (p < 0.0001; two-tailed Wilcoxon’s signed-rank test), but did not reveal a statistically significant pairwise change for the remaining 3 USEAs. 2.4.2 Decoding USEA recordings allowed intuitive control of many movements USEA recordings from individual days provided subjects with online proportional control of multiple DOF of the virtual hand. In formal evaluations, S4 controlled up to 5 DOFs independently, including flexion/extension of the thumb, index, middle, and ring fingers, as well as the wrist. Both subjects reported the experience of moving their fingers to be emotionally meaningful. In an informally-assessed online decode setting, S4 was able to control up to 12 DOFs, including flexion/extension of all 5 digits; abduction/adduction of the thumb, ring, and little fingers; and wrist flexion/extension and rotation (supplemental Video 1). In S4, multiple USEA electrodes displayed neuronal spiking activity concurrent with movement cues (Figure 2.3). The spike morphology and timing suggest that the activity was of neuronal rather than muscular origin (Figure 2.3B, inset). Additionally, the pattern of movement-correlated firing on USEA electrodes for a given individual training movement was unique, and differed for different individual movements (Figure 2.4). For S3, while neuronal and EMG spiking recorded from nearby muscles contributed to realtime decodes, EMG signals were dominant input recorded from the USEAs, typically limiting the performance of S3 decodes to 1-2 DOF (data not shown). USEA neural recordings demonstrated poor stability over 4 weeks for S4. The number of electrodes with activity correlated to movement (“driven electrodes”) peaked on day 13, and decreased until little neural activity was detected on day 30 (Figure 2.5). 41 Despite this, USEA recordings on individual days provided sufficient source of information for decoding. Formal assessments of online decode performance were carried out via a targettouching task. S4 demonstrated independent control of up to 5-DOFs, including flexion of the thumb, index, middle, and ring fingers, as well as the wrist (20/21 successful trials). See Figure 2.6 for the raw 5-DOF decode output and spike raster during several trials of each DOF tested. Notably, S4 was able to perform novel combination movements (e.g., thumb-index pinch) during multi-DOF online Kalman-filter decodes that had been trained on only single-DOF training movements (Figure 2.6 inlay e). Thus, dexterous, multi-DOF control can be achieved using a limited set of simplistic training data. Proportional position control was formally verified for a 4-DOF online decode in S4 by presenting targets at two different distances (‘near and ‘far’) on different trials for each DOF (40/40 successful trials). See supplemental Video 2 for an example of 4 DOF proportional control target-touching task in S4. For S3, proportional control was verified for a 2-DOF decode (46/52 successful trials). S4 also performed a target-tracking task in which he was instructed to follow virtual targets with specific DOFs as the targets moved in virtual space. Specifically, during a 3-DOF decode, S4 tracked targets independently with the thumb, index and middle finger, followed by combined-DOF target tracking (Figure 2.7). Informal sessions where the subject was allowed to control all 12 available DOF were also performed (Supplemental Video 1). Because the subject controlled the hand under “freeform” conditions in which intent was not objectively specified and hence errors were not directly measureable, this is considered an informal result. However, due to the 42 cross talk between several degrees of freedom, it is unlikely that the subject could have completed a formal 12-DOF target-touching task. 2.4.3 Offline decode performance With this approach, the automated selection algorithm selected 49 USEA electrodes for an offline decode of a training set from S4, in contrast to the 24 electrodes previously selected with manual electrode selection for this training set. The highest offline performance for a 12-DOF decode resulted from using automated electrode selection, a standard Kalman filter, and limiting movement to flexionranges only (i.e., digits were not allowed to extend in relation to baseline position). Specifically, the mean correlation between the intended position and decoded position across all 12 DOFs improved from 0.28 to 0.53 when automated electrode selection and flexion-only constraints were used (see Table 2.2). The correlation coefficient between the decoded finger position and the intended finger position generally decreased as the number of DOFs of the offline decode increased. Offline analysis was also performed on all possible DOFs in order to determine the “best possible” multi-DOF decode (Figure 2.8). Correlation coefficients between the decoded movement and intended movement were calculated for the top N simultaneous DOFs tested. Mean correlation coefficient values were greater than 0.8 for up to 7 simultaneous DOFs indicating a high level of independent DOFs could potentially be achieved during an online decode. 43 2.4.4 USEA microstimulation produced numerous sensations spanning the hand For each subject, microstimulation via USEA electrodes produced nearly 100 or more unique proprioceptive and cutaneous percepts that spanned the phantom hand, providing a rich selection of percepts potentially useful as feedback from a prosthetic limb. Importantly, subjects described many of the evoked sensations in a positive manner and sometimes asked for repeated delivery of pleasurable stimuli. In S4, 131 of 192 (68%) USEA electrodes produced proprioceptive or cutaneous sensory percepts spanning the hand (Figure 2.9a), and in S3, 97 of 192 (51%) USEA electrodes produced sensory percepts (primarily cutaneous). Percepts were evoked using different electrodes across the slanted 10x10 USEA. There was no apparent somatotopic arrangement across the nerve cross-section; however, we often observed fascicular organization (Figure 2.9b). Subjects also successfully discriminated among sensory percepts of different locations and qualities (a preliminary report for S3 has been provided [20], and a comprehensive report across multiple subjects is pending future publication). Proprioceptive percepts for S4 included 17 unique perceived phantom hand movements (i.e., proprioceptive percepts), including flexion and extension of each finger; adduction and abduction of the index, ring, and little fingers; thumb flexion; and wrist extension. In S3, a proprioceptive percept was evoked only once (presumably due to implant location). Cutaneous percepts were of many qualities, including ‘pressure’, ‘vibration’, ‘tingle’, and ‘sting’ (Figure 2.9c; ‘sting’ was described only by S3). Many percepts were naturalistic and enjoyable to the subjects (e.g., ‘vibration’ and ‘pressure’), whereas some 44 percepts were undesirable or nonnaturalistic (e.g., ‘sting’, and ‘tingle’). We compared subjects’ perceived percept location distributions for median- and ulnar-nerve percepts with the anatomically-determined median and ulnar innervation distributions of an intact hand reported in literature. For S3, on weeks 1-4, respectively, a total of 84%, 90%, 86%, and 95% of median- and ulnar-USEA percepts were within the expected anatomical innervation regions of the hand (Figure 2.10). For S4, on week 2 and week 5, respectively, 63% and 75% of median- and ulnar-USEA percepts were within their expected innervation regions (including the unique innervations for proprioceptive vs. cutaneous percepts). For both subjects, the location and quality of percepts evoked by single electrodes was generally stable during 3-4-h experimental sessions. However, single-electrode percepts often changed location and/or quality across weeks. Specifically, for S3, acrossweek means of 91% and 78% of ulnar- and median-USEA electrodes evoked percepts that changed either location or quality in a 1-week period, respectively (percentages are based on the 43 ulnar- and 17 median-nerve USEA electrodes that evoked percepts on all 4 weeks). For S4, 83% of the 12 median-nerve USEA electrodes that evoked percepts both on week 2 and week 5 changed either location or quality across this 3-week period. Importantly, no percepts were evoked via ulnar-nerve USEA stimulation on week 5, possibly due to infection-related swelling or USEA movement. Median stimulation thresholds (and interquartile ranges) for each USEA across the implant duration are provided for both subjects in Figure 2.11. For the 43 ulnar-nerve USEA electrodes on S3 that evoked percepts on all 4 weeks, threshold amplitudes changed significantly over time (p < 0.01, Friedman test). A post-hoc contrast test showed that 45 stimulation thresholds tended to increase on these electrodes between week 1 and week 4 (p < 0.01, two-tailed Wilcoxon’s signed-rank test). Similar significant increases were evident for the 17 median-nerve USEA electrodes that evoked percepts on all 4 weeks for S3 (p < 0.01, Friedman test, and p < 0.01, post-hoc two-tailed Wilcoxon’s signed-rank test). For S4, stimulation thresholds for full USEAs were mapped only on week 2 and week 5, due to limitations on experiment time. Notably, none of the electrodes on the ulnarnerve USEA evoked percepts on week 5. For the 12 median-nerve USEA electrodes that evoked percepts on both week 2 and week 5 there was no significant evidence of changing thresholds over time (p = 0.11, two-tailed Wilcoxon’s signed-rank test). 2.4.5 USEA-evoked sensations can be used for closed-loop control During two closed-loop target-touching sessions, S3 used a cutaneous sensation on his ring fingertip (evoked by stimulation of a single ulnar-nerve USEA electrode) as feedback to determine the location of the target in virtual space which was placed in either close to the finger tips from the neutral start position (‘close’), or further along the arc of flexion of the finger tips (‘far’). In this task, S3 controlled flexion/extension of fingers 1-4 (fingers linked together into 1 DOF, decode via median-nerve USEA recording, driven by both neural and EMG signals). In the absence of visual feedback from the computer monitor, the subject successfully moved the fingers into the target region and identified the location (‘close’ or ‘far’) of virtual targets in 41/47 trials. See Figure 2.12 for a confusion matrix of session 2, and Figure 2.13 for an example of decode output in this closed-loop task. In order to successfully perform this task, S4 used the USEA-restored sensation as 46 feedback in addition to proprioceptive feedback from intact muscles of the forearm and/or efference copy to determine hand position (see supplemental Video 3). Of the 6 failed trials, 2 resulted from timeouts and 4 resulted from misclassifications. The subject’s successful classification of ‘close’ versus ‘far’ (p < 0.001, binomial test), along with subject’s verbal report of differences in the muscular effort required to move to the different positions, suggests that the subject could use residual function proprioceptive feedback or efference copy to identify his hand position (as distinct from the cutaneous sensory percept evoked by the USEA stimulation), despite not being explicitly provided by the experimenters. 2.4.6 Subjects describe their experience in a positive manner Both subjects appeared to enjoy the experiments, as evidenced by their verbal expressions and eagerness to volunteer again for future studies. When asked if the USEA stimulation was something he would want to continue simply because it felt good, S3 responded: “Yeah. I would like it if you could keep it stimulated.” Following an online decode, S4, whose hands had been amputated 16 years prior, stated, “[…] when I tried to move my thumb and the thumb moved on the screen—that was the coolest thing that’s happened to me in 16 years.” 2.4.7 Limited adverse effects S4 developed an implant-related infection 4-5 weeks post-implant, from which he fully recovered, and from which he suffered no long-term deficits. Both subjects reported no long-term functional deficits due to the procedure, with a full return of phantom hand 47 function to its pre-implant state after explantation of USEAs (data not shown). 2.5 Discussion We used USEAs implanted in peripheral arm nerves to: 1) provide subjects with independent, proportional position control of movement of many degrees of freedom via a virtual prosthetic hand (5 DOFs in formal testing, up to 12 informally); 2) evoke numerous meaningful proprioceptive and cutaneous percepts across subjects’ phantom hands (up to 131). The number of DOFs achieved and the number of percepts evoked are greater than achieved in previous work with USEAs or other neural interfaces after amputation. We also provided one subject with limited closed-loop control of a virtual prosthetic hand. No long-term deficits were reported by the subjects after explant, although one subject experienced an implant-related local infection from which he recovered fully. Future implants should employ use of improved percutaneous site maintenance and/or wireless, nonpercutaneous implants to help prevent infections. Both subjects appeared to enjoy having control over finger movements and experiencing phantom hand sensations evoked by microstimulation. 2.5.1 Impedance These results suggest that some USEAs may maintain a low-impedance condition in future long-duration implant studies, potentially allowing for chronic use of multichannel neuronal recordings for decoding movements and intraneural stimulation for providing sensory feedback. However, failures potentially may occur at the electrode level, the wire-bundle level, or the connector level. Failure rates may be improved in future 48 implants with improved external connectors, additional strain relief for USEA lead wires, and wireless devices. 2.5.2 Decode Both subjects demonstrated proportional control of a virtual prosthetic hand via multi-DOF decodes. S4 had higher-DOF decodes compared with S3. This improvement may have been due, in part, to implanting the USEAs proximal to the motor nerve branches to extrinsic hand muscles. Additionally, the higher amplitude EMG signals recorded from the distal site of S3 masked much of the neural activity, limiting the performance of online and offline decodes for S3. Similar EMG spiking has previously been reported for intraneural recordings [5]. Further offline processing revealed that neural activity can be recovered from the USEA signals using virtual referencing techniques [36]. Online decodes were driven by neuronal activity in S4, which, when decoded with a Kalman filter, provided independent proportional control of numerous movements (5 or more DOFs). This is in contrast to past approaches using EMG signals and/or classifier decodes, which have been limited to only 4 DOFs [37]. Furthermore, neural decodes offered control of several intrinsic hand and thumb movements, which would be inaccessible using EMG recordings from a typical amputated arm. In contrast to past, lowchannel-count neural interface decodes, such as those performed using LIFEs and TIMEs, which have been limited to 3 DOFs, the high channel-count of the Utah Electrode Array has allowed us to provide subjects with high-DOF decodes (12 DOFs, informally), allowing restoration of control of combination movements and dexterous finger manipulations. 49 We also demonstrated that combination movements can be generated using an online Kalman decode trained with a limited number of simplistic, single-DOF movements. Training was performed in less than 5 min. Kalman decoding of this training can provide subjects with meaningful control of complex hand grips, pinches, and grasps as well as control of individual DOFs. In chronic implants, the short duration of training is important because training sessions may need to be carried out on a daily or weekly basis. The ability to generate novel grasps on the basis of simple training sets increases the functional range of useful movements for activities of daily living without increasing the time necessary for training. Future improvements to online decodes should include incorporation of automated electrode selection algorithms, which improved performance of offline decodes performed after explant. Successful decodes leveraged information from subpopulations of several USEA electrodes, in contrast to using single channels for each movement. However, we observed subjectively that inclusion of too many USEA channels generally seemed to result in poor decode performance. An effective automated electrode selection algorithm should select all channels that produce relevant activity, while excluding channels that have no activity or new information relevant to the movements. In subsequent work, we have begun to implement on-line automated channel selection [19], [20], [38]. The ability of the USEAs to record neural activity in S4 substantially declined from post-implant day 13 to 30. The decline may have been due to the decreased number of working electrodes, foreign body response to the USEA, array movement relative to the nerve, and/or fluid buildup in the intraneural space resulting from infection. The periphery is a harsh environment for the USEA. The median and ulnar nerves are subject to stretch, 50 torsion, and compression during elbow bending, which was not restricted for S4. Such perturbation may cause the array to shift or pull out of the nerve. Due to the proximal location of the implant in S4, histological analysis of the intact array and nerve segment could not be performed to investigate array position with respect to the nerve at the end of the study. Additionally, MRI could not be used to image the arrays in situ as the USEA is not MRI-compatible. Although we have demonstrated the feasibility of using neural recordings from a peripherally-implanted USEA for realtime multi-DOF decoding, the general inability to detect motor-driven neural activity 30 d after implant precludes the use of USEAs for recording purposes in a commercially-viable prosthetic hand. 2.5.3 Stimulation Microstimulation via USEAs produced a rich selection of up to 131 different proprioceptive and cutaneous percepts spanning the hand. USEA stimulation required no long-term training or reassociation or substitution of sensations. Proprioceptive percepts included flexions and extensions of each finger, flexion of the thumb, several intrinsic finger movements, and wrist extension. The improved ability to produce proprioceptive percepts in S4 compared with past subjects was likely due to placement of USEAs proximal to extrinsic hand muscle motor branches in S4. In addition to restoring much of the functionality of an intact hand to amputees, quasi-continuous restoration of the sense of proprioception and cutaneous touch may help amputees perceive their prosthesis as an embodied replacement limb rather than a tool [2], which may decrease prosthesis rejection rates and improve amputees’ perception of the usability of the device [39]. Our subjects seemed to appreciate both the cutaneous and 51 proprioceptive sensations evoked by USEA stimulation. The high percentages of percepts in expected median and ulnar distributions suggests that cortical boundaries between median- and ulnar-nerve innervation regions for these subjects were still partially intact despite the amputation greater than 16 years prior. However, some projected fields for USEA-evoked cutaneous percepts spanned the edges of two adjacent digits, suggesting the possibility of blurring of digit boundaries in cortex. Importantly, proprioceptive percepts were more common in S4 compared with previous subjects, presumably due to implantation of USEAs midway along the upper arm, proximal to many nerve branches to the extrinsic hand-muscles. We did not perform exhaustive testing of the effect of stimulation frequency on percept quality, location, intensity, and/or size. Future work should be performed to encode percept properties such as pressure gradations, joint angles, or joint velocities, via modulation of stimulation parameters, such as stimulation frequency. Additionally, activation of subpopulations of afferents with stimulation patterns faithful to each respective receptor type (e.g., slowly-adapting I type or II, rapidly-adapting type I or II, or group Ia or II intrafusal muscle fibers) may improve the naturalism, discriminability, and stability of percepts [40]. Naturalistic touch, such as the sensation experienced during motor task phase transitions, activates a diverse subpopulation of axons in distinct patterns, producing a fused population and temporal code [41]. In contrast to cuff electrodes, USEAs offer the opportunity to activate subpopulations of single axons in biofidelic patterns via independent control of stimulation via different electrodes, potentially offering unprecedented naturalism and variety in the nature of evoked percepts. Notably, the stimulation thresholds for this study were mapped using the ascending 52 method of limits. This method was selected to reduce the amount of time required to map the sensory perception threshold for the nearly 200 USEA electrodes, and because we were unable to reliably anticipate the expected level of the threshold values via USEAs prior to these subjects and wanted to avoid delivering overly strong, potentially painful stimuli. One disadvantage of this approach is that some observers may become accustomed to indicating that they do not perceive a sensation during initial subthreshold stimuli, which may result in a higher false-negative reporting rate when perithreshold amplitudes are reached. For other observers, the opposite phenomenon may occur, in which the observer may make a premature judgement of arrival at threshold (increased false-positive reporting rate at perithreshold amplitudes [42]). Alternative approaches that at least partially resolve some of these limitations include the staircase procedures or averaging of the thresholds identified with the method of ascending limits and the method of descending limits. Instabilities of percepts over time may be due to movement of the USEA electrodes relative to nerve fibers or due to the tissue foreign body response. Both potential issues may be ameliorated as improvements are made to the implantation procedure and the USEA materials and structure, and with longer implant times as processes reach asymptote. Further research is warranted to investigate and potentially improve USEA stability over time. Improved sensory percept stability for USEA-evoked sensations will need to be demonstrated for them to be functionally useful as a source of prosthesis sensory feedback. 2.5.4 Closed-loop control This is the first use of USEAs for closed-loop control of a virtual prosthetic hand in transradial amputees. Future closed-loop control with multi-DOF decodes and several 53 unique sensory percepts may allow for dexterous manipulations with a prosthetic hand. Although we did not provide USEA-evoked proprioceptive feedback during closed-loop control for these subjects, we anticipate that this capability may be important in cases where the prosthesis encounters external counterforces, or when velocity control is desired (instead of position control). Importantly, improved methods for evaluating the extent and usefulness of closed-loop prosthesis control, including comparisons with control trials, need to be developed and implemented. Ultimately, we foresee development of a portable, wireless system (i.e., no percutaneous wires) with USEA-enabled closed-loop control of a physical robotic hand that subjects may take home for use in activities of daily living [43]. Closed-loop control of multiple DOFs of a robotic prosthetic hand with graded feedback from multiple cutaneous and proprioceptive sensors via USEAs may allow users to perform activities of daily living while paying little visual attention to their prosthesis, or engage in tasks for which visual feedback is not readily possible (e.g., grasping the back side of an opaque object). In addition to restoring lost function, chronic use of such a device may transform subjects’ perception of their prosthesis from simply being a useful tool to being an integral part of their body. Embodiment of a prosthesis may not only reduce prosthesis rejection rates, but may also alleviate phantom limb pain and contribute to a restored sense of wellbeing and completeness [1], [44], [45]. The subject’s ability to correctly classify ‘close’ versus ‘far’ degrees of hand closure in the present study implies that he could successfully use of proprioceptive feedback from residual extrinsic hand muscles in the forearm, without these signals having been explicitly provided by the USEA neural interface. However, proprioceptive feedback 54 would not be present for intrinsic hand muscles, or in the case of transhumural amputation (for which extrinsic hand muscles would be missing), or in the case where external forces deflected the position of the prosthetic hand, suggesting that neural interfaces that provide proprioceptive feedback could still prove useful. 2.5.5 Study limitations There are several limitations to this study including a small sample of two subjects, short implant durations (< 6 weeks), confounding factors to the online decode performance, and the unstable recording/stimulating ability of the USEAs. Short implant durations do not allow for the long-term assessment of electrode recording or stimulation stability. Confounding factors to online decode performance include the number of available units on a given day, the ability of the experimenters to visually identify correlated firing rates, and on the ability of the subject to accurately repeat movements during training sessions, which may change over time. Such limitations are partially addressed in subsequent and ongoing, longer-duration studies [19], [21], [46] . Results of the present study are intended partly as proof-of-concepts, rather than demonstrating long-term viability or full functionality in activities of daily living. 2.6 Conclusions We have demonstrated that recording and stimulation via multiple USEAs implanted in the peripheral arm nerves 3 of human amputees can provide subjects with both 1) simultaneous proportional movement control of the digits and wrist of a virtual prosthesis; 2) a rich selection of proprioceptive and cutaneous sensations spanning the 55 phantom hand. Our achievement of a 5-DOF decode and 131 USEA-evoked cutaneous and proprioceptive percepts exceeds what has previously been accomplished with neural implants in the peripheral nerves of transradial amputees. Furthermore, we demonstrated that USEA stimulation and recording can be used for closed-loop control of a virtual prosthesis. Further investigation is warranted to demonstrate meaningful and repeatable closed-loop prosthesis control. No long-term functional deficits were reported by our subjects, although the implant did lead to a local infection in S4 that resolved with antibiotic treatment and explant of the devices. The subjects described the microstimulation-evoked sensations on their phantom hand and moving the virtual prosthesis in a positive manner. However, improved stability of these sensory percepts will be necessary in order for them to be functionally useful for prosthesis feedback. Future work will include development of automated channel selection and improved signal pre-processing algorithms for movement decodes, and use of biofidelic stimulation patterns and encoding of percept intensity gradations for sensory encodes. Ultimately, we expect that USEA-restored sensation and motor control could be used in closed loop as part of a robotic upper-limb prosthesis that amputees may take home for use in activities of daily living. 2.7 List of abbreviations DOF: Degree Of Freedom EMG: Electromyogram IQR: Interquartile Range RMS: Root Mean Square S1-S4: Subject 1 – Subject 4 56 USEA: Utah Slanted Electrode Array 2.8 Declarations 2.8.1 Ethics approval and consent to participate This study was approved by the University of Utah Institutional Review Board (IRB study 00055621). All subjects consented willingly to participate in the study. 2.8.2 Funding This research was sponsored by the Defense Advanced Research Projects Agency (DARPA) Microsystems Technology Office (MTO) under the auspices of Dr. Jack Judy, and Biological Technologies Office (BTO) Hand Proprioception and Touch Interfaces (HAPTIX) program under the auspices of Dr. Doug Weber, both through the Space and Naval Warfare Systems Center, Pacific; contract nos. N66001-12-C-4042 and N6600115-C-4017, respectively. Additional funding was provided via the National Institutes of Health (NIH NCATS Award No. 1ULTR001067). 2.8.3 Authors’ contributions DMP planned and carried out experiments, performed data analysis related to sensory percepts (stimulation) and impedances, and helped draft the manuscript. SW planned and carried out experiments, performed data analysis related to motor decodes (recording) and impedances, assisted with experimental software development, and helped draft the manuscript. TD designed and constructed the software used in the experiments, and helped carry out many of the experiments. HACW helped plan and carry out 57 experiments for S3. DTK performed pre-implant USEA impedance testing and packaging and assisted with data analysis. CD assisted in pre-implant and post-explant clinical assessments for S4, and provided general clinical consulting. DJW helped plan experiments and designed much of the hardware used in the experiments. DTH helped design the study, performed surgical implant and explant procedures, recruited and consented subjects, and managed IRB approvals and regulatory affairs. GAC helped design the study, coordinated the project, helped carry out many of the experiments, and helped manage IRB approvals, other regulatory affairs, and funding. 2.9 References [1] D. M. Page, “Restored hand sensation in human amputees via utah slanted electrode array stimulation enables performance of functional tasks and meaningful prosthesis embodiment,” Ph.D. Thesis, University of Utah, 2016. [2] P. D. Marasco, K. Kim, J. E. Colgate, M. A. Peshkin, and T. A. Kuiken, “Robotic touch shifts perception of embodiment to a prosthesis in targeted reinnervation amputees,” Brain, vol. 134, no. 3, pp. 747–758, Mar. 2011. [3] V. S. Ramachandran and D. Rogers-Ramachandran, “Synaesthesia in phantom limbs induced with mirrors,” Proc. Biol. Sci., vol. 263, no. 1369, pp. 377–386, Apr. 1996. [4] R. A. F W Clippinger, “A sensory feedback system for an upper-limb amputation prosthesis.,” Bull. Prosthet. Res., pp. 247–58, 1974. [5] G. S. Dhillon, S. M. Lawrence, D. T. Hutchinson, and K. W. Horch, “Residual function in peripheral nerve stumps of amputees: Implications for neural control of artificial limbs,” J. Hand Surg., vol. 29, no. 4, pp. 605–615, Jul. 2004. [6] G. S. Dhillon, T. B. Krüger, J. S. Sandhu, and K. W. Horch, “Effects of short-term training on sensory and motor function in severed nerves of long-term human amputees,” J. Neurophysiol., vol. 93, no. 5, pp. 2625–2633, May 2005. [7] G. S. Dhillon and K. W. Horch, “Direct neural sensory feedback and control of a prosthetic arm,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 13, no. 4, pp. 468– 472, Dec. 2005. 58 [8] P. M. Rossini et al., “Double nerve intraneural interface implant on a human amputee for robotic hand control,” Clin. Neurophysiol., vol. 121, no. 5, pp. 777– 783, May 2010. [9] S. Micera et al., “Decoding of grasping information from neural signals recorded using peripheral intrafascicular interfaces,” J. NeuroEngineering Rehabil., vol. 8, p. 53, Sep. 2011. [10] K. Horch, S. Meek, T. G. Taylor, and D. T. Hutchinson, “Object discrimination with an artificial hand using electrical stimulation of peripheral tactile and proprioceptive pathways with intrafascicular electrodes,” IEEE Trans. Neural Syst. Rehabil. Eng. Publ. IEEE Eng. Med. Biol. Soc., vol. 19, no. 5, pp. 483–489, Oct. 2011. [11] D. W. Tan, M. A. Schiefer, M. W. Keith, J. R. Anderson, J. Tyler, and D. J. Tyler, “A neural interface provides long-term stable natural touch perception,” Sci. Transl. Med., vol. 6, no. 257, p. 257ra138-257ra138, Oct. 2014. [12] Warwick K, Gasson M, Hutt B, and et al, “The application of implant technology for cybernetic systems,” Arch. Neurol., vol. 60, no. 10, pp. 1369–1373, Oct. 2003. [13] M. Gasson, B. Hutt, I. Goodhew, P. Kyberd, and K. Warwick, “Invasive neural prosthesis for neural signal detection and nerve stimulation,” Int. J. Adapt. Control Signal Process., vol. 19, no. 5, pp. 365–375, 2005. [14] S. Raspopovic et al., “Restoring natural sensory feedback in realtime bidirectional hand prostheses,” Sci. Transl. Med., vol. 6, no. 222, p. 222ra19-222ra19, Feb. 2014. [15] T. S. Davis et al., “Restoring motor control and sensory feedback in people with upper extremity amputations using arrays of 96 microelectrodes implanted in the median and ulnar nerves,” J. Neural Eng., vol. 13, no. 3, p. 36001, Jun. 2016. [16] G. A. Clark et al., “Using multiple high-count electrode arrays in human median and ulnar nerves to restore sensorimotor function after previous transradial amputation of the hand,” in 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2014, pp. 1977–1980. [17] D. M. Page et al., “Restoration of sensory and motor hand function via two Utah Slanted Electrode Arrays (USEAs) in residual arm nerves after prior hand amputation,” in Neuroscience Meeting Planner, Washington, DC, 2014, p. 636.19. [18] S. Wendelken et al., “Partial restoration of sensorimotor function after hand amputation using multiple electrode arrays,” in Biomedical Engineering Society Annual Meeting, San Antonia, TX, 2014, p. 132. [19] Suzanne Wendelken et al., “Using multiple Utah Slanted Electrode Arrays (USEAs) to control 5-degrees-of-freedom of a virtual prosthetic hand and provide sensations 59 in the phantom hand for a human, transradial amputee,” presented at the Society For Neuroscience (SFN), Chicago, IL, 2015. [20] D. J. Warren et al., “Recording and decoding for neural prostheses,” Proc. IEEE, vol. 104, no. 2, pp. 374–391, Feb. 2016. [21] G. A. Clark et al., “Restoring cutaneous and proprioceptive somatosensory function with nerve stimulation after hand amputations,” presented at the 4th International Conference on Medical Bionics, Brisbane Australia, 2016. [22] A. Branner, R. B. Stein, and R. A. Normann, “Selective stimulation of cat sciatic nerve using an array of varying-length microelectrodes,” J. Neurophysiol., vol. 85, no. 4, pp. 1585–1594, Apr. 2001. [23] G. A. Clark, D. J. Warren, N. M. Ledbetter, M. Lloyd, and R. A. Normann, “Microelectrode array system with integrated reference microelectrodes to reduce detected electrical noise and improve selectivity of activation,” US8359083 B2, 22Jan-2013. [24] R. L. Rennaker, J. Miller, H. Tang, and D. A. Wilson, “Minocycline increases quality and longevity of chronic neural recordings,” J. Neural Eng., vol. 4, no. 2, pp. L1–L5, Jun. 2007. [25] P. J. Rousche and R. A. Normann, “A method for pneumatically inserting an array of penetrating electrodes into cortical tissue,” Ann. Biomed. Eng., vol. 20, no. 4, pp. 413–422, Jul. 1992. [26] L. Spataro et al., “Dexamethasone treatment reduces astroglia responses to inserted neuroprosthetic devices in rat neocortex,” Exp. Neurol., vol. 194, no. 2, pp. 289– 300, Aug. 2005. [27] Y. Zhong and R. V. Bellamkonda, “Dexamethasone coated neural probes elicit attenuated inflammatory response and neuronal loss compared to uncoated neural probes,” Brain Res., vol. 1148, pp. 15–27, May 2007. [28] M. B. Christensen, H. A. C. Wark, and D. T. Hutchinson, “A histological analysis of human median and ulnar nerves following implantation of Utah slanted electrode arrays,” Biomaterials, vol. 77, pp. 235–242, Jan. 2016. [29] K. Gunalan, D. J. Warren, J. D. Perry, R. A. Normann, and G. A. Clark, “An automated system for measuring tip impedance and among-electrode shunting in high-electrode count microelectrode arrays,” J Neuroscience Methods, vol. 178, no. 2, pp. 263–269, 2009. [30] R. Davoodi, C. Urata, M. Hauschild, M. Khachani, and G. E. Loeb, “Model-based development of neural prostheses for movement,” IEEE Trans. Biomed. Eng., vol. 60 54, no. 11, pp. 1909–1918, Nov. 2007. [31] W. Wu, M. J. Black, D. Mumford, Y. Gao, E. Bienenstock, and J. P. Donoghue, “Modeling and decoding motor cortical activity using a switching Kalman filter,” IEEE Trans. Biomed. Eng., vol. 51, no. 6, pp. 933–942, Jun. 2004. [32] V. Gilja et al., “A high-performance neural prosthesis enabled by control algorithm design,” Nat. Neurosci., vol. 15, no. 12, pp. 1752–1757, Dec. 2012. [33] S. F. Cogan, “Neural stimulation and recording electrodes,” Annu. Rev. Biomed. Eng., vol. 10, no. 1, pp. 275–309, 2008. [34] “Palmar Cutaneous Branch of Ulnar Nerve.” [Online]. Available: http://nervesurgery.wustl.edu/ev/upperextremity/ulnar/Pages/PalmarCutaneousNer ve.aspx. [35] H. Gray, “IX. Neurology. 6b. The anterior divisions. Gray, Henry. 1918. Anatomy of the Human Body.,” in Anatomy of the Human Body, Twentieth edition., Philadelphia: Lea & Febiger, 1918. [36] Z. B. Kagan et al., “Linear methods for reducing noise in peripheral nerve motor decodes,” presented at the Engineering in Medicine and Biology Society (EMBC), Orlando, FL, 2016. [37] C. Cipriani, J. L. Segil, J. A. Birdwell, and R. F. Weir, “Dexterous control of a prosthetic hand using fine-wire intramuscular electrodes in targeted extrinsic muscles,” IEEE Tran.s Neural Systems Rehab. Eng., vol. 99, pp. 1–1, 2014. [38] J. G. Nieveen et al., “Channel selection of neural and electromyographic signals for decoding of motor intent,” presented at the Myoelectric Control (MEC) Symposium 2017, Fredericton, New Brunswick, CA, 2017. [39] P. Zhang and N. Li, “The importance of affective quality,” Commun. ACM, vol. 48, no. 9, pp. 105–108, Sep. 2005. [40] H. P. Saal and S. J. Bensmaia, “Touch is a team effort: Interplay of submodalities in cutaneous sensibility,” Trends Neurosci., vol. 37, no. 12, pp. 689–697, Jan. 2014. [41] D. Perruchoud, I. Pisotta, S. Carda, M. M. Murray, and S. Ionta, “Biomimetic rehabilitation engineering: The importance of somatosensory feedback for brainmachine interfaces,” J. Neural Eng., vol. 13, no. 4, p. 41001, Aug. 2016. [42] G. Gescheider, “Chapter 3. The Classical Psychophysical Methods,” Psychophyiological Fundamentals. 3rd Ed Mahwah Lawrence Erlbaum Assoc., 1997. 61 [43] M. Ortiz-Catalan, B. Håkansson, and R. Brånemark, “An osseointegrated humanmachine gateway for long-term sensory feedback and motor control of artificial limbs,” Sci. Transl. Med., vol. 6, no. 257, p. 257re6-257re6, Oct. 2014. [44] C. Dietrich et al., “Sensory feedback prosthesis reduces phantom limb pain: Proof of a principle,” Neurosci. Lett., vol. 507, no. 2, pp. 97–100, Jan. 2012. [45] J. S. Schofield, K. R. Evans, J. P. Carey, and J. S. Hebert, “Applications of sensory feedback in motorized upper extremity prosthesis: a review,” Expert Rev. Med. Devices, vol. 11, no. 5, pp. 499–511, Sep. 2014. [46] S. M. Wendelken et al., “Motor decoding and sensory stimulation for upper-limb prostheses using implanted neural and muscular electrode arrays,” presented at the NANS2-Neural Interface Conference, Baltimore, MD, 2016. 62 Figure 2.1 USEAs implanted in human peripheral arm nerves were used to provide amputees with multi-DOF control of virtual prosthetic hand movement and restore numerous hand sensations. A) Scanning electron microscope image of a USEA [16]. B) Two USEAs were implanted in each subject (S4, shown here), one in each of the median and ulnar arm nerves. An organic nerve wrap, fastened with vascular clips, enclosed each USEA. C) USEA lead wires and ground and reference wires were connected to external connectors via a percutaneous incision (S3, shown here). D) USEA recordings were used to provide subjects with control of movement of a virtual prosthetic hand (S3, shown here). E) USEA stimulation was used to provide subjects with numerous sensations on the phantom hand. Subjects documented the nature of each sensation (location, quality, and intensity/size) using custom software. 63 Figure 2.2 Number of working electrodes (impedance < 500 kΩ) of 96 total recording electrodes per array over time for S3 and S4. The number of working electrodes for the ulnar array of S3 and median array of S4 is relatively stable, whereas the number of working electrodes steadily declines for the median array of S3 and ulnar array of S4. 64 Figure 2.3 USEA recordings were collected during a training session and used to train a decode (either a standard Kalman filter or a ReFIT Kalman filter). Subjects were then given online control of the virtual hand via realtime output from the trained decode. A) Training data were collected by recording via USEAs while the subject imitated preprogrammed, single-DOF or multi-DOF virtual hand movements with their phantom hand. B) Neuronal spiking was observed during intended movements (inset shows neuronal action potential waveforms). C) For each trial, the firing rate on a given electrode was computed and compared to the movement cue position via correlation. D) The correlation between firingrate and movement cue was determined for each USEA electrode across many trials of a given movement, and electrodes with high correlations and independent activation for specific movements were selected as input for an online decode. 65 Figure 2.4 Distinct patterns of USEA electrodes with firing rates correlated to movement position (across two electrode arrays, median array (m) and ulnar array (u)) are apparent for different movements (i.e., unique subpopulations of axons fire with specific movement efforts). Shown here are the patterns of the firing rates during movement cues for 2 DOFs (middle finger and wrist pitch) for S4. 66 Figure 2.5 Number of electrodes with driven units for S4. Electrodes with activity correlated to volitional movement (r > 0.5) were tabulated periodically throughout the experiment duration. The total number of driven electrodes peaks on post-implant day 13, then steadily decreases over time. 67 Figure 2.6 Decode output and raster plot during a 5-DOF target-touching task for S4. Each figure shows the target position (solid black line) and decode output (dashed and colored line) on the top 5 lines. DOFs displayed in order from the top are thumb flex, index flex, middle flex, ring flex, and wrist flex/extend. Displayed below in black hash marks is the raster plot from 19 electrodes shown (15 from the median nerve array and 4 from the ulnar array). a.) thumb flexion targets b.) index finger flexion targets, c.) middle finger flexion targets, d.) ring finger targets, e.) thumb-index pinch target. Note that the subject was able to perform this combination movement despite training only single single-DOF training movements. f.) wrist flexion and extension targets. 68 Figure 2.7 S4 tracked the position of three different moving virtual targets with the thumb, index, and middle fingers of a virtual prosthetic hand and then tracked the combined movement of all three targets with at least the middle finger and index finger. The top three traces depict the target location (solid line) and the subject-controlled, decoded virtual finger location (dashed line) for the thumb, index, and middle fingers. The subject independently tracked single thumb, index, and middle finger targets (depicted from left to right, respectively), and simultaneously tracked at least the middle and index finger targets in combination even though training data included no combination movements. The lower portion of the figure shows a raster indicating the times of recorded spike events from 8 selected median or ulnar nerve USEA electrodes during this task. The firing rates of spike events on these electrodes are uniquely tuned to different movements. 69 Figure 2.8 “Best case scenario” for multi-DOF offline decode. Bar chart depicts the correlation coefficients between predicted and attempted training movements of the most correlated movements in a multi-DOF offline decode of a single data set. The data set contained 8 trials of each DOF where 4 trials were used for training the decoder and 4 trials were used for testing. The solid bar represents the mean correlation coefficient of the highest correlated N movements, where N is the number of simultaneous DOF tested (whiskers are standard deviations). 70 Figure 2.9 USEA microstimulation provided a rich selection of percepts of various qualities and locations spanning the phantom hand (S4 shown here). A) Stimulation of individual electrodes via two USEAs restored 131 percepts across the phantom hand, including both proprioceptive and cutaneous percepts (collected over a 2-d period). Numerous cutaneous percepts were evoked on each digit and the palm, and proprioceptive percepts were restored for 17 different movements, including flexion and extension of each finger and flexion of the thumb. For proprioceptive percepts, upward arrows indicate extension, whereas downward arrows indicate flexion. B) 131 electrodes across the 10x10 USEAs evoked the percepts shown in part A, with no apparent somatotopic arrangement across the nerve cross-section. C) Evoked percepts were of various qualities, with 26% of evoked percepts described as proprioceptive, and 74% of evoked percepts being cutaneous (including ‘tingle’, ‘vibration’, and ‘pressure’). 71 Figure 2.10 Percepts evoked by median and ulnar nerve USEAs are generally within the established intact-hand innervation regions for each nerve. For the example shown (subject S3, week 2), 92% and 89% of median-nerve-USEA- and ulnar–nerve-USEA-evoked percepts are within their expected distributions, respectively. . 72 Figure 2.11 Number of sensory percept-evoking electrodes by week. A. The number of electrodes which evoked a sensory percept for each USEA across the implant duration (max amplitude, 120 µA, pulse duration of 200 µs). Note that perceptual thresholds for S4 were not tested on weeks 1, 3, and 4. Also, on week 5, S4’s ulnar n. USEA did not evoke any sensory percepts. B. Weekly median and interquartile range boundaries threshold amplitudes across electrodes on each USEA. Outliers not shown. 73 Figure 2.12 Confusion matrix for a 1-DOF closed-loop experiment performed by S3. * denotes that one “Far” trial resulted in a timeout which was scored as a misclassification. In this trial, the subject did not acquire the target and provide an answer in the allotted 30-s trial time. 74 Figure 2.13 Example of decode output (dashed line) and target distance (solid line) during two sequential trials of a 1-DOF closed-loop session for S3. Digits 1-4 were tied together and moved in unison. S3 was not allowed to look at the virtual hand display to determine how far his fingers moved. Stimulation was applied when the fingers entered the target zone (+/- 7.5% of the solid line value). To achieve a successful trial, S3 had to maintain the position of the fingers within the target zone for 0.3 s and answer correctly whether the target was “close” or “far.” In this segment of data, a “far” trial (distance of 0.8*flexion range of motion) occurred at 6 s, and a “close” trial (distance of 0.4*flexion range of motion) occurred at 16 s. In both trials, the subject hovers near the target distance and not the nontarget distance. In the first trial, the subject passes through the ‘close’ potential target distance without substantial pausing to reach the ‘far’ distance. These results indicate that the subject is able to use the USEA-evoked cutaneous percepts to determine when the virtual prosthetic fingers are in the target zone, and to proportionally control finger position. 75 Table 2.1 Weekly medians and interquartile ranges (IQR) of electrode impedances for all USEAs. Post-hoc testing between the first and final post-implant sessions revealed a significant pairwise drop in impedance for electrodes on S4’s median-nerve USEA (p<0.0001; two-tailed Wilcoxon’s signed-rank test, denoted with *), but did not reveal a statistically significant pairwise change for the remaining 3 USEAs (p = 0.82 S3 ulnar, p = 0.12 S3 median, p = 0.99 S4 ulnar). 76 Table 2.2 The best 12-DoF offline decode performance resulted from automatic electrode selection and constraining the training and testing movements to be in the flexion range only. Mean correlation coefficients between predicted position and actual position were computed across each DOF of a 12-DOF offline decode. Flex-range-only constraints involved limiting digit movement to be forward from baseline position only, whereas flexand-extend allowed movement both forward and backward from the baseline position. Manual electrode selection (24 electrodes selected) was performed by experimenters during online collection of the data, whereas automated electrode selection was performed offline using a thresholding algorithm. 77 2.10 Additional files Filename: “Video1- Freeform S4.mp4” Title of Video: Supplemental video showing a 12-DOF online decode “freeform” session for S4 Description: In this video, S4 controls 12 DOFs in a “freeform” session. S4 was instructed to move the hand in any way he desired. Movements available to the subject included flexion/extension of all 5 digits; abduction/adduction of the thumb, ring, and little fingers; and wrist flexion/extension and rotation. 78 Filename: “Video2-Targets Example.mp4” Title of Video: Supplemental video showing an example of the target-touching task Description: To verify simultaneous and individual control of multiple DOF, a targettouching task was designed. In this task, one or more spherical virtual target(s) was positioned away from the resting position of one or more digit(s)/wrist along the arc of movement. To demonstrate proportional control, targets were placed in one of two different positions: “Near” targets were positioned at 25% flexion and “far” targets were positioned at 75% flexion. A successful trial resulted when the subject moved the specified digit(s) inside the radius of the spherical target(s) for at least 250 ms while keeping the other DOFs in resting position. The virtual targets did not exclude the virtual fingers, and fingers could pass all the way through the spheres. To provide visual feedback, the target spheres change color from red to green when the desired finger enters the target sphere. The target radii were set to be 15% of the arc of motion in one direction. A trial was considered failed if the subject did not complete the task within a 30 s time-limit. After successful completion of a trial, virtual targets were automatically reset to the resting positions, and the subject was required to maintain all degrees-of-freedom in their resting positions for 1 s before the next trial was presented. This video shows an example of a target-touching task in which proportional position control was formally verified for a 4-DOF online decode in S4. Several examples of target-touching trials for 4 DOFs, including the thumb, index, middle, and ring fingers, at “near” and/or “far” distances are shown. 79 Filename: “Video3-Closed Loop Control S3.mp4” Title of Video: Video3-Closed Loop Control S3 Description: In this video, S3 used a cutaneous sensation on his ring fingertip (evoked by stimulation of a single ulnar-nerve USEA electrode) as feedback during an online, 1-DOF decode of 4-finger flexion/extension (decode via median-nerve USEA recording, driven by both neural and EMG). In the absence of visual feedback from the computer monitor, the subject successfully encountered and identified the location (‘close’ or ‘far’) of virtual targets in 41/47 trials (p<0.001, binomial test), using the USEA-restored sensation as feedback in addition to proprioceptive feedback from intact muscles of the forearm and/or efference copy to determine hand position. CHAPTER 3 DECODING MOTOR INTENT FROM HUMAN TRANSRADIAL AMPUTEES USING IMPLANTED NEURAL AND INTRAMUSCULAR ELECTRODE ARRAYS 3.1 Abstract We are developing a control strategy for decoding motor intent from upper-limb amputees for use in next-generation, sensorized, multiarticulated robotic prosthetic hands. In this study, two human, transradial amputees were implanted with 100-electrode Utah Slanted Electrode Arrays (USEAs) in each of the median and ulnar nerves above the elbow, and with a 32-channel intramuscular electromyogram (iEMG) array in residual forearm muscles for 3 months (HS1) to one year (ongoing, HS2). Using a combination of USEA and iEMG signals, both subjects were initially able to achieve 7-8 simultaneous and independent DOFs of motor control in realtime. After approximately 3 months, recording neural signals were largely unavailable. Subjects were eventually able to control 7-10 DOFs in realtime using iEMG alone. We thus found iEMG to be the more useful signal for long-term decodes of high-DOF motor intent in the hybrid neuromuscular system. Additionally, we show that decode parameters are stable for at least one week, thus reducing the need for retraining the decoder on a daily basis. Finally, we show that our decode techniques are robust in the presence of muscle compound action 81 potentials (MCAPs) inadvertently caused by neural stimulation in a bidirectional paradigm. Although the USEA lost the ability to detect motor units over 3 months in these subjects, it was still useful to provide sensory feedback through neural stimulation in chronic implants. Other functional considerations such as the effects of arm position and posture (such as upward reaching) on decoding accuracy are also discussed here. This study supports the use of iEMG to decode motor intent in a bidirectional neuromuscular interface. 3.2 Background In an ongoing effort as part of the DARPA HAPTIX program [1], we are developing a hybrid neuromuscular system capable of high degree-of-freedom (DOF) motor decoding and sensory feedback through neural stimulation, extending our previous work using neural signals alone (see Chapter 2 and [2], [3]). In this chapter, we focus on decoding motor intent using neural and/or intramuscular electrodes. Specifically we explore using new techniques to maximize the number of simultaneous DOFs a subject is able to control, the stability of decode calibration over many weeks, the effects of arm position on decode performance, decoding in the presence of extraneous muscle twitches inadvertently caused by stimulation, and finally the ability of the neural and muscle electrodes to record signal over many months. 3.2.1 Bidirectional interfaces To date, a number of approaches to interface with the nerve and muscle have been proposed and attempted, with promising results. Interfaces require two major 82 components: 1) the ability to record and decode electrical activity of residual muscles and/or nerves; 2) the ability to provide sensory feedback via electrical of the nerve or sensory substitution. Early bidirectional interfaces included single Longitudinal Intrafascicular Electrodes (LIFE electrodes) implanted into residual arm nerves to provide limited, single-DOF control and provide sparse localized sensations on the phantom hand [4]–[6]. More recent interfaces utilize a combination of implanted neural and sEMG electrodes to provide up to 4 movements (by a classification-based decode) and 2 regions of sensation in closed-loop operation [7], [8]. Because providing sensory feedback is important for embodiment of the prosthesis [9]–[11], several groups have focused primarily on restoring stable sensory feedback via neural stimulation or sensory substitution. In sensory substitution, feedback is provided by mechanical pressure or vibrations applied to the end of the stump. This noninvasive method is a viable approach for providing sensory feedback [10]. However, truly naturalistic sensations cannot be provided in this manner due to the inherent mismatch between the applied stimulus location/modality and environmental stimulus. Thus, restoration of naturalistic sensation through direct neural stimulation is being explored. Clark, Page et al. have demonstrated restoration of up to 131 highly localized cutaneous and proprioceptive sensations spanning the phantom hand of amputees using Utah Slanted Electrode Arrays (USEAs) implanted in residual peripheral arm nerves [3], [9], [12]. Additionally Tan, Tyler et al. have demonstrated that restoration of naturalistic pressure sensations on up to 20 regions on the phantom hand in 2 subjects have been achieved using cuff electrodes, which wrap around the nerve, with stability of up to 2 years [11]. 83 Several advanced robotic prostheses capable of supporting a closed-loop control interface are currently undergoing development. Examples include the Modular Prosthetic Limb (Johns Hopkins Applied Physics Laboratory, Baltimore, MD, USA) and “Luke” arm (DEKA, Manchester, NH, USA) [13], [14]. These prostheses have multiarticulated joints, capable of complex movements and dexterous, high-DOF control. Additionally, pressure and torque sensors positioned on the fingers, joints, and palm of the prosthesis can provide information about the environment and position of the robotic digits and wrist. This sensory information can then be translated into neural stimulation parameters (frequency, amplitude, etc.) [9], [11]. In contrast to existing commercial systems, which typically are only capable of discrete, one-DOF control, and lack sensory feedback, this next generation of prosthetic limbs may have more intuitive and dexterous control, opening up the ability of users to do more activities of daily living (ADL). Additionally, such prostheses may be easier to embody, which has the potential of reducing phantom pain syndrome [9]. These advancements may address many of the reasons amputees abandon EMG prosthetics in favor of body-powered prosthetics, or no prosthetic at all [4], [15]–[18]. Here, we implement a bidirectional interface that uses high-channel-count microelectrode arrays implanted into residual arm nerves in combination with implanted intramuscular electromyographic (iEMG) electrodes in the residual forearm muscles, to record and decode volitional electrophysiological signals, and, to provide sensory feedback through direct electrical stimulation of the nerves (see Figure 3.1). 84 3.2.2 Decoding approaches A number of groups have demonstrated the feasibility of using peripheral nerve signals, iEMG, and/or sEMG for decoding intended movement in both animals and humans [5], [8], [19]–[28]. The most common control strategy for commercially available myoelectric prostheses is the “direct control” method. In this method, the power of EMG signals from specific electrodes are mapped “directly” to a single direction of one DOF. For example, EMG power from surface electrodes placed over forearm flexor muscles can be mapped to the “hand close” command, and electrodes placed over the forearm extensor muscles can be mapped to the “hand open” command. Using such methods, typically 1 or 2 DOFs such as hand open/close, and/or wrist pronation supination are employed in commercially available devices. Advanced prostheses such as the “i-limb” hand allow the user to select from a large number of hand grasp (16 in total) such as “tool grip” or “pencil grip.” However, in these scenarios, the grasp open/close commands are still controlled by only a single DOF (e.g., flexor surface electrodes control the fingers closing together in the specific grip pattern and extensor surface electrodes control the fingers opening together in the specific grip pattern). A limitation of this approach is the small number of DOFs that can be simultaneously controlled. In direct control, crosstalk between the DOFs is also problematic. Because surface electrodes pick up signals from adjacent muscles beneath the electrodes, it is difficult to isolate signal sources. The separability of DOFs in the direct control depends on isolation of the signal sources. In a laboratory setting, Weir et al. have made efforts to isolate signals by using transcutaneous, fine-wire electrodes inserted directly into specific 85 muscle bellies. By isolating signals directly from specific muscles, subjects were able to achieve 4 DOF control using the “direct control” strategy [25]. In research settings, pattern recognition strategies are used to decode grasps, as opposed to individual DOFs [29]. A discrete classification decoder is trained using categorical grips or movements. Classification techniques such as support vector machines, artificial neural networks, linear discriminant analysis, hidden Markov models, and Gaussian mixture models, and fuzzy logic classifiers are among those used [30]–[34]. Zhou, Kuiken et al. have demonstrated up to 16 grip classifications using high-count sEMG recorded from a transhumeral amputee after targeted muscle reinnervation (TMR) procedure [19]. Although classification approaches have proven to be highly accurate (approaching 100% accuracy) [27], they offer a limited number of grips the user can control, and are sequential in nature (only one movement can be performed at a time). Additionally, users must train on all categorical grips desired for the classification “vocabulary,” which may be burdensome. Continuous regression algorithms that predict individual DOFs are also used [24]. In contrast to discrete classification methods, regression strategies provide realtime, intuitive proportional control of individual degrees of freedom, allowing for an unlimited number of grip possibilities and arm positions. Smith, Hargrove et al. demonstrated 3DOF proportional control using fine wire EMG implanted in the forearm of intact subjects [24]. 86 3.2.3 Utah neuromuscular decode approach Our approach involves using a modified Kalman filter (described in Chapter 3) to estimate the position of individual virtual fingers from the firing rate and/or EMG power from the residual arm. This method is a type of continuous decoder that has previously been used to successfully decode intended movements from firing rates of a population of neurons in cortex [35]–[39]. Our algorithm, in contrast to discrete classification approaches, provides simultaneous and proportional control of individual DOFs, allowing users to intuitively combine individual movements to create an unlimited number of hand positions for grips and gestures, without training the decoder in all positions. 3.2.4 Previous studies In four previous subjects, we implanted only USEAs in the residual forearm nerves [2] and upper arm nerves [3] (also see Chapter 2) of transradial amputees for up to 5 weeks in attempt to restore bidirectional sensorimotor control. In these studies, distally implanted subjects were able to control 2 DOF in realtime and perceive up to 81 cutaneous perceptions on the phantom hand [2]. Proximally implanted subjects controlled up to 5 DOF in realtime and perceived up to 131 cutaneous and proprioceptive sensations on the phantom hand (see Chapter 2). In the present experiments, we expand on this work and adapt our methods to incorporate a combination of neural (using USEAs) and/or iEMG signals for implants lasting 3 months (Subject HS1) or one year (Subject HS2, ongoing). The combination of USEA and iEMG in tandem has not been previously studied, and offers an opportunity to compare directly the relative merits of each approach 87 within the same individual. 3.2.5 Functional considerations In an ideal decoder system, the decoder output would be highly accurate under all circumstances of standard use. The user would not need to retrain the decoder more than once per day, and the outputs would be robust to arm position and weight bearing. Here, we address such functional considerations by testing our methods under several pertinent conditions: 1) We test the stability of the decode calibration over days to weeks in online and post-hoc (“offline”) settings. This relates to the frequency the user will need to recalibrate the decoder. 2) We investigate a number of arm positions (e.g., upward reach, downward reach, across body reach) and attempt to quantify the effects on the decode performance. Posture changes, reaching, and weight bearing necessitate different patterns of muscle flexions. We investigate the effects of such functional stressors on the decode accuracy. 3) We investigate the effects of neural stimulation on decode performance. Because sensory feedback through neural stimulation is a highly desired and important component of advanced prosthetic limb function, the ability to electrically stimulate while accurately decoding motor intent is crucial. However, some stimulation may evoke muscle compound action potentials (MCAP) that interferes with signals used for EMG decodes. Here we present a strategy for decoding in the presence of extraneous muscle activity caused by stimulation. 88 3.3 Methods 3.3.1 Study population In this IRB approved study (University of Utah, Institutional Review Board), two male, human, transradial amputees were recruited to participate in the study. Subject HS1 was a right-dominant 45-year-old bilateral, transradial amputee with amputations approximately 15-16 cm distal to the elbow (measured from the medial epicondyle to the most distal portion of the stump) on each side. HS1 sustained an electrical injury 26 years prior to the study start. Subject HS2 was a 58 year-old, left-dominant, left-sided transradial amputee, with amputation approximately 18 cm distal to the elbow. HS2 sustained an electrical injury 14 years prior to the study start. Both subjects were evaluated by an orthopedic surgeon, physical medicine and rehabilitation physician, and psychologist prior to enrolling in the study. Each subject offered voluntary consent. 3.3.2 Neural implants 100-electrode Utah Slanted Electrode Arrays [40] (USEAs, Blackrock Microsystems, Salt Lake City, UT, USA) were used for neural recording and stimulation. 96 of 100 electrodes were used for recording and stimulation, and 4 electrodes were used as an on-array reference. Each array measured 4x4 mm, with electrodes arranged in a 10x10 grid, spaced at 400 um, and containing electrodes of 0.7-1.5 mm lengths. The electrodes were wire-bonded to an 10 cm silicon-potted platinum wire bundle which was soldered to a custom 100-channel Gator connector (Ripple LLC, Salt Lake City, UT, USA). 89 3.3.3 Muscle implants The intramuscular implant (Ripple LLC) consisted of a 34-electrode, 9-lead array of flexible, silicon potted coiled wires, similar in diameter to a cardiac pacemaker. Exact dimension specifications were not available from the manufacturer. Hence iEMG device implant descriptions are approximated here. An illustration of a similar iEMG prototype can be seen in Figure 3. 3. Leads 1-8 were approximately 20 cm long; each contained 4 cylindrical electrodes 2 mm wide, spaced at approximately 2 mm at the distal end used for recording EMG signals. Lead 9, approximately 5 cm long, contained two electrodes (same size and spacing of leads 1-8) and served as the reference and ground. Additionally, a small flexible plastic barb was attached to the end of each lead for anchoring the leads into the tissue. All leads were grafted together at the proximal end and soldered to a custom 32-channel Gator connector (Ripple LLC). 3.3.4 Surgical procedure Subjects were given a prophylactic antibiotic (100 mg minocycline, 7 days b.i.d., starting the day before the implant surgery) which potentially improves the quality of chronic neuronal recordings [41]. Under general anesthesia, two USEAs were surgically implanted into each subject—one in the residual median nerve and one in the residual ulnar nerve. USEAs were placed in the upper arm, approximately 2 centimeters proximal to the medial epicondyle. This location was chosen due to access to motor nerve fibers that innervate flexor and extensor muscles of the hand and forearm, and sensory fibers that innervate receptive fields on the entire palmar aspect of the hand. 90 In both subjects, the epineurium was dissected from the surface of the nerves prior to pneumatic insertion of the USEAs [42]. The USEA wire bundle, ground, and reference wires were sutured to the epineurium, and a protective collagen wrap (AxoGen Inc., Alachua, FL, USA) was placed around the nerve, USEAs, and reference/ground wires. The wrap was secured with vascular clips and sutured to the epineurium for stability. 0.1 mg/kg of dexamethasone was administered intravenously immediately following the implantation to potentially mitigate the foreign body response and improve neural recording capability [43], [44]. Intramuscular EMG leads were implanted through incisions on the flexor and extensor sides, using a custom insertion tool. For HS1, 4 leads were inserted flexors, 3 in extensors, and 1 through the interosseous membrane from the flexor side. Three leads were placed in extensors and 5 in flexors for HS2. The common bundle was routed under the skin over the lateral epicondyle and exited the skin near the USEA percutaneous sites. The reference and ground lead was passively placed near triceps tendons in the posterior aspect of the upper arm, near the wire bundle exit site. Figure 3.2 shows a photograph of USEA implants from a previous subject, and an X-Ray of the iEMG leads in HS1. Percutaneous wire-passage sites were re-dressed as needed throughout the study, on at least a weekly basis. Antibiotic wound dressings (Biopatch, Ethicon US LLC, Somerville, NJ, USA) were placed directly over the percutaneous site circumscribing the wire bundles throughout the study duration to reduce the risk of infection. After 12 weeks, the USEAs were surgically explanted from HS1 after developing an infection around the USEA wire bundle (determined to be common skin flora). Additionally, the iEMG array was explanted at 16 weeks due to spreading infection. 91 Following explant and antibiotics (Bactrim), HS1recovered completely with no long-term effects. HS2 has an ongoing implant (currently week 49) with no sign of infection. 3.3.5 Experimental methods and metrics The process of our decoding method from data collection to decode output to the virtual or robotic prosthetic hand is depicted in a block diagram in Figure 3.4. 1) USEA and iEMG data are collected and conditioned. 2) Features are calculated and buffered. 3) selected feature channels and decode algorithm coefficients determined from the calibration phase are used to compute decode outputs. 4) Decode outputs are converted into commands for the virtual or robotic prosthetic hand. 3.3.6 Data collection Signals from 96 channels in both USEAs were amplified and digitized using Micro2+stim Front End Modules (Ripple, LLC) at 30 kHz. 32 channels of iEMG were amplified and digitized using an “Active Gator” (32-channel Nano Front End Module, Ripple LLC) at 1 kHz. Data from all implants were collected simultaneously using the Grapevine bioamplification system (Ripple LLC). Neural data were filtered with a 750 Hz high pass filter (4th order Butterworth filter). iEMG data were filtered with a 15-375 Hz bandpass and 60/120/180 Hz notch software filters. Data were streamed and further processed in realtime using custom-built software written in Matlab and Labview programming languages. 92 3.3.7 Feature extraction Firing rates from USEA electrodes were used as the neural features for the decode algorithm. Threshold crossings from each channel were automatically detected using an adaptive threshold of -6* Root Mean Square (RMS) of the previous 15 s of data. Times of threshold crossings were recorded converted to the firing rate feature for all neural channels. The mean firing rate was computed over a sliding 300 ms window, updated every 33 ms. Amplitude of single-ended and software-differenced channel pairs were used as the iEMG features for the decode algorithm. Amplitude was computed by rectifying the raw voltage of single ended (up to 32 total), and software differenced pairs (up to 496 total, from 32-choose-2 pairs). The data were then smoothed with a boxcar filter (300 ms window length). All features were updated every 33 ms. Baseline data while the subject was not moving were collected for 15 s prior to decode calibration. The mean of the baseline data (mean firing rate for neural data and mean amplitude for iEMG data) was continuously subtracted from the feature data. 3.3.8 Decode calibration (training phase) To train the decode algorithm, subjects were instructed to imitate with their phantom or intact hand a series of single-DOF virtual-hand movements shown on a computer screen while EMG data are collected and saved. Subjects were instructed to keep their arm in approximately the same position and orientation for training and testing sessions (e.g., same amount of elbow flexion and wrist pronation). Training sets included 93 5 to 10 trials of each movement, with each movement trial lasting for 1 to 2 s. The complete training session generally lasted 5-10 min. Subjects trained on 6, 8 or 10 DOFs and one combination full-grasp movement (D1-5 flexion/extension in unison). In 6 DOF training sets, DOFs 1-5 and 9 were included to match the DOFs available on the DEKA Luke hand [14], [45], [46]. In 8 DOF training sets, DOFs 1-6, 10, and 12 were included. In 10 DOF training sets, DOFs 1-6, 7/9 (combined), and 10-12 were included. Occasionally extensions of DOFs 4 and 5 (ring and pinky finger) were excluded due to the difficulty of performing these motions independently during training and the limited utility in virtual activity of daily living (ADL) tasks. 3.3.9 Data alignment Following the training period, feature data were aligned with movement cues in order to improve feature selection and the generation of decode algorithm coefficients. Here, feature data are shifted globally by the lag which maximizes the correlation between all features and movement cues. 3.3.10 Feature selection Features were then down-selected from the 720 possible features (192 neural features, and 528 iEMG features, from 32 single ended channels + 32-choose-2 differential pairs) using the Grahm-Schmidt orthoginalization method [47], with a limit of 48 features. The time-aligned and down-selected features were used to generate coefficients for the decode algorithm (calibration) 94 3.3.11 Realtime decoding For realtime decoding, down-selected features were computed and passed through a position-estimating modified Kalman Filter (KF) decoder (described below), which was updated at 30 Hz (33 ms). To further minimize crosstalk between DOFs, user-selected gains (G > 0) and thresholds (which must have the range -1< Thresh < 1) were applied to the outputs as described below. 3.3.11.1 Modified Kalman filter decoder Kalman filter (KF) decoding is commonly used approach for decoding motor intent from neural signals in the central nervous system. It is briefly described below. Further detail can be found in [35], [36], [38], [39], [48], [49]. The KF decoder was chosen for its stability and accuracy given noisy inputs such as neural firing rates. The KF is a type of predictive estimator that relies on current measurements of a feature, in addition to past estimates of the output state to make a prediction about the current state. We have modified the standard KF to account for the user selected output gains and thresholds mentioned above. Additionally, the outputs are bounded to prevent outputs that run out of a fixed -1 to 1 range. 3.3.11.2 Calculating algorithm coefficients from training data The KF uses current measurements to predict future states of a system. We start by assuming that the next state in our system, x, at sample k+1 (where k is the current sample) can be defined as a linear progression from the current state: 𝒙𝒌#𝟏 = 𝑨𝒙𝒌 + 𝒘𝒌 , (1) 95 Here, x is our kinematic variable, representing the position of an individual DOF. xk+1 is the sequential decoded output following xk, A is the state-to-state transition matrix, and wk is the prediction error vector (from observation noise) with covariance matrix W. Similarly, the current feature zk can be expressed as: 𝒛𝒌 = 𝑯𝒙𝒌 + 𝒒𝒌 (2) where H is a transform matrix and q is the modeling error (model noise) with covariance matrix of feature noise Q. A, Q, W, and H are determined from training data. Time-aligned, down-selected features (Z) and kinematics from the movement cue (X) are used to compute A and W and then solve for Q and H, which are then held constant for decoding in realtime. An example pseudocode implementation (Matlab-style, with starting index 1) of the training coefficient calculations is as follows. Let X be a matrix of training kinematics (movement cues) n-DOF by m-samples. Let Z be a matrix f-features by m-samples. We start by calculating the state-to-state transition matrix A, which transforms the previous states, X(:, 1:(m-1)) to the next state, X(: , 2:m) : A1 = X(: , 2:m) * X(:, 1:(m-1))T (3) A2 = X(:,1:(m-1)) * X(:,1:(m-1))T (4) A = A1 * A2-1 (5) Next, the covariance of the noise in the kinematics, W, is computed by: W1 = X(:,2:m) * X(:,2:m)T (6) W2 = X(:,1:(m-1)) * X(:,2:m)T (7) W = (1 / (m-1))*(W1 – A * W2) (8) 96 In order to calculate the prediction coefficient matrix H, we first need to calculate the cross-correlation, Pzx, and autocorrelations, Rxx, Rzz, of the kinematics matrix X and feature matrix Z. Pzx = Z(:,1:m) * X(:,1:m)T (9) Rxx = X(:,1:m) * X(:,1:m)T (10) Rzz = Z(:,1:m) * Z(:,1:m)T (11) The transformation matrix, H, is computed: H = Pzx * Rxx-1 (12) Next, the covariance of the observed features, Q, is computed: Q = (1/m) * Rzz - H * PzxT (13) Note that the pseudo-inverse function was used for the Matlab implementation of the inverse function in equations 5 and 12. 3.3.11.3 Realtime prediction and update After the A,W, Q, and H are calculated, we begin the two-step process of prediction and update for the realtime output of the decoder 𝐱. Here, a vector of features is given as an input to the decoder at each time point. The prediction step is as follows. The a-priori state estimate 𝐱 𝒌- is computed from the previous output, 𝐱 𝒌-𝟏 , and the a-priori error covariance matrix 𝐏𝒌- is computed from the previous iteration or initial value of Pk-1. 𝐱 𝒌- = 𝐀 ∗ 𝐱 𝒌-𝟏 𝐏𝒌- = 𝐀 ∗ 𝐏𝒌-𝟏 ∗ 𝑨𝑻 + 𝑾 (14) (15) The update step is as follows: Kalman Gain K for the current step k is computed: 97 Kk = Pk * H * (H*P *HT + Q)-1 (16) The output state, 𝐱 𝒌 , and error covariance matrix, Pk are then updated: 𝐱 𝒌 = 𝐱 𝒌- + 𝐊 ∗ ( 𝒛𝒌 − 𝐇 ∗ 𝐱 𝒌- ) Pk = (1 – Kk*H)*Pk- (17) (18) Because A, Q, W, and H are held constant, only parameters P, K, and x are predicted and updated during every step of the realtime decoder. 3.3.11.4 Modifications to the realtime decoder output To prevent the output 𝐱 𝒌 from running out of the -1 to 1 allowable range, 𝐱 𝒌 is bounded to this range at the last step in the “update” step. Additionally, to account for the user-selected output gain G, 𝐱 𝒌 is also divided by the user-selected output gain before the next iteration. Thus 𝐱 𝒌-𝟏 in the next iteration “prediction” step is the scaled and bounded output of the current iteration. An example pseudocode implementation is as follows. if |𝐱 𝒌 | > |1/G| 𝐱 𝒌 = 1/G (19) end After each update step, the user-selected gains and thresholds are applied to the output (see below, equation 20) before being sent as a command to the virtual or robotic prosthetic hand. Gains are typically chosen G < 3. Individual gains for each DOF can be set by the user. This allows the user to make certain movements “stronger” by amplifying the output for that movement with the same amount of effort. Thresholds typically are chosen to be in the range 0.2< |Thresh| < 0.5. In this step, the output is set to zero if the value of the output is between the user selected positive and negative threshold values. 98 The threshold step greatly reduces cross talk. The combination of user-selected gains and thresholds results in a highly configurable decode with minimal crosstalk between DOFs. An example of this step in pseudocode is as follows: Let xcommand be the system output to the hand, G is the user set gain, Th is the user set threshold, and 𝐱 𝒌 be the output from the decoder. Here, xcommand, G, and Th are all positive values. xcommand = (𝐱 𝒌 *G – Th) / (1 - Th) (20) The xcommand value is not fed back into the decoder as 𝐱 𝒌-𝟏 as this would cause unstable output due to the mismatch of actual commanded state and predicted state. 3.3.11.5 Position and velocity modes Several control strategies were available to the user including position control, velocity control (“latching”), or a hybrid mode (“leaky integration”). In position mode, no integration was applied to the output. In velocity mode, outputs were continuously integrated, causing the hand to hold the position corresponding to the sum of previous efforts. This allowed subjects to maintain grip positions without sustained effort. In the hybrid position-velocity mode, a leaky integration window is applied to a moving window of data (the window length was experimenter-specified). Here, the hand relaxed slowly back to neutral position when no effort was attempted. Outputs of the decode in a user selected integration mode were then used to control the virtual hand in realtime. During target-touching tasks (described below), typically “leaky” mode was used. In the hand-matching tasks, velocity mode was used. 99 3.3.12 Online decode performance assessment To verify simultaneous and individual control of multiple DOFs online, two test strategies were employed: 1) a target-touching task, and, 2) a hand-position-matching task. In the target-touching task, one or more spherical virtual target(s) is positioned away from the resting position of one or more digit(s)/wrist along the arc of movement. A successful trial results when the subject moves the specified digit(s) inside the radius the spherical target(s) (radius = 15% the total allowed arc length in one direction for each DOF) for at least 300 ms while keeping the other DOFs in resting position. Three to six trials of each movement were performed in these experiments. A movement was considered “successful” if the subject successfully completed two-thirds of the total trials for a given movement. Timing and path length metrics were computed including: the number of successful trials, trial time (the time from the start of a trial until successful completion), path length (sum of the rectified output for all DOFs during trials), time to the first target touch, and number of target pass-throughs. Timing, pass-through, and path length metrics were computed for successful trials only. These metrics were used for statistical comparison between data sets using the Kruskal-Wallis rank-sum method, ANOVA, or paired t-tests where appropriate. Medians and interquartile ranges (IQR) for trial sets were computed and displayed in tables for global comparisons. Metrics tabulated in tables include total DOFs achieved/DOFs trained on, total movements achieved/movements trained on, total successful trials/trials performed, percentage success, median trial time of successful trials (IQR), median time to first target touch (IQR), median number of target pass-throughs (IQR), and median path length (IQR). 100 Note that by random movement alone, it would be highly improbable to achieve a trial success within the 30-s time limit. However, with the assumption of perfect control of the hand and instantaneous movement, then without visual feedback, it would take 1 s per DOF per direction to successfully accomplish a target-touching trial by “gaming the system” via using sequential incremental movement steps, whose magnitudes corresponded to a sphere’s diameter and were separated by 300 ms stops, to reach the average target position from the start position (0.5 * 300 ms window * number of possible sphere locations (1/0.15)). For example, the time to accomplish a 20-movement targettouching task would be 20 s. In the hand-position-matching task, the subject performed 26 trials of a 3-DOF hand position matching task in the Mujoco VRE [50]. Here, the subject was given 20 s to match novel hand positions that were not previously explicitly trained, involving combinations of wrist flexion/extension, thumb flexion/extension, and finger (DOFs 2-5, locked together) flexion/extension. The decoded hand position must match the test position to within a 5% tolerance (total, for all DOFs) and hold there for 1 s to be considered a success. Color-coded visual feedback regarding the degree of matching was provided to the subjects. Target hand regions change from red to yellow to green based on the closeness of the decoded hand to the target position (i.e., the target hand turns red when the decoded output > 15%, yellow when the output 5% to 15%, and turns green when the output < =5% from the target positions) (See Figure 3.5). Similar to the targettouching task, performance metrics such as number of successful trials, trial timing, path length, and trial velocity (total distance needed to travel in all DOFs in the trial divided by the trial time) were computed when the data were available. The hand matching task 101 was available only for subject HS2. Early versions of this test did not save raw data, so these data are unavailable for timing and path length metrics. Target-touching tasks and hand matching tasks were performed periodically (at least once/week) to assess decode performance, stability, and susceptibility to arm position variations and neural stimulation. 3.3.13 Post-hoc decode performance assessment Offline analysis of decoded output was also where predicted movements were compared with movement cues during the training phase. Here, 4 trials in the first half of the training set were used for training the decode algorithm, and 4 trials from the last half of the training set were used for testing the decode performance. Performance metrics were computed for each DOF to assess the offline decode including root mean square error (RMSE) and correlation coefficient between the decoded output. 3.3.14 Signal quality and device stability assessment Omnibus data sessions where USEA and EMG was recorded while the subject was performing a series of movements meant to elicit any possible motor unit activity from USEAs and iEMGs were performed. These sessions included 10 s of moving each digit individually and sequentially, 10 s of opening and closing the fist, 10 s of moving the wrist around in all 3 axes, and 10 s of rest. Figure 3. 6 depicts a segment of data during an omnibus session for HS2. USEA data were analyzed for the presence of neuronal spiking correlated with intended movement. Neuronal spiking was identified using an automated threshold crossing detector and verified by visual inspection. Power and RMS 102 from iEMG during movement and baseline were computed and tracked over time. Impedance measurements were used to identify failed USEA electrodes/ channels as well as to monitor the over-time stability of working electrodes. Impedances were measured using the Grapevine system (Ripple LLC) and Micro2 +Stim or “Active Gator” front-ends. Here, we defined failed electrode as one which had an impedance greater than or equal to 500 kΩ. Nonfailed electrodes are defined as never having an impedance value above 500 kΩ for the duration of the implant. 3.4 Results 3.4.1 Subjects can perform high-DOF decodes using iEMG and USEA implants Both subjects were able to achieve > 8 DOF decode as verified by the targettouching task. HS1 achieved 8 DOFs (iEMG decode, 29/30 successful targets, 10 movements included D1-5 flex, D1 extend, ab/adduction, wrist flex/extend, see video “P201501 7-DoF targets_v20151015-1808_largeFormat.mp4”, for an example). HS2 achieved 10 DOFs (iEMG decode, 58/60 successful targets, 19 movements included D1-5 flex/extend, D1ab/adduction, wristflex/extend, pronate/supinate, ulnar/radial deviation, index-pinky ab/adduction). HS2 was also able to perform intuitive movements of the DEKA Luke prosthetic hand using iEMG decode in a bidirectional setting during ADLs (see video “HS2_Luke_Hand_Highlight Reel_v20170327.mp4”, data quantified elsewhere). 103 3.4.2 iEMG is the primary contributor to decoder information To investigate the contribution of each signal modality (USEA and iEMG) to decode performance, online and offline analysis of sequential target-touching tasks where both USEA and iEMG data, USEA alone (if possible), or iEMG alone data. Table 3.1 summarizes the results of sequential online target-touching tasks for both subjects. On post-implant-day 38, HS1 was able to perform 7 DOFs (7 independent movements) using either combined USEA and iEMG features or iEMG features alone. He was able to perform 5 DOFs and 5 movements using USEA signals alone. To perform these online decodes, 6 USEA features (4 from electrodes from the median array and 2 from electrodes the ulnar array) and/or 80 iEMG features were selected as inputs to the decoder. HS2 was able to perform 8 DOF and 13 movements using either USEA and iEMG features, or iEMG features alone on post-implant-day 16. For these trials, 5 USEA electrodes (all on the median array) and 79 iEMG features were used for the decoder. Informally, he was not able to perform more than 1-DOF using USEA signals alone, and this condition was not further tested. Rank sum analysis on target metrics comparing USEA+iEMG trials with iEMG alone trials revealed no significant difference for HS1 (all p>0.2) or HS2 (all p > 0.08). Offline testing comparing USEA and iEMG contributions to the decoder were performed on 5 sessions of training data. Table 3.2 and Figure 3.1 summarize the results of this analysis. RMS and correlation coefficients were computed between decode output and movement cue for several DOFs using either USEA or iEMG signals for the decoder. iEMG outperformed USEA significantly (t-test p < 0.001) in all metrics for all DOFs tested. Given these preliminary results, further offline testing comparing 104 iEMG with USEA signals was not performed. 3.4.3 iEMG decode calibration can be stable for up to 14 days To investigate the stability of decode algorithm calibrations online, targettouching tasks and/or hand matching tasks were performed periodically using the same decode coefficients on subsequent days. Decode coefficients from three training sets were tested online using target-touching tasks for HS1 for up to 48 d (Figure 3.8). Two of three sets of decode calibration coefficients were stable for 9-14 d (HS1 was able to achieve at least 85% of the original number of movements). The hand-matching task was used to test HS2 3-DOF, 6-movement decode calibration for up to 43 d (Figure 3.9). This decode set was stable for all metrics for 8 d, when he achieved 26/26 successful trials at his fastest time per trial. The performance then showed decline until day 43, where he achieved 17/26 successful trials but verbally expressed frustration about the amount of control he had over the virtual hand. 3.4.4 Offline decode stability testing To test the stability of decode coefficients over time in an offline setting, decode coefficients from each training set were used to decode feature data from each subsequent training set. Overall, the correlation coefficient decreases rapidly over the first 20-50 d for both subjects, indicating loss of decode accuracy (see left panels of Figure 3.12 and Figure 3.14). However, these data do not differentiate among decodes based on training sessions performed early or later after implantation and related training (see Figure 3.11 and Figure 3.13). For HS2, three trends can be observed in RMSE (see right panel of Figure 3.14): 1) RMSEs in general increase over the first 105 20-50 d after a training session, indicating some loss in decode accuracy. 2) The RMSE errors are higher for decodes based on early training sessions (i.e., longer data lines), whereas decodes based on later training sessions show lower RMSEs, matched for the same duration interval post-training. 3) RMSE errors are more variable across sessions for decodes based on early training sessions than for decodes based on later training sessions. These observations suggest that decode stability improves over the course of the implant and/or subsequent training, and that the subject’s performance may become more consistent with practice or with stability of the recorded signal. 3.4.5 Arm position effects on decode performance Arm position during various ADLs such as reaching upwards, downwards, and across the body could potentially degrade decode performance due to co-contraction and electrode shift in the muscle due to stretching or torsion. To investigate the effects, several sets of training data were collected for offline analysis while HS2 held his arm in the reach positions. Figure 3.15 shows the results of decoding training sets at nonstandard reach position, using the standard position decode coefficients and baseline. Offline, the most affected DOFs were DOFs 5, 6, 10, and 12 where there were large amounts of crosstalk during these movements. Figure 3.16 shows another example in which the decoder was trained in one position (“down”) and the coefficients used to decode training data from another position (“up”). There was generally more crosstalk when the training data were collected in a different position (particularly DOFs 1-5). Additionally, the subject was asked to subjectively rate his sense of control for each direction (flex or extend) of each DOF in the standard, upward reach and downward 106 reach arm positions. HS2 was able to volitionally move all DOFs trained on in all positions, but verbalized the most difficulty while the arm was in the “upward reach” and “downward reach” positions. Figure 3.17 summarizes HS2’s ratings of all movements at all positions. Surprisingly, there was no global trend in the subjective ratings, and some movements in certain positions were rated higher than those in the standard position. 3.4.6 Attempts to “rescue” the decode by subtracting out positional baseline were unsuccessful An attempt was made to “rescue” the decode offline by subtracting out a baseline that was collected in the same position from the training data (instead of the standard arm position baseline data, as would be done operationally). The results of this attempt were not successful (Figure 3.18). 3.4.7 Decoding can be robust during muscle activity inadvertently induced by neural stimulation To verify that a robust decode can be performed in the presence of muscle compound action potential (MCAP)-inducing stimulation, we compared decode performance in the hand-matching task with and without stimulation enabled, while maintaining constant decode parameters (filter coefficients, channels used, gains, and thresholds). In this experiment, neural stimulation of 1 electrode at (stimulation parameters 30 µA, 200 μs pulse width, 30 Hz, 500 ms trains) caused a large amplitude, nonadapting MCAP that initially appeared in the decode output. Figure 3.19 shows an example of a stimulation train evoking a large amplitude MCAP (for the data presented 107 in the figure, collected on a different day, the response habituated). For the task, MCAP was detected by a channel whose features were used for the decode. The experimenters then adjusted the gains and thresholds applied to the decode so that the muscle twitch no longer affected the decode output. Figure 3.20 shows that the mitigation effort was successful during a 6-trial session where stimulation was present for 3 of the trials. Here, there was no significant difference between trial timing metrics including mean trial time and trial velocity in the stimulation on/off trials. 3.4.8 iEMG signal quality does not degrade over 3 to 12 months Omnibus trials were collected frequently throughout the course of the study for both HS1 and HS2. iEMG Amplitude during volitional movements and SNR during omnibus trials are shown for HS1 in Figure 3.26 and HS2 in Figure 3.27. Here we define SNR as the variance of each channel during movement divided by the variance of each channel during rest. Amplitudes during movement periods steadily increased over time. There was no degradation trend of SNR over time for either HS1 or HS2. 3.4.9 The ability of USEA to detect motor units in the peripheral nerve declines over 3 months HS1 and HS2 driven motor units detected by USEAs acutely increased over the first 2 weeks, then decreased until they were no longer detectable in approximately 3 months. Figure 3.25 illustrates the number of driven units for each array for each subject. Units were visible in HS1 neural data until post-implant day 83 and in HS2 neural data until post-implant day 96. 108 3.4.10 Number of working electrodes decreases substantially over 3 months The number of “working” electrodes (impedance ≤ 500 kOhms) declined over the course of the study for both HS1 and HS2 by approximately 15-45% (see Figure 3.21, Figure 3. 22, and Figure 3.23). However, impedances for working electrodes remained fairly stable after an initial post-implant drop (see Figure 3.21, Figure 3. 22, and Figure 3.24). Statistical analysis has not been performed on this data set. 3.5 Discussion We have shown that it is possible to decode up to 8 independent DOFs and 13 movements from a combination of neural data from USEAS implanted in residual median and ulnar arm nerves and EMG data from iEMGs implanted into residual forearm muscles. However, we also show online and offline that iEMG is the primary contributor to the decode. Using iEMG signals alone, it was possible to decode up to 10 DOFs and 19 movements in HS2. This is more simultaneous and independent DOFs than have been previously reported using iEMG or hybrid neuromuscular systems. It should be noted that the 10-DOF decode was performed many months after neural units were no longer detectable in USEA data. Thus a standardized comparison between USEA+iEMG and iEMG only could not be performed for the 10 DOF scenario. In both subjects, the thumb adduction movement, controlled by intrinsic hand muscles, was able to be reliably decoded using iEMG signals, even though no recordings were being made directly from the intrinsic hand muscles. Two-DOF capability on the thumb is particularly useful for a number of grips that users would want to create during ADLs such as power grip and key grip. Additionally, HS2 was also able to achieve index- 109 pinky adduction, which is useful movement for dexterous and intuitive manipulation of tools such as writing or eating utensils. This illustrates the capability of our algorithm to decode movements from complex patterns of movements that are still present many years after amputation. One important advantage of simultaneous and independent multi-DOF control, compared with classification approaches, is that the user may combine any DOF in any proportion to make many possible novel grip patterns that the decoder has not been explicitly trained on. This capability is advantageous for many ADLs requiring dexterous manipulation of small objects and hand tools such as eating utensils, writing implements, keyboards, and hand-held electronic devices. The utility of this approach was not directly compared with that of the classifier approach in this study. It is likely that certain ADLs will be easier to perform with independent control, whereas others will be easier to perform with the classifier approach. Examining a hybrid decode strategy is an opportunity for future studies. Precise, dexterous manipulation of a highly articulated robotic hand would require a high level of competence in controlling all DOFS. It was anecdotally observed that, the more DOFs attempted in training, the more difficult was the precise control over all DOFs during tasks. This effect was not formally quantified in this study. However, the stability of the decode coefficients may allow for repetitive practice, allowing the user to learn precise control over all DOFs in time. In both subjects, there was a substantial decline in detectable neural units and in the number of “working electrodes” for USEAs over the first 3 months post-implant. This decline is a major concern if neural data are desired for decoding motor intent in 110 neuromuscular interfaces. Many possibilities can account for the decline in observed motor units and working electrodes, including 1) wire breakage resulting from incidental bending of the wire bundle during ADLs and experimental testing; 2) foreign body response, causing deinsulation of USEA electrodes, neural remodeling in response to axonal injury [51], [52], macrophage cell and fluid build-up between electrodes and neurons; and 3) shifting or backing out of the USEA, either due to nerve stretch, wire tugging, or granulomatous buildup on the undersie of the USEA. Failure analysess of previous USEAs implanted in transradial subjects is currently underway, and attempts are being made to improve the longevity of USEA viability in the peripheral nervous system (such as coiled wire bundles to provide strain relief). The declines in observed motor units and “working” electrodes were seen in one USEA before the other USEA, within the same subject. The USEA with inferior long-term viability was implanted in different nerves for each subject (median for HS1 and ulnar for HS2), which implies the failure is not due solely to the particular nerve in which it was implanted. However, reasons for using USEA in upper-limb amputees remain compelling. The main reason for continued use of the USEA is that USEAs are capable of providing regionally specific cutaneous and proprioceptive percepts in the phantom hand via electrical stimulation of the nerve [2], [3], [9](see also Chapter 2). Additionally, if USEAs can be made to better withstand the harsh environment of the peripheral arm nerves in active transradial amputees, it may be possible to use USEA signals or supplement iEMG signals to decode motor intent for long-term implants. Further, USEAs could prove useful after transhumeral amputations in which extrinsic hand muscles are no longer present to allow iEMG recordings. 111 In some amputees, the residual forearm muscles may be missing or otherwise insufficient for iEMG decodes. In this situation, USEA implants may be a useful source of motor intent signals. In previous studies, we showed that it is possible to decode up to 5 DOF using USEA signals alone (see Chapter 2), which thus may be sufficient for providing adequate motor intent signal if needed. Because sensory feedback through neural stimulation is a desired feature in future bidirectional neuromuscular prosthetic systems, it is important that either 1) muscle compound action potentials (MCAPs) are not induced by neural stimulation or 2) that motor decoding is not affected by the MCAP, as this could cause extraneous movements in the decode. There are a number of mitigation strategies which can be employed. The simplest strategy is to use only electrodes and stimulation amplitudes that do not induce MCAP for sensory feedback in closed-loop applications. However, use of such electrodes/amplitudes may be unavoidable due to the need to stimulate in a particular region on the phantom hand which cannot be reached with another electrode, or, due to unintentional motor fiber recruitment during variable-amplitude stimulation (which is potentially used for percept intensity discrimination). We presented a mitigation strategy in which decode interference was minimized by increasing the threshold on the affected DOF. Due to time constraints, this was the only mitigation strategy tested online. Other mitigation strategies include software filtering of the MCAPs or algorithm recalibration using only unaffected iEMG channel(s). It may be possible to identify and ignore the stimulation-associated MCAPs based on timing, and waveform morphology or frequency signature. Recalibration of the decode may be possible by forcing the feature selector to ignore the affected channel(s) and rerunning the training data through the calibration 112 routine. An additional factor to consider during neural stimulation in a bidirectional system is the electrical signal of the stimulation itself. The amplitude is large enough that the bioamplifier used in these experiments (Ripple, LLC) temporarily switches to a different setting during stimulation pulses in order to prevent the sensitive amplifiers from saturating. This results in a temporary blanking period which obscures any action potential signals during stimulation and for a short period afterwards (0.5 ms “fastsettle”). Because iEMG signals were the primary source of information for the decoders in these experiments, we were able to decode motor intent during stimulation without interference. However, if only neural signals are used for decoding, another mitigation strategy based on the timing of the stimulus delivery or waveform morphology would need to be employed. In our online testing, we showed that the decode calibration is stable for at least a week. Additionally, even long after declines in decode performance for a given calibration is seen offline, the subjects are still able to control many of the original DOFs trained on by adapting their flexion patterns to achieve desired movements. This stability will reduce the need for constant recalibration by the user. We also investigated the effect of arm position on decode performance. The muscle flexion patterns generated when the arm is moved or held in a different position than the calibration position have the potential to degrade decode performance by causing crosstalk and other unintended movements. In our preliminary studies, there was an effect, but it was difficult to quantify as we did not find a trend in online or offline performance, or subjective interpretation regarding the quality of control between 113 different arm positions. This effect is complicated by rapid adaptation of the user to the decode output using visual feedback. Further study of arm positional effects is needed. The techniques presented herein may be applicable to new advancements in neuromuscular interface technology. These advances include 1) targeted muscle reinnervation (TMR), where residual peripheral nerve branches are rerouted to existing muscles (after partial deinnervation) where EMG can be recorded via iEMG or sEMG [19], [53]; and 2) regenerative peripheral nerve interfaces, where branches of residual peripheral nerve are placed in autologous partial muscle grafts, which act as a bioamplifier for the peripheral nerve signal [54]. Such studies provide a new method to produce robust and specific motor signals from transhumeral amputees, or amputees with insufficient residual forearm muscles. Our methods may be applicable in these scenarios as well, allowing any level of upper limb amputee to control a high number individual DOFs of a prosthetic arm. 3.6 Conclusion In summary, we implemented a hybrid neuromuscular bidirectional prosthetic hand interface. We have shown that it was possible to use iEMG signals from the residual forearm of a transradial amputee to decode up to 10 individual and simultaneous DOFs. The iEMG decode calibrations were typically stable for up to a week, reducing the need for frequent and burdensome recalibration by the user. Our technique was robust in the presence of MCAP inadvertently caused by stimulation in a closed-loop simulation by the addition of configurable thresholds applied to the decode output. In the two subjects presented here, iEMG was the primary contributor to decoder information as the USEAs’ 114 motor neural signals were not as plentiful or specific as the iEMG signals, and the ability to detect motor units quickly deteriorated from weeks 4-8 for both subjects. Because of these findings, we recommend the use of iEMG from residual forearm muscles from transradial amputees (if possible) for motor decoding of a multiarticulate robotic hand with or without bidirectional sensory feedback capability. 3.7 Acknowledgements This work was sponsored by the Defense Advanced Research Projects Agency (DARPA) Biological Technologies Office (BTO) Hand Proprioception and Touch Interfaces (HAPTIX) program under the auspices of Dr. Doug Weber through the Space and Naval Warfare Systems Center, Contract No. N66001-15-C-4017, No. N66001-12C-4042. Additional funding was also provided via the National Institutes of Health (NIH NCATS Award No. 1ULTR001067). Additionally, other team members at the University of Utah, including Tyler Davis, D. Page, D. Kluger, J. Nieveen, J. George, C. Duncan, D. Hutchinson, D. Warren, and Gregory Clark contributed to this work either in data collection, experiment design, medical procedures, and/or data analysis. Anna Nebling helped make Figure 3.16. 115 Figure 3.1 Diagram of hybrid neuromuscular prosthetic interface concept. Neural recording and stimulation arrays are implanted in upper arm nerve branches (median, ulnar, radial) for recording of motor units and stimulation of sensory nerve fibers. iEMG leads are implanted in residual forearm muscles. A wearable or implantable computer processes and decodes neural and muscle signals. Decode outputs are translated to robotic prosthetic hand movemement commands. Illustration modified from [55] and [56]. 116 Figure 3.2 Photographic and X-ray images of implanted USEAs. (Left) Image of USEAs implanted into residual median and ulnar nerves from a previous study subject. A transparent collagen material is wrapped around the portion of nerve containing the array and held in place with vascular clips. (Right) X-ray image of iEMG array implanted into the residual forearm muscles of HS1. 3 iEMG leads were inserted in extensor muscle bundles, 4 in flexor muscle bundles, and 1 through the interosseous membrane through the flexor side. iEMG leads were inserted in or near the following muscles: Extensor pollicis longus (EPL); extensor carpi radialis longus and brevis (ECRL, ECRB); Extensor indici; flexor digitorum superficialis and profundus (FDP, FDS); flexor pollicis longus (FPL); flexor carpi ulnaris (FCU). 117 Figure 3. 3 Depiction of the 32-channel iEMG array (Ripple, LLC). In this prototype (not used in these experiments but similar in design), there are 8, variable-length leads with 4 cylindrical electrodes (light grey in color) at the distal end of the wires. Each lead is independent and flexible. At the far distal end of each lead, flexible plastic barbs (shown in blue) help to anchor the leads in place for the duration of the implant. At the proximal end of the prototype shown here, a wireless signal amplifier/digitizer is shown as the white “puck.” In contrast, the prototype used in our experiments contained leads which were bundled together and attached to a “Gator” board. The wire bundle passed through a percutaneous opening in the user’s arm. The board was connected to the Ripple Grapevine™ data acquisition system via either “Active Gators” or “Micro2+Stim” front ends (with a “Passive Gator” adaptor). Gator board and connectors are not shown. Illustration modified from [56]. 118 Figure 3.4 Schematic diagram for the hybrid neuromuscular decoding system used in these experiments. 1) USEA and iEMG data are collected and conditioned. 2) Features are calculated and buffered. 3) Selected feature channels and decode algorithm coefficients determined from the calibration phase. 4) Features and decode coefficients are used to compute decode outputs from current signals. User configurable gains, thresholds and integration are applied to decode outputs. 5) Decode outputs are converted into commands for the virtual or robotic prosthetic hand. 119 Figure 3.5 Hand matching task in the Mujoco VRE [50]. Two virtual hands are shown, 1) the target hand (nontransparent red, green, or yellow), and, 2) the virtual hand controlled by the user (beige semitransparent hand). In this task, trials consist of a new target hand position presented to the user. The user must then match the hand position of the target hand with the user-controlled hand. All DOFs must remain within a cumulative 5% tolerance for 1 s for the trial to be considered a success. Visual feedback regarding how close each DOF is to the target position by changing color from red (when the controlled hand is > 15% out of range) to yellow (when the hand is within 5 to 15% of the target position) to green (when the hand is within 5% of the target position). 120 Figure 3.6 Data display of a 6 s window of USEA and iEMG during an omnibus trial from HS2. a) and b) show heat maps of firing rates for all USEA electrodes. Blue represents a low rate, red represents a high rate. c) shows the heat map of iEMG electrode amplitude. The top 4 rows represent amplitude of single- ended channels, and the bottom 6 rows represent the within-lead differential pair amplitude. Each column is a single lead and corresponding differential pairs. d) shows the raw data (blue) from a USEA electrode (here, electrode 94 on the median array) and the threshold used for spike detection (red). Neuronal spiking is observed during movement epics as verified by EMG activity below in f). and waveform morphology shown right. e) shows the neuronal spikes detected from the selected USEA channel in this epic by threshold detection. The time scale and amplitude are consistent with neuronal action potentials. f) shows raw iEMG signal for a selected iEMG channel (blue) and the corresponding amplitude (red). Bursts of activity correspond to hand movements (flexion of fingers). 121 Table 3.1 Online decode results for HS1 and HS2 comparing USEA+iEMG, iEMG, and USEA data sources. Metrics include total DOFs achieved/DOFs trained on, total movements achieved/movements trained on, total successful trials/trials performed, percentage success, median trial time of successful trials (IQR), median time to first target touch (IQR), median number of target pass-throughs (IQR), and median path length (IQR). A list of missed movements is included as the last row (number missed/total attempted). Wilcoxon rank sum tests were performed target metrics on USEA and iEMG compared to iEMG only data. There was no significant difference for HS1 (all p>0.2) or HS2 (all p > 0.08). 122 Table 3.2: Offline decode analysis of data from 5 training sets from HS1 comparing neural (USEA) and iEMG signal sources. Mean squared error (MSE) and correlation coefficients were computed between decoded movements and the movement cue. On average, the EMG decode perfomed much better than the neural decode (t-test p < 0.02 for both metrics ). [Analysis for these data performed by Jacob Nieveen is not published elsewhere] 123 Figure 3.7 Boxplots of MSE and correlation coefficients from offline analysis using 5 training sets from HS1. EMG decodes are significantly better than neural decodes (t-test p < 0.02) in both metrics of MSE and correlation coefficient. [Offline analysis for these data performed by Jacob Nieveen, figure by SMW]. In this and subsequent boxplots, the middle red line is the median, upper and lower blue boundaries represent the upper and lower maximum and minimum points of the interquartile range (IQR), whiskers edges mark the data point + 2.7*σ above and below the IQR limits and not considered an outlier (99% coverage). Notches represent the 95% confidence interval of the true median [57]. 124 Figure 3.8 HS2 3-DOF hand matching task performance improves over time. The blue line shows the number of successful trials (out of 26). The red line shows the mean trial time (standard error bars). Dashed lines represent a linear regression fit to each data set. HS2 improves in both metrics, most likely due to learning of the task and control of the tested DOFs. 125 Figure 3.9 Three decode calibrations were tested 4 to 9 d post training for HS1. Shown is the number of movements achieved during each testing period. Here we define “stable” as 85% of the original number of movements. Set 1 did not remain stable from day 1 to day 9, whereas sets 2 and 3 were stable for 14 and 9 d, respectively. 126 Figure 3.10 3-DOF hand-matching task performance for HS2 using day 0 decode coefficients. The blue line represents the number of successful trials (of 26). The orange line represents the average trial time to completion (with standard error bars). Here, the performance trend was very stable through day 8. The peak performance was on day 8, followed by a decline in performance as seen in both metrics. 127 Figure 3.11 RMSE and correlation coefficients from offline analysis of 8-DOF training data for HS1. There is no obvious consistent trend of improvement or degradation over the course of the study. Note that HS1 performs well near the beginning of the study. Early data (post-implant weeks 1-4) not shown because < 8-DOF training was performed. 128 Figure 3.12 Correlation coefficient (left) and RMSE (right) of 8-DOF offline decodes from HS1, where calibration coefficients from one session are applied to all subsequent sessions. In the correlation figure, the mean value is represented by the central solid blue line and the standard error by the shaded area surrounding the central line. Individual traces of the RMSE for each training set are shown in the RMSE figure. Rapid deterioration of correlation and increase of RMSE occurs over the first 10 d, especially for early training sessions, although some later training sessions exhibit greater stability. 129 Figure 3.13 RMSE and correlation coefficients from offline analysis of 8-DOF training data for HS2. There is no obvious trend of improvement or degradation over the course of the study. Statistics not performed on these data. 130 Figure 3.14 Correlation coefficient r (left) and RMSE (right) of 8-DOF offline decodes from HS2. Left: The mean r value is represented by the central solid blue line, and the standard error by the shaded area surrounding the central line. Overall, the correlation coefficient decreases rapidly over the first 20-50 d, indicating loss of decode accuracy. However, these data do not differentiate among decodes based on training sessions performed early or later after implantation and related training. Right: Individual traces of the RMSE for each training set over subsequent testing sets. Calibration coefficients from one training session are applied to all subsequent test sessions for a given data line. Long data lines (i.e., lines with many data from many post-training sessions) indicate decodes based on early training sets that were performed shortly after implantation, whereas shorter lines (with fewer post-training test sessions) indicate decodes based on later training sets. Three trends can be observed: 1) RMSEs in general increase over the first 20-50 d after a training session, indicating some loss in decode accuracy. 2) The RMSE errors are higher for decodes based on early training sessions (i.e., longer data lines), whereas decodes based on later training sessions show lower RMSEs, matched for the same duration interval post-training. 3) RMSE errors are more variable across sessions for decodes based on early training sessions than for decodes based on later training sessions. These observations suggest that decode stability improves over the course of the implant and/or subsequent training, and that the subject’s performance may become more consistent with practice or with stability of the recorded signal. 131 Figure 3.15 Offline decode analysis of 8-DOF training sets performed by HS2 in different arm positions. Arm position has subtle effects on decode performance. In general, there is more crosstalk (see DOF 5). The top row shows the decode output (colored lines) and the movement cues (black line). The bottom row shows the cumulative standard error in the nonintended DOFs (crosstalk) for each DOF, during intended movements for other DOFs. The blue line represents the “standard” position, the red line represents the “down reach” position, the green line represents the “upward reach,” the magenta line represents the “across body reach,” and the black line represents the movement cue. The standard error is shown by the shaded area around the solid middle line which is the mean. The “standard” arm position coefficients and baseline data were used in all subsequent arm position calculations. 132 Figure 3.16 Offline decode examples of HS2 training data while arm was in position 1 (standard) on the right panel and position 2 (upward reach) on the left panel. Blue lines represent the training movement cue. Red lines represent the decoded output. Here decode coefficients from training in position 1 were used on both test sets. In general, there is more crosstalk (see red traces especially top 6 movements on the right panel) when the incorrect position coefficients are used. 133 Figure 3.17 Using the “standard” position decoder (first bar in each set), HS2 was asked to subjectively rate his sense of control over each DOF in both the “up reach” (second bar in each set) and “down reach” (third bar) positions. There was no global trend in these ratings. Statistical analysis was not performed on these data. 134 Figure 3.18 Attempts to “rescue” the decode performance by subtracting out a baseline taken in the correct position were not successful. In both cases where the proper position of baseline was used, nonsignificant changes are seen. The first column, standard decode with standard-baseline subtracted (S) has the highest correlation and lowest RMSE. The next two columns show arm down-reach with standard-baseline subtracted (D-S) and down-reach with down-reach baseline subtracted (D-D). The next two columns show arm up-reach with standard-baseline subtracted (U-S), and up-reach with up-reach-baseline subtracted. Rank sum tests revealed all p values > 0.50. 135 Figure 3.19 Example of stimulation artifact in the decode output. Top panel shows raw stimulation waveforms (blue lines) and peak-peak amplitudes (read lines) for 1 electrode stimulated at 100 Hz, 0.5 s trains. Middle panel shows the amplitude recorded from iEMG leads revealing stimulation induced MCAPs. Different colored traces represent iEMG features. Bottom panel shows resulting, correlated excursion in the decoded output (blue and red are DOFs 1 and 2, respectively). 136 Figure 3.20 Hand-matching task results for HS2 in the presence or absence of MCAPinducing stimulation. a) shows the overall results looking at the trial speed (path efficiency) and b) average trial time. There was no significant difference (ttest p = 0.89) for these metrics for the stimulation on/off conditions. 137 Figure 3.21 Right panel shows “working electrodes” (with z < 500 kOhms) for the ulnar USEA for HS1 through post-implant day 87. The number of working electrodes declines from approximately 68/96 to approximately 58/96 (15% decrease). Left panel shows a boxplot of impedance values for the ulnar USEA “working” electrodes over the course of the implant. In both figures, blue impedance data were measured using the 96-channel Active Gator (Ripple, LLC), and orange data were measured using Micro front end modules (Ripple, LLC) [Analysis and figures by David Kluger, not published elsewhere]. 138 Figure 3.22 Top panel shows “working electrodes” (with z < 500 kOhms) for the median USEA for HS1 through post-implant day 87. The number of working electrodes declines from approximately 80/96 to approximately 50/96 (38% decrease). Bottom panel shows a boxplot of impedance values for the median USEA “working” electrodes over the course of the implant. In both figures, blue impedance data were measured using the 96-channel Active Gator (Ripple, LLC), and orange data were measured using Micro front end modules (Ripple, LLC) [Analysis and figures by David Kluger, not published elsewhere]. 139 Figure 3.23 Number of “working” USEA electrodes (z < 500 kOhms) declines for both median (blue trace) and ulnar (red trace) arrays for HS2 over the first 3 months. Over the first 3 months, the number of working electrodes declined from approximately 88/96 to approximately 48/96 (45% decrease) for the ulnar array, and from approximately 90/96 to approximately 50/96 (44% decrease) for the median array [Analysis and figures by David Kluger, not published elsewhere]. 140 Figure 3.24 Impedances of “working” electrodes (z < 500 kOhms) for the median (blue trace) and ulnar (red trace) USEA electrodes over the first 3 months of the implant [Analysis and figures by David Kluger, not published elsewhere]. 141 Figure 3.25 Top figure shows driven motor units from median and ulnar USEAs from HS1 over time. The bottom figure shows the driven motor units for HS2. Units were observed in HS1 through post-implant day 83 from both arrays, and in HS2 through day 96 from the median array. 142 Figure 3.26 EMG signal amplitude (top) and SNR (bottom) during omnibus sessions for HS1 from post-implant day 8 to 88. EMG amplitude trends upwards while SNR remains steady. This indicates that the EMG signal does not deteriorate over the course of the approximately 3-month HS1 study. Statistics analysis not performed on these data. 143 Figure 3.27 EMG signal amplitude (top) and SNR (bottom) during omnibus sessions for HS2 from post-implant day 8 to 343. EMG amplitude trends upwards while SNR remains steady. This indicates that the EMG signal does not deteriorate over the course of the approximately year-long HS2 study. Statistics not performed on this data set. 144 3.8 Additional video files File name: “P201501 7-DoF targets_v20151015-1808_largeFormat.mp4” In this Video, HS1 performs a target-touching task. Examples from a 7-DOF test are shown. HS2 used a hybrid USEA-iEMG decode for these trials. File name: “HS2_Luke_Hand_Highlight Reel_v20170327.mp4” In this video, HS2 uses the DEKA Luke hand (with iEMG decode) to do several ADLs , including picking up/pouring from a bottle, grabbing and squeezing a power drill, using a fork to pick up food, and peeling a banana. Sensors on the thumb, index finger, ring finger and little finger selectively activated amplitude and frequency modulated neural stimulation of a population of 13 electrodes (10 on the median array and 3 on the ulnar) proportional to pressure reading. 145 3.9 List of abbreviations ADL: Activity of Daily Living DOF: Degree of Freedom EMG: Electromyogram HS1, HS2: HAPTIX Subjects 1 and 2 IMEG: Intramuscular EMG IQR: Interquartile Range KF: Kalman Filter MCAP: Muscle Compound Action Potential USEA: Utah Slanted Electrode Array VPH: Virtual Prosthetic Hand VRE: Virtual Reality Environment Z: impedance 3.10 References [1] “Hand Proprioception and Touch Interfaces (HAPTIX).” [Online]. Available: http://www.darpa.mil/program/hand-proprioception-and-touch-interfaces. [Accessed: 12-May-2017]. [2] T. S. Davis et al., “Restoring motor control and sensory feedback in people with upper extremity amputations using arrays of 96 microelectrodes implanted in the median and ulnar nerves,” J. Neural Eng., vol. 13, no. 3, p. 36001, Jun. 2016. [3] G. A. Clark et al., “Using multiple high-count electrode arrays in human median and ulnar nerves to restore sensorimotor function after previous transradial amputation of the hand,” in 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2014, pp. 1977– 1980. [4] G. S. Dhillon and K. W. Horch, “Direct neural sensory feedback and control of a 146 prosthetic arm,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 13, no. 4, pp. 468– 472, 2005. [5] G. S. Dhillon, S. M. Lawrence, D. T. Hutchinson, and K. W. Horch, “Residual function in peripheral nerve stumps of amputees: Implications for neural control of artificial limbs,” J. Hand Surg., vol. 29, no. 4, pp. 605–615, Jul. 2004. [6] G. S. Dhillon, T. B. Krüger, J. S. Sandhu, and K. W. Horch, “Effects of short-term training on sensory and motor function in severed nerves of long-term human amputees,” J. Neurophysiol., vol. 93, no. 5, pp. 2625–2633, May 2005. [7] S. Raspopovic et al., “Restoring natural sensory feedback in realtime bidirectional hand prostheses,” Sci. Transl. Med., vol. 6, no. 222, p. 222ra19–222ra19, 2014. [8] P. M. Rossini et al., “Double nerve intraneural interface implant on a human amputee for robotic hand control,” Clin. Neurophysiol., vol. 121, no. 5, pp. 777– 783, May 2010. [9] D. M. Page, “Restored hand sensation in human amputees via utah slanted electrode array stimulation enables performance of functional tasks and meaningful prosthesis embodiment,” Ph.D. Thesis, University of Utah, 2016. [10] C. Antfolk, M. D’Alonzo, B. Rosén, G. Lundborg, F. Sebelius, and C. Cipriani, “Sensory feedback in upper limb prosthetics,” Expert Rev. Med. Devices, vol. 10, no. 1, pp. 45–54, Jan. 2013. [11] D. W. Tan, M. A. Schiefer, M. W. Keith, J. R. Anderson, J. Tyler, and D. J. Tyler, “A neural interface provides long-term stable natural touch perception,” Sci. Transl. Med., vol. 6, no. 257, p. 257ra138-257ra138, Oct. 2014. [12] D. M. Page, “Restoration of sensory and motor hand function via two Utah Slanted Electrode Arrays (USEAs) in residual arm nerves after prior hand amputation,” in Neuroscience Meeting Planner, Washington, DC, 2014, p. 636.19. [13] M. S. Johannes, J. D. Bigelow, J. M. Burck, S. D. Harshbarger, M. V. Kozlowski, and T. Van Doren, “An overview of the developmental process for the modular prosthetic limb,” Johns Hopkins APL Tech. Dig., vol. 30, no. 3, pp. 207–216, 2011. [14] L. Resnik, S. L. Klinger, and K. Etter, “The DEKA Arm: Its features, functionality, and evolution during the Veterans Affairs Study to optimize the DEKA Arm,” Prosthet. Orthot. Int., vol. 38, no. 6, pp. 492–504, 2014. [15] H. L. Benz et al., “Upper extremity prosthesis user perspectives on unmet needs and innovative technology,” in 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2016, pp. 287–290. 147 [16] D. Perruchoud, I. Pisotta, S. Carda, M. M. Murray, and S. Ionta, “Biomimetic rehabilitation engineering: the importance of somatosensory feedback for brainmachine interfaces,” J. Neural Eng., vol. 13, no. 4, p. 41001, Aug. 2016. [17] C. Antfolk et al., “Artificial redirection of sensation from prosthetic fingers to the phantom hand map on transradial amputees: vibrotactile versus mechanotactile sensory feedback,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 21, no. 1, pp. 112– 120, 2013. [18] P. D. Marasco, K. Kim, J. E. Colgate, M. A. Peshkin, and T. A. Kuiken, “Robotic touch shifts perception of embodiment to a prosthesis in targeted reinnervation amputees,” Brain, vol. 134, no. 3, pp. 747–758, Mar. 2011. [19] P. Zhou et al., “Decoding a new neural–machine interface for control of artificial limbs,” J. Neurophysiol., vol. 98, no. 5, pp. 2974–2982, Nov. 2007. [20] J. J. Baker et al., “Decoding individuated finger flexions with Implantable MyoElectric Sensors,” Conf. Proc. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. IEEE Eng. Med. Biol. Soc. Conf., vol. 2008, pp. 193–196, 2008. [21] N. G. Hatsopoulos and J. P. Donoghue, “The science of neural interface systems,” Annu. Rev. Neurosci., vol. 32, pp. 249–266, 2009. [22] S. Micera et al., “Decoding of grasping information from neural signals recorded using peripheral intrafascicular interfaces,” J. Neuroengineering Rehabil., vol. 8:53, 2011. [23] L. R. Hochberg and D. M. Taylor, “Intuitive prosthetic limb control,” The Lancet, vol. 369, no. 9559, pp. 345–346, Feb. 2007. [24] L. H. Smith, T. A. Kuiken, and L. J. Hargrove, “Evaluation of linear regression simultaneous myoelectric control using intramuscular EMG,” IEEE Trans. Biomed. Eng., vol. 63, no. 4, pp. 737–746, Apr. 2016. [25] C. Cipriani, J. L. Segil, J. A. Birdwell, and R. F. Weir, “Dexterous control of a prosthetic hand using fine-wire intramuscular electrodes in targeted extrinsic muscles,” IEEE Trans Neural Syst Rehabil Eng, vol. 99, pp. 1–1, 2014. [26] L. J. Hargrove, K. Englehart, and B. Hudgins, “A comparison of surface and intramuscular myoelectric signal classification,” IEEE Trans. Biomed. Eng., vol. 54, no. 5, pp. 847–853, May 2007. [27] E. N. Kamavuako, E. J. Scheme, and K. B. Englehart, “On the usability of intramuscular EMG for prosthetic control: A Fitts’ Law approach,” J. Electromyogr. Kinesiol., vol. 24, no. 5, pp. 770–777, Oct. 2014. 148 [28] Warwick K, Gasson M, Hutt B, and et al, “The application of implant technology for cybernetic systems,” Arch. Neurol., vol. 60, no. 10, pp. 1369–1373, Oct. 2003. [29] L. J. Hargrove, G. Li, K. B. Englehart, and B. S. Hudgins, “Principal components analysis preprocessing for improved classification accuracies in patternrecognition-based myoelectric control,” IEEE Trans. Biomed. Eng., vol. 56, no. 5, pp. 1407–1414, May 2009. [30] G. Purushothaman and K. K. Ray, “EMG based man–machine interaction—A pattern recognition research platform,” Robot. Auton. Syst., vol. 62, no. 6, pp. 864– 870, Jun. 2014. [31] J.-U. Chu, I. Moon, and M.-S. Mun, “A realtime EMG pattern recognition system based on linear-nonlinear feature projection for a multifunction myoelectric hand,” IEEE Trans. Biomed. Eng., vol. 53, no. 11, pp. 2232–2239, Nov. 2006. [32] L. Hargrove, Y. Losier, B. Lock, K. Englehart, and B. Hudgins, “A realtime pattern recognition based myoelectric control usability study implemented in a virtual environment,” in Engineering in Medicine and Biology Society, 2007. EMBS 2007. 29th Annual International Conference of the IEEE, 2007, pp. 4842–4845. [33] K. Englehart, B. Hudgins, P. A. Parker, and M. Stevenson, “Classification of the myoelectric signal using time-frequency based representations,” Med. Eng. Phys., vol. 21, no. 6–7, pp. 431–438, Jul. 1999. [34] Y. Huang, K. B. Englehart, B. Hudgins, and A. D. Chan, “A Gaussian mixture model based classification scheme for myoelectric control of powered upper limb prostheses,” IEEE Trans. Biomed. Eng., vol. 52, no. 11, pp. 1801–1811, 2005. [35] W. Wu, M. J. Black, D. Mumford, Y. Gao, E. Bienenstock, and J. P. Donoghue, “Modeling and decoding motor cortical activity using a switching Kalman filter,” IEEE Trans. Biomed. Eng., vol. 51, no. 6, pp. 933–942, Jun. 2004. [36] W. Wu, Y. Gao, E. Bienenstock, J. P. Donoghue, and M. J. Black, “Bayesian population decoding of motor cortical activity using a Kalman filter,” Neural Comput., vol. 18, no. 1, pp. 80–118, 2006. [37] J. Egan, J. Baker, P. A. House, and B. Greger, “Decoding dexterous finger movements in a neural prosthesis model approaching real-world conditions,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 20, no. 6, pp. 836–844, Nov. 2012. [38] D. J. Warren et al., “Recording and decoding for neural prostheses,” Proc. IEEE, vol. 104, no. 2, pp. 374–391, Feb. 2016. [39] W. Q. Malik, W. Truccolo, E. N. Brown, and L. R. Hochberg, “Efficient decoding with steady-state Kalman filter in neural interface systems,” IEEE Trans. Neural 149 Syst. Rehabil. Eng. Publ. IEEE Eng. Med. Biol. Soc., vol. 19, no. 1, pp. 25–34, Feb. 2011. [40] A. Branner and R. A. Normann, “A multielectrode array for intrafascicular recording and stimulation in sciatic nerve of cats,” Brain Res. Bull., vol. 51, no. 4, pp. 293–306, Mar. 2000. [41] R. L. Rennaker, J. Miller, H. Tang, and D. A. Wilson, “Minocycline increases quality and longevity of chronic neural recordings,” J. Neural Eng., vol. 4, no. 2, pp. L1–L5, Jun. 2007. [42] P. J. Rousche and R. A. Normann, “A method for pneumatically inserting an array of penetrating electrodes into cortical tissue,” Ann. Biomed. Eng., vol. 20, no. 4, pp. 413–422, Jul. 1992. [43] L. Spataro et al., “Dexamethasone treatment reduces astroglia responses to inserted neuroprosthetic devices in rat neocortex,” Exp. Neurol., vol. 194, no. 2, pp. 289– 300, Aug. 2005. [44] Y. Zhong and R. V. Bellamkonda, “Dexamethasone coated neural probes elicit attenuated inflammatory response and neuronal loss compared to uncoated neural probes,” Brain Res., vol. 1148, pp. 15–27, May 2007. [45] P. Ventimiglia, “Design of a human hand prosthesis,” Worcester Polytechnic Institute, 2012. [46] K. L. Kroeker, “Engineering sensation in artificial limbs,” Commun. ACM, vol. 54, no. 4, pp. 16–18, 2011. [47] J. G. Nieveen et al., “Channel selection of neural and electromyographic signals for decoding of motor intent,” presented at the Myoelectric Control (MEC) Symposium 2017, Fredericton, New Brunswick, CA, 2017. [48] W. Wu, A. Shaikhouni, J. P. Donoghue, and M. J. Black, “Closed-loop neural control of cursor motion using a Kalman filter,” in 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2004. IEMBS ’04, 2004, vol. 2, pp. 4126–4129. [49] G. Welch and G. Bishop, “An introduction to the Kalman filter,” [Online] Course material from University of North Carolina-Chapel Hill, July 24, 2006. [50] V. Kumar and E. Todorov, “MuJoCo HAPTIX: A virtual reality system for hand manipulation,” in Humanoid Robots (Humanoids), 2015 IEEE-RAS 15th International Conference on, 2015, pp. 657–663. [51] M. B. Christensen, H. A. C. Wark, and D. T. Hutchinson, “A histological analysis 150 of human median and ulnar nerves following implantation of Utah slanted electrode arrays,” Biomaterials, vol. 77, pp. 235–242, Jan. 2016. [52] M. B. Christensen, S. M. Pearce, N. M. Ledbetter, D. J. Warren, G. A. Clark, and P. A. Tresco, “The foreign body response to the Utah Slant Electrode Array in the cat sciatic nerve,” Acta Biomater., vol. 10, no. 11, pp. 4650–4660, Nov. 2014. [53] T. A. Kuiken et al., “Targeted reinnervation for enhanced prosthetic arm function in a woman with a proximal amputation: a case study,” The Lancet, vol. 369, no. 9559, pp. 371–380, Feb. 2007. [54] Z. T. Irwin et al., “Chronic recording of hand prosthesis control signals via a regenerative peripheral nerve interface in a rhesus macaque,” J. Neural Eng., vol. 13, no. 4, pp. 46007–46007, 2016. [55] Ripple LLC, “NIH and Ripple SPARC Memorandum of Understanding (MOU): Exhibit C-Company Materials.” [Online] Available: https://commonfund.nih.gov/sites/default/files/Ripple%20Info.pdf [Accessed Jul 2017]. [56] “About Ripple Neuro Ripple Neuro.” [Online]. http://rippleneuro.com/about?tab=projects. [Accessed: 31-Jul-2017]. Available: [57] “Box plot MATLAB boxplot.” [Online]. Available: https://www.mathworks.com/help/stats/boxplot.html;jsessionid=46c7b9c556e226 b594d153ea4d36. [Accessed: 05-Aug-2017]. CHAPTER 4 10 DEGREE-OF-FREEDOM DECODES FROM FOREARM MUSCLES OF INTACT AND TRANSRADIAL HUMAN AMPUTEES USING SURFACE ELECTROMYOGRAM SIGNALS 4.1 Abstract Common present strategies for decoding motor signals from surface electromyogram (sEMG) signals from the forearm of intact or amputee subjects provide limited control. For example, direct control methods typically provide only 3-4 degrees of freedom (DOF), and pattern classification approaches allow only a limited number (typically < 16) of predefined grasps. In this chapter, we present a method for simultaneously decoding 10 independent DOFs from surface electrodes placed on the forearm muscles. Because independent DOFs can be individually controlled, the user may construct a large number of novel possible grasp patterns to which the decode algorithm has not been explicitly trained. Here we demonstrate the ability to decode up to 10 DOFs (20 movements) in three transradial amputee subjects and three intact subjects, using two types (wet or dry) of high-channel-count (14-32) monopolar, surface EMG electrodes and a modified Kalman Filter (KF) decode algorithm. Decode calibrations were stable over the course of a few hours, and somewhat stable between doffing/donning electrodes. We show that using nonstandard techniques such as software differenced pairs, and applying small 152 thresholds to the decode output significantly (p < 0.001) improves offline performance. Further, within a session, our methods work comparably or better using sEMG, compared with intramuscular EMG (iEMG) recordings. These studies support the potential use of sEMG and our decoding method to provide control for a multiarticulated prosthetic hand. 4.2 Introduction In this chapter, I explore using decode advancements we have made during the previous studies (see Chapters 2 and 3) in the setting of high count (³14) surface EMG electrodes, and demonstrate that it is possible to control up to 10 independent DOFs of a virtual or robotic prosthetic hand. Within this context, we will also investigate 1) the use of different types (wet and dry) and numbers of EMG electrodes; 2) the stability of decode calibration within and between experimental sessions in which electrodes have been doffed and donned; 3) whether it is possible to decode movements controlled by intrinsic hand muscles using only signals from extrinsic hand muscles; 4) whether high-channel-count sEMG is a suitable substitute for high-channel-count intramuscular EMG (iEMG); and 5) alternative approaches to industry-standard signal processing of sEMG signals. 4.3 Background There is a growing body of literature that supports the use of sEMG, and/or iEMG (fine wire, implantable myoelectric sensors (IMES), or other implanted electrodes) for decoding intended movement from the residual forearm, or upper arm/chest muscles following TMR procedure. [1]–[7]. Currently, the most common control strategy for commercially available electromyographic-driven prosthetics is the “direct” control 153 method. In this method, differential signals from pairs of electrodes are mapped directly to a single motion. Using such methods, it is difficult to achieve high-DOF decodes using sEMG due to the overlapping signal sources of the muscles below the electrodes. Commercially available EMG prosthetics using this method typically have one set of electrodes over the forearm digit extensors, and another set over forearm digit flexors. These hands are limited to 1-DOF control at a time. These prosthetics may be capable of more than one movement (i.e., hand open/close, or wrist rotate); however, the user must “clutch” between movements, or perform movements manually using their intact hand. Direct control strategies advanced by Wier et al. have demonstrated 2-DOF decodes using sEMG, and 4 DOF decodes using fine-wire iEMG [6]. Limitations of this approach include the small number of DOFs that can be controlled and the crosstalk between the DOFs. Another common decode approach is to use a classifier to decode grasps, as opposed to individual DOFs. A classification decoder is trained using categorical grips or movements. Classification techniques such as principal component analysis (PCA), support vector machine (SVM), neural networks (NN), and linear discriminant analysis (LDA) are commonly used [8]–[10]. High accuracy classification of 12-16 hand positions have been achieved using high-channel-count (15-100) surface EMG [2], [11]. Although classification approaches have proven to be highly accurate (approaching 100% accuracy offline) [7], they offer a limited number of grips the user can control, are typically binary (“open/close” mode with no partial outputs), and are sequential in nature (only one classified movement can be performed at a time, in a noncontinuous mode). Additionally, users must train on all categorical grips desired for the classification “vocabulary,” which 154 may be burdensome. Yet another decode strategy is to use regression algorithms to predict individual DOFs. In contrast to classification methods, regression strategies provide realtime, intuitive proportional control of individual degrees of freedom, allowing for an unlimited number of grip possibilities and arm positions. Smith, Hargrove et al. demonstrated 3 DOF of proportional control using fine-wire EMG implanted in the forearm of intact subjects [4]. Our approach involves using a modified Kalman Filter to estimate the position of individual virtual fingers from select EMG features. This method is a type of predictive Bayesean estimator that has previously been used to successfully decode intended movements from firing rates of a population of neurons in cortex and from the iEMG signals of nonhuman primates [12]–[14]. Details are discussed in Chapter 3. The typical sEMG feature used in commercial prosthetics and in many decoding strategies is the power computed from hardware-differenced electrodes placed in pairs over the muscles. Although this approach removes common noise between adjacent electrodes, it reduces the number of signal sources by a factor of two. In our decode strategy, we use monopolar, or “single-ended,” signals from individual electrodes and compute the difference between all possible pairs of electrodes. This, in effect, creates n-choose-2 number of virtual signal sources. A similar approach was successfully by Zhou et al. to decode signals from a 100 channel grid in an amputee who had undergone targeted muscle re-innervation (TMR)[2]. Zhou used sEMG features created by differencing all neighboring electrode signals. Several groups have compared sEMG to iEMG decodes [6], [11], [15]. Because 155 implanted electrodes tend to provide a higher amplitude and more specific signal, it is generally assumed that iEMG will be the preferred method for signal acquisition for motor decodes. Direct comparisons between the two methods have shown mixed results. Cipriani et al. showed that fine wire EMG was a better signal source [6], whereas Hargrove et al. found no significant difference [11]. Another group (Kamavuako et al.) has chosen to combine sEMG and iEMG into a single decode, showing that it may be possible to supplement sEMG with iEMG, or vice versa [16]. iEMG electrodes may also be more stable across sessions, allowing longer use of a given set of parameters for a decode algorithm. 4.4 Methods 4.4.1 Study population In this IRB approved study (University of Utah, Institutional Review Board), three male, human, transradial amputees (Subjects Transradial_S1, Transradial_S2, and Transradial_S3, mean age/standard deviation 52/6.6 y.), and three intact individuals (Intact_S1, Intact_S2, Intact_S3), two males and one female, mean age/standard deviation 33/8.7 y.) were recruited to participate in the study. Transradial_S1 was a left-dominant, left-sided transradial amputee, who underwent amputation 24.1 cm distal to the medial epicondyle 25 years prior to the study following a crush injury. Transradial_S2 was a rightdominant 45-year-old bilateral, transradial amputee who underwent amputations approximately 15 cm distal to the medial epicondyle on each following an electrical injury 26 years prior to the study. Transradial_S3 was a left-dominant, left-sided transradial amputee, who underwent amputation 18 cm distal to the medial epicondyle 15 years prior 156 to the study following an electrical injury. As part of another study, Transradial_S3 was at the time implanted with a 32-channel intramuscular EMG array in his left residual forearm [See Chapter 3 for details]. Transradial_S1 and Transradial_S2 had previously been implanted with USEAs in median and ulnar nerves (See Chapter 2 and 3), and Transradial_S2 also had previously been implanted with iEMG. All implants from Transradial_S1 and Transradial_S2 had been removed > 1.3 years prior to this study. Intact_S1 was left-dominant, and Intact_S2 and Intact_S3 were both right-dominant. All subjects offered voluntary consent. 4.4.2 Experimental methods 4.4.2.1 Electrode placement The left forearms or residual forearm of study participants were instrumented with either 24-mm diameter “sticky,” ECG-style medical-grade electrodes (Coviden, Mansfield, MA, USA), or a neoprene sleeve studded with dry, 12.5-mm diameter stainless steel “snap” electrodes (Motion Control, Salt Lake City, UT, USA) or 15-mm diameter dry stainless steel “button” electrodes (Northwest Tarp and Canvas, Bellingham, WA, USA). See Figure 4.1 for more details on each type of configuration. In “dry” metal electrode configurations, conductive gel was not applied under the electrode. However, “sticky” electrodes all contained conductive hydrogel under the lead. All configurations contained 14-32 electrodes, clustered above flexors and extensors of the digits and wrist. Due to equipment availability or forearm surface area limitations, variable numbers of electrodes were used in certain configurations, as described in figures and tables. 157 “Button” electrodes were used in 32-channel sleeves. We chose to prototype with this style of electrode due to cost of constructing a sleeve with 32 electrodes. “Buttons” cost approximately $0.10 USD each whereas commercially used “snap” electrodes cost approximately $25-$30 USD. Thus, instrumenting a sleeve with 32 electrodes would cost > $900 USD. The iEMG array (prototype by Ripple LLC, Salt Lake City, UT, USA) consisted of 8 leads (approximately 20 mm long) containing 4 cylindrical electrodes at the distal end (approximately 2 mm wide, spaced at approximately 2 mm, exact dimensions not available), tethered together at the proximal end. Five leads were placed in flexor muscle bundles and 3 in extensor muscle bundles (See Chapter 3 for more details). 4.4.2.2 Experiment types Several types of experiments were performed: 1) Acute experiments in which target-touching tasks were attempted immediately after training. These experiments were performed to assess the greatest number of independent DOFs intact and amputee subjects could perform with sEMG, and to assess the viability of different electrode types (e.g., “button” vs. “snap” electrodes; see section on Electrode Placement below for details). Systematic comparisons among electrode types were not performed here. Rather, we addressed the question of whether dry electrodes (rather than “sticky” electrodes, which potentially have a better signal due to the conductive gel) can be used for sEMG decodes, and whether “button” electrodes are sufficient quality for collecting sEMG signals as compared with the commercially-used “snap” electrodes. 2) Stability experiments where target-touching tasks were attempted > 1 h after 158 training (in some cases after removing and replacing electrodes), using the same algorithm coefficients. These experiments were performed in order to verify the stability of decode calibration during continuous use, and across sessions where electrodes have been doffed and re-donned. 3) Comparison of iEMG to sEMG by performing multiple sessions of targettouching tasks, using either iEMG or sEMG decodes. Here iEMG and sEMG are simultaneously collected, but only one decode, sEMG or iEMG, can be used at a time during target-touching tasks. In the first attempt, iEMG and sEMG decodes were calibrated using separate training sets (each 8 DOF, but collected sequentially). In the second attempt, both iEMG and sEMG decodes used the same training time period to generate algorithm coefficients. 4) Post-hoc “offline” analysis, comparing results of various signal processing strategies. In this manuscript, we compare using forced differential pairs (field standard) to using unconstrained software differential pairs. 4.4.2.3 Signal collection and processing EMG data were collected at 1 kHz using a multichannel bioamplifier (Grapevine System by Ripple LLC, Salt Lake City, Utah). Data were filtered with a 15-375 Hz bandpass and 60/120/180 Hz notch software filters. For experiments comparing sEMG and iEMG, both sEMG and iEMG were collected simultaneously with the same processing applied. Data were streamed and further processed in realtime and using custom-built software written in Matlab and Labview programming languages. 159 4.4.2.4 Virtual environment Here we define a degree of freedom (DOF) as the motion in a digit or the wrist in a single linear or rotational axis in either direction. Thus, a single DOF includes deviation from a rest position in both flexion and extension direction (e.g., a positive value indicates motion in the flexion direction and a negative value indicates motion in the extension direction). We define an individual ‘movement’ as a DOF including the directional component for each DOF. The virtual hand used in these experiments was the Java based Musculoskeletal Modeling Software (MSMS)[17]. Although the virtual hand has 24 actuating joints, interphalangeal joint movements were tied to the metacarpal phalangeal joint movements, giving the virtual hand a total of 12 DOF (flexion/extension of digits 1-5; adduction/abduction of digits 1, 2, 4, and 5; flexion/ extension, ulnar/radial deviation, and pronation/supination of the wrist). See Table 4.1 for a list and description of commonly used DOFs. Note that for these experiments, DOFs 7 and 9 (index and pinky ad/abduction) were always tied together (i.e., moved in unison at the same amplitude) and that DOF 8 (ring finger ab/adduction) was never used. 4.4.2.5 Decode algorithm calibration To train the decode algorithm, subjects are instructed to imitate with their phantom or intact hand a series of single-DOF virtual-hand movements shown on a computer screen while EMG data are collected and saved. Subjects were instructed to keep their arm in approximately the same position and orientation for training and testing sessions (i.e., same amount of elbow flexion and wrist pronation). Training sets included 5 to 10 trials of each movement, with each movement trial lasting for 1 to 2 s. The complete training session generally lasted 5-10 min. In these experiments, subjects trained 160 on 6, 8 or 10 DOFs and one combination full-grasp movement (D1-5 flexion/ extension). In 6-DOF training sets, DOFs 1-5 and 9 were included. In 8-DOF training sets, DOFs 1-6, 10, and 12 were included. In 10-DOF training sets, DOFs 1-6, 7/9 (combined), and 10-12 were included. Occasionally extensions of DOFs 4 and 5 (ring and pinky finger) were excluded due to the difficulty of performing these motions independently during training and limited utility in virtual ADL tasks. 4.4.2.6 Feature extraction and decoding strategy Amplitude of single-ended and software-differenced channel pairs were used as the features for the decode algorithm. Amplitude was computed by rectifying the raw voltage of single-ended (up to 32 total), and software-differenced pairs (up to 496 total, from 32choose-2 pairs). The data were then smoothed with a boxcar filter (300 ms window length). Features were updated every 33 ms. Baseline data while the subject was not moving were collected for 15 s prior to decode calibration. The mean of the baseline data (mean firing rate for neural data and mean amplitude for iEMG data) was subtracted from the feature data in realtime. 4.4.2.7 Data alignment Following the training period, feature data were aligned with movement cues in order to improve feature selection and the generation of decode algorithm coefficients. Here, feature data were shifted globally by the lag which maximizes the correlation between all features and movement cues. 161 4.4.2.8 Feature selection Features were then down-selected from the 528 possible features (from 32 single ended channels + 32-choose-2 differential pairs) using the Grahm-Schmidt orthoginalization method [18], with a limit of 48 features. The time-aligned and down-selected features were used to generate coefficients for the decode algorithm (calibration). 4.4.2.9 Realtime decoding For realtime decoding, down-selected features were computed and passed through a position-estimating modified Kalman Filter (KF) decoder (described in detail in Chapter 3), which was updated at 30 Hz (33 ms). To further minimize crosstalk between DOFs, user-selected gains (G > 0) and thresholds (which must have the range -1< Threshold < 1) were applied to the outputs as described below. 4.4.2.10 Modified Kalman filter decoder Kalman filter (KF) decoding is a commonly used approach for decoding motor intent from neural signals in the central nervous system. Further detail can be found in [12], [13], [19]–[22]. The KF decoder was chosen for this application due to its stability and accuracy given noisy inputs such sEMG signals. The KF is a type of Bayesian estimator that relies on current measurements of a feature, in addition to past estimates of the output state to make a prediction about the current state. We have modified the standard KF to account for the user selected output gains and thresholds mentioned above. Additionally, the outputs are bounded to prevent outputs that run out of a fixed -1 to 1 range. 162 4.4.2.11 Control strategies Although the decoder estimates positions, the outputs can be integrated over time, providing several control strategies for to the user including the standard position control, velocity control (“latching”), or a hybrid mode (“leaky integration”). In position mode, no integration was applied to the output. As a result, the virtual hand moved with subject effort, but relaxed back to a neutral position (corresponding to decode outputs of 0) when no effort was made. In order to provide velocity and hybrid modes, integration was applied to the position estimate of the decode output. In the case of velocity control, outputs were continuously integrated, causing the hand to hold the position corresponding to the sum of previous efforts. This allowed subjects to maintain grip positions without sustained effort. In the hybrid position-velocity mode, a leaky integration window (length is user specified ³ 0.6 s). In this scenario, the hand relaxes slowly back to neutral position when no effort is attempted. Outputs of the decode in a user-selected integration mode were then used to control the virtual hand in real time. Figure 4.2 depicts a schematic diagram of the signal processing steps during realtime decoding. 4.4.3 Data analysis To verify simultaneous and individual control of multiple DOF online, a targettouching task was designed. In this task, one or more spherical virtual target(s) were positioned away from the resting position of one or more digit(s)/wrist along the arc of movement. A successful trial resulted when the subject moved the specified digit(s) inside the radius of the spherical target(s) for at least 300 ms while keeping the other DOFs in resting position. The virtual targets do not exclude the virtual fingers, and fingers could pass all the way through the spheres. To provide visual feedback, the target spheres 163 changed color from red to green when the desired finger entered the target sphere. The target radii were typically set to be 15% of the arc of motion in one direction. The targets were placed at 50% of the arc of motion in either the flexion or extension direction. Both flexions and extensions of each DOF were typically attempted. A trial was considered failed if the subject did not complete the task within a 30 s time limit. After successful completion of a trial, virtual targets were automatically reset to the resting positions, and the subject was required to maintain all DOFs in their resting positions for 1 s before the next trial was presented. See Figure 4.3 for a depiction of the target-touching task. Three to six trials of each movement were performed in these experiments. A movement (rather than a single trial) was considered “successful” if the subject successfully completed twothirds of the total trials for a given movement. Metrics of performance for online target-touching task were calculated including 1) number of successful trials, 2) trial duration (the time from the start of a trial until successful completion), 3) path length (sum of all DOF output for the duration of the trial), 4) time to the first target touch, and 5) number of target pass-throughs. Timing, pass-through, and path-length metrics were computed for successful trials only. These metrics were used for statistical comparison among data sets using the Kruskal-Wallis rank-sum method, ANOVA, or paired t-tests as appropriate. Medians and interquartile ranges (IQR) for trials sets were computed and displayed in tables for global comparisons. Offline analysis of decoded data was also performed. Here, 4 trials in the first half of the training set were used for training the decode algorithm, and 4 trials from the last half of the training set were used for testing the decode performance. Performance metrics were computed for each DOF to assess the offline decode including root mean square error (RMSE) between the output and movement cue, and, the correlation between the 164 output and movement cue. 4.5 Results 4.5.1 Acute studies Both transradial and intact subjects were able to achieve 9-10 independent DOF using 32 surface electrodes. Results of best target testing sessions are summarized in Table 4.2, which includes best performances metrics during online decode targettouching tasks for intact and amputee subjects. Metrics include total DOFs achieved/ DOFs trained on, total movements achieved/movements trained on, total successful trials/trials performed, percentage success, median trial time of successful trials (IQR), median time to first target touch (IQR), median number of target pass-throughs (IQR), and median path length (IQR). Below timing metrics include a list of missed movements (name of the movement followed by total missed/total attempted). All three intact subjects were able to achieve 10 independent DOFs and 19-20 independent movements, including movements performed primarily with intrinsic hand muscles (such as thumb adduction, index/pinky abduction and adduction), even though no electrodes were placed over intrinsic hand muscles. However, such intrinsic motions were the most commonly missed movements. Figure 4.4 shows the online decode output for a 10-DOF target-touching trial for Intact_S3 where 100% of targets were successfully acquired with minimal crosstalk between most DOFs. The subjects used either “sticky” or dry electrodes and achieved similar levels of performance regardless of electrode type. Additionally, Intact_S1 was able to achieve 6 DOF using only 14 dry electrodes (both “snap” and “button” configurations). See supplemental videos for examples of online testing. 165 Transradial subjects were able to achieve up to 9-10 independent DOFs and 16-20 movements. Figure 4.5 shows the online decode output during a 10-DOF targettouching task for Transradial_S1 where 95% of targets (57/60) were successfully achieved with some crosstalk between DOFs, notably DOF 6 (thumb ab/adduction), and DOF 7/9 (index/pinky ab/adduction). All transradial subjects were able to perform hand intrinsic-controlled movements for some of the trials, although index/pinky abduction was the most commonly missed. However, transradial subjects were able to achieve index/pinky adduction and thumb adduction 100% of the time. 4.5.2 Stability studies Intact and transradial subjects both achieved ³ 1 h stability of decode calibration without significant decline in performance (rank sum p> 0.08 for all target metrics, except path length for Transradial_S3 where rank sum p = 0.02) . Table 4.3 summarizes results from Intact_S1-S3 and Transradial_S1 target-touching-tasks performed ³ 1 h after initial decode training. In two cases (Intact_S1 and Intact_S2), electrodes were removed entirely (doffed) and replaced (donned) in approximately the same position ³ 1 d later. In these cases, performance degraded when calibration was ³ 1 d (significantly for Intact_S2, rank sum p < 0.02 for all target metrics). Intact_S1 showed 6 DOF stability for 5 d (rank sum p > 0.33 for all target metrics). Performance metrics showed a nonsignificant decline between day 0 and day 4 testing. Intact_S3 showed improvement in performance at the 1 h time point (nonsignificant rank sum p>0.1). Transradial_S1 showed a decline in performance for “success” statistics (e.g., percentage of successful trials) but an improvement of other metrics (nonsignificant, rank sum p > 0.23 except for median path length which was significant with rank sum p = 0.02). This subject did note substantial 166 between day 0 and day 4 testing. Intact_S3 showed improvement in performance at the 1 h time point (nonsignificant rank sum p>0.1). Transradial_S1 showed a decline in performance for “success” statistics (e.g., percentage of successful trials) but an improvement of other metrics (nonsignificant, rank sum p > 0.23 except for median path length which was significant with rank sum p = 0.02). This subject did note substantial fatigue after performing other tasks involving forearm use between initial training and the next 1 h time point. 4.5.3 Realtime comparison sEMG and iEMG Realtime comparison of simultaneously collected iEMG and sEMG in Transradial_S3 revealed similar decode performance between the two modalities in terms of the number of independent DOFs and movements achieved. Table 4.4 summarizes results from online target-touching sessions using sequential iEMG or sEMG decodes. In target-touching tasks, Transradial_S4 was able to achieve 10 independent DOFs and ³ 19 movements for both modalities. In online timing and path length metrics Transradial_S4 performed better overall using sEMG compared with using iEMG (significantly for the path length metric, rank sum p =0.02, nonsignificantly for other metrics, rank sum p >0.23). 4.5.4 Post-hoc comparison of sEMG with iEMG Post-hoc analysis of sEMG and iEMG training data revealed similar performance in correlation between decode output and training movement cue and in RMSE. In session 1, iEMG and sEMG had similar correlations (means 0.827 and 0.832, respectively), but sEMG had higher RMSE (0.039 and 0.049, respectively). Figure 4.6 and Figure 4.7 show culminated decode output for each type of movement during training cues. These figures 167 show similar performance for both modalities. 4.5.5 “Button” electrodes work as well as commercial “snap” electrodes To verify the feasibility of using “button” electrodes in place of the commerciallyused “snap” electrodes, Intact_S1 performed two sessions of 8-DOF training using either 14 “snap” or “button” electrodes, placed in the same positions, through the same holes, on the neoprene sleeve. The neoprene sleeve was placed in approximately the same position on the arm for each session. Figure 4.9 shows a boxplot of the median amplitudes during an 8-DOF training session for all 7 forced differential pairs (from 14 electrodes). Amplitudes collected with button electrodes did not differ significantly from those collected using snap electrodes (see Figure 4.9, paired t-test p > 0.2). Figure 4.10 shows boxplots of correlation and RMSE metrics for all DOFs for the two sessions. Offline metrics showed a slight but nonsignificant advantage of “snap” electrodes compared to “button” electrodes (p >0.22 paired t-test for both RMSE and correlation metrics). Because of the similar performance in online tasks and offline analysis, we concluded that “button” electrodes were a suitable substitution for “snap” electrodes for use in our prototyping and experimentation. 4.5.6 Unconstrained software-differenced pairs of monopolar sEMG signal improves offline performance compared to traditional forced-bipolar-pairs To test the hypothesis that unconstrained software-differenced pairs of monopolar sEMG signals performed as well or better than the field-standard forced differential pairs, six 10-DOF training sets were analyzed offline (3 training sets from intact subjects and 3 168 from transradial subjects). Figure 4.8 shows boxplots of offline correlation and RMSE metrics for all DOFs. Comparisons were made between 5 groups: Forced Pairs (FP), forced pairs and single ended (FP_and_SE), all pairs (AP), all pairs and single ended (AP_and_SE), and single ended alone (SE). In general, forced pairs performed worse than all other groups in both RMSE and correlation metric. Force pairs had significantly higher RMSE than all other groups (rank sum p-values <0.001). AP_and_SE had significantly less RMSE compared to SE alone (rank sum pvalue = 0.03). All other rank-sum comparison showed no significant difference (pvalues were > 0.41). For the correlation metric, FP significantly differed from FP_and_SE and AP (rank sum p-value < 0.03) but there were no other significant between-group differences (all other rank sum p-values > 0.19). 4.5.7 Addition of thresholds to decode output improves individual DOF independence and reduces crosstalk Offline analysis of seven 10-DOF training sets (4 from intact subjects and 3 from transradial subjects) with and without a threshold (threshold = 0.2) applied to the decode output revealed that using a threshold can greatly improve individual DOF independence and reduce crosstalk. Figure 4.12 shows the boxplot for RMSE of nonintended DOFS during training (crosstalk) is significantly reduced (t-test p < 0.001) when a threshold is applied. Figure 4.13 shows an example offline decode output for Intact_S3 with and without thresholds applied. The nonintended DOFs’ jitter around baseline (crosstalk) is greatly reduced with the application of threshold. 169 4.6 Discussion These studies show that, using our methodologies, sEMG can be used to decode at least 10 independent and simultaneous DOFs, and 20 independent movements. This is more independent DOFs than have been previously reported from direct control strategies. Independent control of DOFs allows the user to construct a large number of intuitive grasp patterns without having trained on them. Intuitive and independent movement of individual digits may be important for prosthetic embodiment, which may reduce physical and mental operational effort, fatigue, and phantom pain [23]. We showed that it is possible to achieve these high-DOF decodes using dry sEMG electrodes. This is important because dry metal electrodes can be integrated into a socket liner or sleeve, are more comfortable for long term use, and are easier to don/doff in high numbers (compared to “sticky” electrodes). Additionally a conductive gel is not needed in these dry electrode configurations. Transradial amputee subjects were able to perform similarly to intact subjects in terms of the total number of DOFs and movements achieved, and timing metrics of target-touching tests. Of interest, transradial amputees could perform movements even though there was no physical connection to a hand as in the intact subjects. Thus patterns of movements decoded by our methods are not reliant on stretch reflexes resulting from physical tethering of the muscle, tendons, and ligaments to a physical hand, which may influence the patterns in intact subjects. Subjects in this study were able to perform high-DOF decodes regardless of number of electrodes used and in variable placements around the forearm. This shows that our methodology is configurable to any given number of electrodes, limited only by hardware or residual limb surface area constraints. The placements of the electrodes do not 170 need to be precise; rather, they need only to span the surface area of most residual muscle groups. Because of this ability to generalize to various electrode number, type, and positional configurations, our methodology could be used to decode sEMG or iEMG signals from TMR patients who do not have a residual forearm. 4.6.1 Intrinsic hand motions Amputees and intact subjects were capable of performing decodes of movements controlled primarily by intrinsic muscles of the hand, specifically thumb abduction and index/pinky ab/adduction, even though we were not directly recording any signals from these muscles per se. Our algorithm appears to infer intended movements from the constellation of activity recorded from other muscles residing only in the forearm. TwoDOF control of the thumb (flex/extend + ab/adduction) is particularly useful for many grasps used in ADLs such as “key grasp” and “power grasp.” Index and/or pinky adduction is essential for dexterous manipulation of tools such as a writing instrument. Although such intrinsic movements were occasionally more difficult than other DOFs, important intrinsic patterns (thumb and index adduction) could be decoded using sEMG signals from the residual forearm alone. 4.6.2 Advantages and disadvantages of sEMG compared with iEMG We also show that a subject using sEMG is capable of performing better than one using iEMG of greater than or equal to channel count on multiple metrics in acute settings. This indicates that sEMG is a suitable alternative to iEMG. However, these studies were performed using only one subject and may not generalize to a larger population. Until chronic iEMG is a more common technology, it will be difficult to find a large enough 171 population to adequately compare the two modalities. Additionally, it is possible that placement of the iEMG leads in the subject studied here was not optimal or was constrained by device limitation such as lead length and lead tethering. The iEMG array has 8 leads, and due to its tethered nature, not all desired muscles could be reached surgically (e.g., each individual digit flexor or extensor bundle could not be targeted separately with only 8 available leads). Finally, the stability of the sEMG decode calibration for this subject was not studied for more than 1 h which is not sufficient to compare to the iEMG decode stability, which was studied for several weeks (see Chapter 3). There are several advantages to the sEMG approach for the amputee patient, mainly, its noninvasive and more economical nature compared with iEMG. Using SEMGs circumvents the need for a potentially costly surgical procedure which may have adverse health effects such as infection, pain, and stump volume changes requiring new sockets to be made. Additionally, sEMG electrodes are more easily repaired or replaced if they happen to fail. Finally, there are few physical restraints to placing the electrodes on the residual forearm. This allows the prosthetist to target all available muscle bellies for optimized signal collection. The main disadvantages of the sEMG approach compared to the iEMG approach include the need for precise repeated electrode placement when donning electrodes (if recalibration is to be avoided), and sEMG sensitivity to forearm pronation/ supination. The sEMG decode calibration is sensitive to electrode position relative to the muscle signal source. If relative electrode position is changed between doffing and donning, the signal strength and patterns from the original muscle sources will be different from those expected from previously obtained decode algorithm coefficients. 172 In this scenario, the original algorithm coefficients and new signal patterns will no longer produce an accurate decode. It was anecdotally noted by the intact subjects that it was difficult to perform some movements when their arm was not in the “right” position, where the “right” position referred to the position/orientation of the upper arm/forearm, and wrist during training for that particular movement. These subjects would sometimes slightly adjust the bend of their elbow or rotation of the wrist in order to achieve success on certain targets during the target-touching tasks. One subject noted that when her wrist was supinated, some of the decoded movements were “shifted” (i.e., digit 1 flexion resulted in the digit 2 flexion decode output, digit 2 flexion resulted in digit 3 flexion decode output, and digit 3 flexion resulted in digit 4 decode output, while digit 4 and 5 flexion were largely unaffected). This sensitivity to wrist rotation (pronation/supination) can be attributed to the relative shift in position of surface electrode relative to the underlying muscle (see Figure 4.11 for an illustration of electrode position shift due to supination). This sensitivity may vary with forearm length (i.e., the longer the forearm, the greater the relative shift at distal electrode sites). However, the effect on decode performance during supination was not studied here. Other disadvantages of the sEMG (not addressed in this study) include a sensitivity to skin and adipose tissue thickness, loose skin, perspiration, and pressure on the electrodes (normal to the skin)[24]–[27]. These disadvantages are largely overcome by the use of iEMG which is not affected by perspiration, loose skin, or adipose thickness as leads are inserted and anchored directly into the muscles. In our experiments, we also found that sEMG is sensitive to electrical interference caused by wire movement caused by incidental bumping or shifting of the wire bundle 173 (data not shown). This could cause unintentional sporadic movements of the prosthetic. Future prototypes of surface sleeve and socket designs will include better electrical shielding, and motion artifact rejection. Although implanted EMG will most likely be the most robust, specific, and stable signal source for motor decoding, many upper-limb amputees may not have the desire or the financial means for surgical implantation of these devices. Ultimately we plan to devise or utilize existing high-electrode-count sleeves (such as the JPL “Biosleeve” [28] or the Myo armband [29]) that could be worn inside the prosthetic socket. We envision using small, wearable computers to decode sEMG signals and control the motors of existing advanced prosthetic hands. 4.6.3 Signal processing We show that using software differential pairs has significantly better performance compared with using the standard of hardware differenced pairs, or the “forced” bipolar pair paradigm. We suggest that, when possible, software differential pairs are used for computing EMG features, rather than forced differential pairs differenced at the source by hardware. Additionally, we show that the postdecode addition of thresholds significantly improves performance in offline decodes. Such measures greatly improve independence of DOFs by reducing inter-DOF crosstalk. Applying thresholds does introduce a certain amount of unintended nonlinearity to the decode output. However, this nonlinearity was advantageous in online and offline performance. We recommend the use of configurable gains and thresholds post-decode regardless of the decode strategy used (direct control, independent-DOF, or pattern classification). 174 4.7 Conclusion In these experiments, we have demonstrated the feasibility of using high-channelcount ³ 14) sEMG electrodes to decode 6 or more (up to 10) independent hand and wrist DOFs from three transradial amputees and three intact individuals, regardless of the type of electrode used (“sticky” ECG-style leads or dry stainless steel leads). Our methodology is configurable to any number of electrodes, limited only by hardware constraints or surface area of the residual limb. Furthermore, the methods we present allow stable decodes between doffing and donning electrodes, reducing the need for daily recalibration. Finally, we show that sEMG is feasible alternative to iEMG for decoding motor signals from the residual forearm. 4.8 Acknowledgements This work was sponsored by the Defense Advanced Research Projects Agency (DARPA) Biological Technologies Office (BTO) Hand Proprioception and Touch Interfaces (HAPTIX) program under the auspices of Dr. Doug Weber through the Space and Naval Warfare Systems Center, Contract No. N66001-15-C-4017, No. N66001-12C-4042. Additional funding was also provided via the National Institutes of Health (NIH NCATS Award No. 1ULTR001067). Additionally, other team members at the University of Utah, including Tyler Davis, David Page, David Kluger, Jacob Nieveen, Jacob George, Nathan Olsen, Chistopher Duncan, Douglas Hutchinson, and Gregory Clark contributed significantly to this work either in data collection, experiment design, medical procedures, and/or data analysis. 175 Figure 4.1 Electrode configurations. a) neoprene sleeve studded with 32 “button” electrodes worn by Transradial_S1; b) neoprene sleeve studded with 14 “snap” electrodes; c) flexor aspect, 32 “sticky” electrode configuration for Intact_S2. d) extensor aspect, 32 wet “sticky” electrode configuration for Intact_S2. e) close-up of “sticky” (left), “snap” (middle), and “button” (right) electrodes used in this study. 176 Figure 4.2 Schematic diagram of EMG data collection and decoding signal processing steps. EMG signals are preconditioned with bandpass and notch filters. Differential pairs are computed in software. Amplitude is computed by rectifying single-ended and differential pair signals. This feature is smoothed with a 300 ms boxcar filter. For decode calibration, selected features from the training period are used to coefficients. For online decoding, the selected features are passed through the decoder, and user selected gains, thresholds, and integration modes. Finally the scaled output for each DOF is converted into commands to control a virtual or robotic hand. 177 Figure 4.3 Example depiction of target-touching task. 1) The subject must keep all DOFs in the neutral “home” position for 1 s. 2) One or more spherical virtual target(s) is positioned away from the resting position of one or more digit(s)/wrist along the arc of movement. 3) A successful trial results when the subject moves the specified digit(s) inside the radius of the spherical target(s) for at least 300 ms while keeping the other DOFs in resting position. The virtual targets do not exclude the virtual fingers, and fingers could pass all the way through the spheres. To provide visual feedback, the target spheres change color from red to green when the desired finger enters the target sphere. The target radii are typically set to be 15% of the arc of motion in one direction. The targets are placed at 50% of the arc of motion in either the flexion or extension direction. Both flexions and extensions relative to the neutral position of each DOF were typically attempted. A trial is considered failed if the subject did not complete the task within a 30-s time limit. 4) After successful completion of a trial, virtual targets are automatically reset to the neutral positions. 178 Table 4.1 List and descriptions of DOFs. DOF Description 1 Thumb Flex/Extend 2 Index Flex/Extend 3 Middle Flex/Extend 4 Ring Flex/Extend 5 Little Flex/Extend 6 Thumb Abduction/Adduction 7 Index Abduction/Adduction 8 Ring Abduction/Adduction 9 Little Abduction/Adduction 10 Wrist Flex/Extend 11 Wrist Ulnar/Radial Deviation 12 Wrist Pronate/Supinate 179 Table 4.2 Best performances during online decodes for target-touching tasks for a) intact and b) amputee subjects. Metrics include total DOFs achieved/DOFs trained on, total movements achieved/movements trained on, total successful trials/trials performed, percentage success, median trial time of successful trials (IQR), median time to first target touch (IQR), median number of target pass-throughs (IQR), and median path length (IQR). A list of missed movements is included as the last row (number missed/total attempted). a) Subject Intact_S1 Calibration Age 0h Number of Electrodes 14 Electrode Type snap DOFs Achieved 6/6 Movements Achieved 10/10 Total trials 30 Successful trials 30 Percent Success (%) 100 Median Trial Time (IQR) s 2.54 (0.76) Median Time to 1st 2.26 (0.76) Target Touch (IQR) s Median Number of Pass0 (0) throughs (std) s Median Path Length (IQR) 28.73 (7.63) Missed Movements none Intact_S1 0h 32 button 10 /10 19 /20 60 57 95 2.01 (1.53) Intact_S2 1h 32 sticky 10/10 20/20 120 119 99 1.95 (1.33) Intact_S3 3h 32 sticky 10/10 20/20 120 120 100 3.13 (2.90) 1.58 (0.84) 1.48 (0.94) 2.15 (1.78) 0 (0.0) 8.96 (14.14) 'IndexPinkyAB' (3/3) 0 (0.0) 8.28 (9.39) 'IndexPinkyAd'(1/6) 0 (1.0) 17.35 (29.41) none 180 Table 4.2 continued. b) Subject TransRadial_S1 TransRadial_S2 Transradial _S3 Transradial _S3 Calibration Age < 1h 0h 0h <1h Number of Electrodes 32 32 22 32 Electrode Type button sticky sticky sticky DOFs Achieved 10/10 9/10 8/8 10/10 Movements Achieved 19/20 16/20 15/16 20/20 Total trials 60 60 48 60 Successful trials 57 46 45 59 Percent Success (%) 95 77 94 98 Median Trial Time (IQR) s 2.57 (3.59) Median Time to 1st Target Touch (IQR) s 2.01 (3.12) Median Number of Passthroughs (std) s 0 (0.0) 4.78 (11.58) 3.14 (3.41) 1.65 (2.55) 1.7 (1.85) 1.98 (1.67) 1.32 (1.38) 1 (2.0) 1 (1.25) 0 (0.0) Median Path Length (IQR) 14.03 (37.10) Missed Movements 'MiddleE' (3/3) 48.65 (162.21) 'ThumbF' 'ThumbE' 'RingE' 'WristF' 'WristE' 'WristRadialDev' 'IndexPinkyAB' 'IndexPinkyAd' (2/3) (1/3) (1/3) (1/3) (3/3) (1/3) (2/3) (3/3) 24.46 (43.72) 'ThumbE' 'MiddleE' 8.23 (25.33) (1/3) IndexPinkyAB' (1/3) (2/3) 181 Table 4.3 Decode stability for each subject. Tables 4.3(1), 4.3(2), 4.3(3), and 4.3(4) include target-touching task results from intact and transradial subjects at various time points after intitial decode calibration (> 1 h, up to 5 d). For each subject, the same calibration was used at each time point. Metrics include total DOFs achieved/DOFs trained on, total movements achieved/movements trained on, total successful trials/trials performed, percentage success, median trial time of successful trials (IQR), median time to first target touch (IQR), median number of target pass-throughs (IQR), and median path length (IQR). Below timing metrics is a list of missed movements. Wilcoxon Rank Sum p-values are presented in the far columns for the timing metrics. Significant values (p< 0.05) are bolded. Columns with grey background indicate that electrodes have been doffed and donned again. (1) Subject: Intact_S1 Online Stability Tests a 0h 14 snap 6/6 10/10 30 30 100 2.54 (0.76) Calibration Age Number of Electrodes Electrode Type DOFs Achieved Movements Achieved Total trials Successful trials (n) Percent Success (%) Median Trial Time (IQR) s Median Time to 1st Target Touch (IQR) s 2.26 (0.76) Median Number of Pass-throughs (IQR) s 0 (0) Median Path Length (IQR) 28.73 (7.63) Missed Movements none b 5d 14 snap 6/6 10/10 30 28 93 2.36 (3.47) p value (a, b) 2.06 (3.35) 0.60 0.75 0 (0) 0.33 26.10 (92.75) 0.36 'ThumbF' (1/3) 'WristF' (1/3) 182 Table 4.3 continued (2) Subject: Intact_S2 Online Stability Tests a 0h 32 sticky 10/10 19/20 120 117 98 1.95 (1.10) Calibration Age Number of Electrodes Electrode Type DOFs Achieved Movements Achieved Total trials Successful trials (n) Percent Success (%) Median Trial Time (IQR) s Median Time to 1st Target Touch (IQR) s 1.62 (0.91) Median Number of Passthroughs (IQR) s 0 (0) Median Path Length (IQR) Missed Movements b 1h 32 sticky 10/10 20/20 120 119 99 1.95 (1.33) c 1d 32 sticky 10/10 19/20 60 56 93 2.71 (2.95) 1.48 (0.94) 2.10 (2.57) 0 (0.0) 0 (0.5) 17.51 (54.18) 8.15 (7.62) 8.28 (9.39) 'ThumbABd' 'IndexPinkyAd'(1/6) 'MiddleF' (3/6) (1/6) 'ThumbE' (1/6) 'IndexE' (2/6) p value p value (a,b) (a,c) 0.28 3.5E-05 0.08 3.2E-03 0.26 2.6E-02 0.80 1.2E-05 183 Table 4.3 continued (3) Subject: Intact_S3 Online Stability Tests a Calibration Age 0h Number of Electrodes 32 Electrode Type sticky DOFs Achieved 10 Movements Achieved 20 Total trials 120 Successful trials (n) 120 Percent Success (%) 100 Median Trial Time (IQR) s 3.61 (3.93) Median Time to 1st Target Touch (IQR) s 2.19 (1.62) Median Number of Pass-throughs (IQR) s 0 (2) Median Path Length (IQR) 17.10 (36.0) Missed Movements none b 3h 32 sticky 10/10 20/20 120 120 100 3.13 (2.90) p value (a, b) 2.15 (1.78) 0.24 0 (1.0) 17.35 (29.41) none 0.15 0.67 0.10 184 Table 4.3 continued (4) Subject: TransRadial_S1 Online Stability Tests a 0h 32 button 10/10 19/20 60 57 95 2.57 (3.59) Calibration Age Number of Electrodes Electrode Type DOFs Achieved Movements Achieved Total trials Successful trials (n) Percent Success (%) Median Trial Time (IQR) s Median Time to 1st Target Touch (IQR) s 2.01 (3.12) Median Number of Pass-throughs (IQR) s 0 (0.0) Median Path Length (IQR) 14.03 (37.10) Missed Movements 'MiddleE' (3/3) b 1h 32 button 10/10 16/20 60 47 78 2.21 (4.21) p value (a, b) 1.55 (1.95) 0.25 0.23 0 (1) 0.42 10.15 (84.2) 0.02 'IndexE' (3/3) 'MiddleE'(3/3) 'LittleE' (1/3) 'ThumbADd' (1/3) 'WristE' ((3/3) 'IndexPinkyAB' (2/3) 'IndexPinkyAB'(1/3) 185 Figure 4.4 Example of a 10-DOF online decode target-touching task for Intact_S3. The black lines and blue lines show the target position and decode output for each DOF, respectively. Here, 100% (60/60) targets were successfully acquired with minimal crosstalk between DOFs. 186 Figure 4.5 Example of a 10-DOF online decode target-touching task for Transradial_S1. The black lines and blue lines show the target position and the decode output for each DOF, respectively. Here 95% (57/60) of targets were successfully acquired, with some crosstalk between DOFs, notably between DOFs 6 (thumb ab/adduction) and others, e.g., 7/9 (index/pinky ab/adduction). 187 Table 4.4 iEMG vs. sEMG online results. Table shows results from two sessions of targettouching trials for Transradial_S3. Overall scores between iEMG and sEMG trials were very similar. There was no overall significant difference between timing metrics (only ¼ timing metrics showed significant difference for each session). Metrics include total DOFs achieved/DOFs trained on, total movements achieved/movements trained on, total successful trials/trials performed, percentage success, median trial time of successful trials (IQR), median time to first target touch (IQR), median number of target pass-throughs (IQR), and median path length (IQR). Below timing metrics is a list of missed movements. Below timing metrics is a list of missed movements. Wilcoxon Rank Sum p-values are presented to in the far columns for the timing metrics. Significant values (p< 0.05) are bolded. iEMG v sEMG Subject: Transradial_S3 Session 1 a b Number of Electrodes 32 22 32 32 Electrode Type iEMG Sticky sEMG iEMG "sticky" sEMG DOFs Achieved 8/8 8/8 10/10 10/10 Movements Achieved 8/8 8/8 19/20 18/20 Total trials 24 24 120 120 Successful trials (n) 24 24 113 111 Percent Success (%) 100 100 94.2 92.5 3.28 (3.55) 1.88 (1.90) 0.03 2.31 (2.76) 1.75 (1.44) 0.07 1.48 (0.63) 0.14 1.58 (1.64) 1.35 (1.08) 0.06 0 (1) 0.13 0 (0.25) 0 (0) 0.19 8.71 (28.23) 0.07 10.6 (39.9) 8.38 (19.56) 0.04 none 'IndexPinkyAd' 'WristSupinate' (2/6) (6/6) 'ThumbADd' 'IndexPinkyAB' (1/6) (3/6) 'IndexPinkyAB' (3/6) 'ThumbF ' (1/6) Median Trial Time (IQR) s Median Time to 1st Target Touch (IQR) 1.77 (1.67) s Median Number of Pass-throughs (IQR) 1 (2) s 39.87 (53.73) Median Path Length (IQR) Missed Movements none Session 2 p value (a,b) a b p value (a,b) 188 Figure 4.6 Top row: Offline analysis of simultaneously collected iEMG (red) and sEMG (green) during flexion movement cues (blue). The bold middle line represents the mean, and the shaded area bounds represent the standard error. Bottom row shows the crosstalk (sum of the noncued DOFs output) for simultaneously collected sEMG (magenta), and iEMG (turquoise). These results show similar offline decode performance between sEMG and iEMG data. 189 Figure 4.7 Top row: Offline analysis of simultaneously collected iEMG (red) and sEMG (green) during extension movement cues (blue). The bold middle line represents the mean, and the shaded area bounds represent the standard error. Bottom row shows the crosstalk (sum of the noncued DOFs output) for simultaneously collected sEMG (magenta), and iEMG (turquoises). Note that DOF 4 and 5 (ring finger and pinky finger extension) were not trained on in session 1. These results show similar offline decode performance between sEMG and iEMG data. 190 Figure 4.8 Boxplots of offline decode performance metrics of 6 training sets (RMSE, top, and Correlation, bottom) for offline decodes computed using forced-differential pairs (“FP,” first column), forced pairs and single-ended channels (“FP _and_SE”second column), all differential pairs (“AP,” third column), all differential pairs and single-ended channels (“AP_and_SE,” fourth column), and single ended pairs only (“SE,” fifth column). Force pairs differed significantly from all other groups (Rank sum p-values <0.001 for FP compared to all other groups for RMSE). For RMSE, AP vs. SE p-value = 0.053, and AP_and_SE vs. SE p-value = 0.03 (also significant). All other rank-sum comparison pvalues were > 0.41. For the correlation metric, FP significantly differed from FP_and_SE and AP (rank sum p-value < 0.03) but there was no other significant between-group differences (all other rank sum p-values > 0.19). 191 Figure 4.9 Boxplot of median EMG amplitude (32 features) during training session using snap or button electrodes in the same positions on neoprene sleeve for Intact_S1. Paired ttest did not reveal statistically significant difference (p > 0.24). 192 Figure 4.10 Boxplots of offline metrics commanded movement correlation and RMSE for “snap” electrodes vs. “button” electrodes for Intact_S1 8-DOF training Data. Paired t-test did not reveal statistically significant difference (p > 0.22). 193 Figure 4.11 Surface electrode position shifts relative to underlying muscle during wrist rotation. a) the line of electrodes marked with a green dashed line lies over regions of the brachioradialis, flexor pollicis longus, and abductor pollicis longus when the wrist is in neutral position. b) the same line of electrodes lies over the region of the flexor digitorum superficialis bundle when wrist is supinated. 194 Figure 4.12 Boxplot of RMSE during nonmovement time period (“X-talk RMSE”) for 7 10-DOF training sets (4 from intact subjects, and 3 from transradial subjects). RMSE is significantly less (paired t-test p < 0.001) and independence is greatly improved when a low threshold value of 0.2 is applied to decode outputs. 195 Figure 4.13 Example offline decode output from a 10-DOF training set from Intact_S3, with threshold (red trace) and without threshold (dashed-blue trace) applied (black trace represents movement cue). Using a small threshold of 0.2 can greatly improve individual DOF independence and reduce crosstalk. Note the large amount of crosstalk in nonintended DOFs during training movements if no threshold is applied. 196 4.9 List of abbreviations ANOVA: Analysis of Variance DOF: Degree of Freedom EMG: Electromyogram iEMG: Intramuscular EMG KF: Kalman Filter RMSE: Root mean square error sEMG: Surface EMG USEA: Utah Slanted Electrode Array 4.10 References [1] P. M. Rossini et al., “Double nerve intraneural interface implant on a human amputee for robotic hand control,” Clin. Neurophysiol., vol. 121, no. 5, pp. 777– 783, May 2010. [2] P. Zhou et al., “Decoding a new neural–machine interface for control of artificial limbs,” J. Neurophysiol., vol. 98, no. 5, pp. 2974–2982, Nov. 2007. [3] L. R. Hochberg et al., “Reach and grasp by people with tetraplegia using a neurally controlled robotic arm,” Nature, vol. 485, no. 7398, pp. 372–375, May 2012. [4] L. H. Smith, T. A. Kuiken, and L. J. Hargrove, “Evaluation of linear regression simultaneous myoelectric control using intramuscular EMG,” IEEE Trans. Biomed. Eng., vol. 63, no. 4, pp. 737–746, Apr. 2016. [5] J. J. Baker et al., “Decoding individuated finger flexions with Implantable MyoElectric Sensors,” Conf. Proc. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. IEEE Eng. Med. Biol. Soc. Conf., vol. 2008, pp. 193–196, 2008. [6] C. Cipriani, J. L. Segil, J. A. Birdwell, and R. F. Weir, “Dexterous control of a prosthetic hand using fine-wire intramuscular electrodes in targeted extrinsic muscles,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 99, pp. 1–1, 2014. [7] E. N. Kamavuako, E. J. Scheme, and K. B. Englehart, “On the usability of intramuscular EMG for prosthetic control: A Fitts’ Law approach,” J. Electromyogr. 197 Kinesiol., vol. 24, no. 5, pp. 770–777, Oct. 2014. [8] G. Purushothaman and K. K. Ray, “EMG based man–machine interaction—A pattern recognition research platform,” Robot. Auton. Syst., vol. 62, no. 6, pp. 864– 870, Jun. 2014. [9] J.-U. Chu, I. Moon, and M.-S. Mun, “A realtime EMG pattern recognition system based on linear-nonlinear feature projection for a multifunction myoelectric hand,” IEEE Trans. Biomed. Eng., vol. 53, no. 11, pp. 2232–2239, Nov. 2006. [10] L. Hargrove, Y. Losier, B. Lock, K. Englehart, and B. Hudgins, “A realtime pattern recognition based myoelectric control usability study implemented in a virtual environment,” in Engineering in Medicine and Biology Society, 2007. EMBS 2007. 29th Annual International Conference of the IEEE, 2007, pp. 4842–4845. [11] L. J. Hargrove, K. Englehart, and B. Hudgins, “A comparison of surface and intramuscular myoelectric signal classification,” IEEE Trans. Biomed. Eng., vol. 54, no. 5, pp. 847–853, May 2007. [12] W. Wu, M. J. Black, D. Mumford, Y. Gao, E. Bienenstock, and J. P. Donoghue, “Modeling and decoding motor cortical activity using a switching Kalman filter,” IEEE Trans. Biomed. Eng., vol. 51, no. 6, pp. 933–942, Jun. 2004. [13] W. Wu, Y. Gao, E. Bienenstock, J. P. Donoghue, and M. J. Black, “Bayesian population decoding of motor cortical activity using a Kalman filter,” Neural Comput., vol. 18, no. 1, pp. 80–118, 2006. [14] J. Egan, J. Baker, P. A. House, and B. Greger, “Decoding dexterous finger movements in a neural prosthesis model approaching real-world conditions,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 20, no. 6, pp. 836–844, Nov. 2012. [15] P. Morel et al., “Long-term decoding of movement force and direction with a wireless myoelectric implant,” J. Neural Eng., vol. 13, no. 1, p. 16002, 2016. [16] E. N. Kamavuako, E. J. Scheme, and K. B. Englehart, “Combined surface and intramuscular EMG for improved realtime myoelectric control performance,” Biomed. Signal Process. Control, vol. 10, pp. 102–107, Mar. 2014. [17] M. Hauschild, R. Davoodi, and G. E. Loeb, “A virtual reality environment for designing and fitting neural prosthetic limbs,” IEEE Trans. Neural Syst. Rehabil. Eng., vol. 15, no. 1, pp. 9–15, Mar. 2007. [18] J. G. Nieveen et al., “Channel Selection of Neural and electromyographic signals for decoding of motor intent,” presented at the Myoelectric Control (MEC) Symposium 2017, Fredericton, New Brunswick, CA, 2017. 198 [19] D. J. Warren et al., “Recording and decoding for neural prostheses,” Proc. IEEE, vol. 104, no. 2, pp. 374–391, Feb. 2016. [20] W. Wu, A. Shaikhouni, J. P. Donoghue, and M. J. Black, “Closed-loop neural control of cursor motion using a Kalman filter,” in 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2004. IEMBS ’04, 2004, vol. 2, pp. 4126–4129. [21] W. Q. Malik, W. Truccolo, E. N. Brown, and L. R. Hochberg, “Efficient decoding with steady-state Kalman filter in neural interface systems,” IEEE Trans. Neural Syst. Rehabil. Eng. Publ. IEEE Eng. Med. Biol. Soc., vol. 19, no. 1, pp. 25–34, Feb. 2011. [22] G. Welch and G. Bishop, , “An introduction to the Kalman filter,” [Online] Course material from University of North Carolina-Chapel Hill, July 24, 2006. [23] D. M. Page, “Restored hand sensation in human amputees via utah slanted electrode array stimulation enables performance of functional tasks and meaningful prosthesis embodiment,” Ph.D. Thesis, University of Utah, 2016. [24] C. Nordander et al., “Influence of the subcutaneous fat layer, as measured by ultrasound, skinfold calipers and BMI, on the EMG amplitude,” Eur. J. Appl. Physiol., vol. 89, no. 6, pp. 514–519, 2003. [25] D. Stewart, A. Macaluso, and G. De Vito, “The effect of an active warm-up on surface EMG and muscle performance in healthy humans,” Eur. J. Appl. Physiol., vol. 89, no. 6, pp. 509–513, 2003. [26] G. J. Lehman and S. M. McGill, “The importance of normalization in the interpretation of surface electromyography: a proof of principle,” J. Manipulative Physiol. Ther., vol. 22, no. 7, pp. 444–446, 1999. [27] P. Zipp, “Recommendations for the standardization of lead positions in surface electromyography,” Eur. J. Appl. Physiol., vol. 50, no. 1, pp. 41–54, 1982. [28] E. Ackerman, “JPL BioSleeve Enables Precise Robot Control Through Hand and Arm Gestures,” IEEE Spectrum: Technology, Engineering, and Science News, 16May-2013. [Online]. Available: http://spectrum.ieee.org/automaton/robotics/robotics-hardware/jpl-biosleeveenables-precise-robot-control-through-hand-and-arm-gestures. [Accessed: 07-Dec2016]. [29] “Myo Gesture Control Armband.” [Online]. Available: https://www.myo.com/. [Accessed: 02-May-2017]. CHAPTER 5 FUTURE DIRECTIONS 5.1 Impacts on the field of neuromuscular decoding interfaces The work presented herein has the potential to lead the field of neuromuscular decoding in new directions. The most impactful result of this work is the demonstration and implementation of a highly configurable interface capable of providing robust, intuitive, high-DOF motor decodes with minimal training. Such an interface has the potential to be applied in a number of ways described below. 5.2 Future sEMG studies and prototypes Ultimately, it is my hope that the decoding techniques described in this work will be used for decoding motor intent in a number of applications, including, but not limited to, the operation of multiarticulate robotic hands or other devices where simultaneous and independent multi-DOF control is desired. For instance, the decode outputs could be used as commands for a computer interface, game controller, wheelchair, or other motorized adaptive device (Figure 5.1). In the context of neuromuscular interfaces, sEMG provides the less invasive, less medically risky, and less expensive option for decoding motor intent. It is thus accessible to amputees who do not desire or have the financial means for implanted devices. It is easy 200 to foresee a miniature, stand-alone computing system mounted on a myoelectric prosthetic arm which serves as the bioamplifier, data acquisition system, and realtime decoder. 5.2.1 An affordable, high-DOF sEMG prosthetic arm Open source efforts are underway that will allow users to cheaply develop and customize prosthetics using 3D printers and off-the-shelf electronics [1], [2]. Small multichannel EMG amplifier and digitizers currently are available for purchase such as the 24 bit 8-channel Texas Instrument ADS1298 chip [3]. Such chips could be used in parallel to collect a high number of sEMG channels from a prosthetic socket. Additionally, such electronics can be driven by programmable Arduino controllers or Raspberry Pi computers. The decoder training could be performed by following movements of the robotic hand or movements displayed on a computer or smartphone screen. Decode gains, thresholds, and output modes could be adjusted through a smartphone application. Such a home-built prosthetic could conceivably be built for less than $1000 USD (2017). 5.2.2 Study limitations to be addressed in future studies The work presented herein was far from an exhaustive. Many potential benefits have yet to be studied and pitfalls identified and remedied before this technology can be implemented into commercial prosthetics. Several limitations to the sEMG approach were discussed in Chapter 4. These included the sensitivity to electrical noise from motion artifact, the stability of the decode calibration, and the adverse effects of arm posture— specifically, wrist pronation/supination—on decode performance. Additionally, these methods have not been extensively tested under activities of daily living conditions. Known 201 issues that adversely affect existing sEMG decoding techniques, such as sweating, socket tightness, weight bearing, and stump volume changes may adversely affect the decode methods described herein. Further studies and prototypes are needed to address these potential shortcomings. 5.2.3 Possibilities for transhumeral amputees Because above-elbow (transhumeral and above) amputees lack residual forearm muscles in which to implant IEMG, another strategy of acquiring motor intent signal is needed, if intuitive, EMG-based decoding is desired. New advancements in neuromuscular interface methods are currently underway that may help this become a reality [4], [5]. These include 1) targeted muscle reinnervation (TMR), where residual peripheral nerve branches are rerouted to existing muscle (after partial deinnervation) where EMG can be recorded via iMEG or sEMG [4], [6]; 2) regenerative peripheral nerve interfaces, where branches of residual peripheral nerve are placed in autologous partial muscle grafts, which act as a bioamplifier for the peripheral nerve signal [5]. Such studies provide a new method to record robust and specific motor signals from transhumeral and above amputees. Advances in our neuromuscular decode system may be applied in these systems. 5.2.4 Possibilities for congenital amputees Individuals born without all, or portions of, limbs (congenital amputees or individuals with limb deficiency) present a unique challenge for the EMG-powered prosthesis paradigm. Such individuals have never had portions of limbs, or digits, to move and thus lack functional cortical representation of the limb, motor patterns, and muscle 202 strength in the residual limb. In a pilot study, we attempted to decode 3 DOF from a transradial congenital amputee with some success. In this pilot study, 22 sEMG electrodes were distributed on the stump of a unilateral congenital amputee. The subject was female, in her 5th decade, with left arm deficiency from congenital banding syndrome. Her residual forearm was approximately 8 cm in length from the medial epicondyle to the distal end (see Figure 5.2). To train the decoder, the subject was instructed to attempt 6 distinct patterns of muscle flexion in her residual forearm during a 3-DOF, 6-movement decode training routine. The subject was able to informally control all 3 DOFs of a virtual and robotic prosthetic hand (see video “sEMG#1_congenital.mp4”). Such results are promising as they show that it may be possible to decode more than 2 DOFs from EMG signals from the residual stump of congenital transradial amputees (2 DOFs is the current maximum achieved [7]). 5.2.5 Possibilities for spinal cord injuries Because spinal cord injury inherently limits the availability of volitionally controlled muscle signals, controlling prosthetics or other assistive devices, such as motorized wheelchairs, using EMG or other peripheral signals is particularly challenging. The higher in the spinal cord the injury, the less available the motor signal, and yet, the greater the need for motor assistance (the “control conundrum”). Current assistive technologies are limited to “sip-and-puff,” joystick, or eye tracking control strategies. However, these strategies can be slow, nonintuitive, and rely on the precise positioning of the joystick/straw/eye tracking device by a caregiver. Other, more invasive, motor intent decoding options, such as cortical neural implants, are being developed and show promise 203 [8]–[11], but have a limited user base due to invasiveness, risk, complexity, and expense. In a pilot study, we use our interface to allow a quadriplegic individual (C2 level, American Spinal Injury Association (ASIA) grade A, 21 months post-injury) to control 10 independent movements of a virtual hand using sEMG signals from low amplitude shoulder, neck, and bicep movements. Here, signals from 32 sEMG electrodes placed on shoulder girdle, neck, and left bicep muscles (Figure 5.3), of which he had partial volitional control, were used for decoding multiple DOFS in a realtime target-touching task. We used the same modified Kalman filter decoder described in previous chapters. He trained the decoder by performing 10 distinct movements, such as left or right shoulder shrug, head turning/tilting, and right bicep flex, in synchrony with a temporally varying sinusoidal waveform presented on a computer monitor. Without tuning of individual DOF gains or threshold values (gain set to 1, threshold set to 0.2 for all DOFs), and without instruction from the experimenters, he intuitively controlled 10 DOFs of a virtual hand and performed a target-touching task where 3 targets of each DOF were presented (see Figure 5.4 and video “20170501_121801_v170509_SCI.mp4”). He was able to achieve 30 successes in 30 trials. Following the pilot session, the participant succinctly encapsulates his candid impressions of the interface’s potential by stating, I think it’s awesome. I’m just starting to imagine the potential of having, the control I have [with this interface.] The applications are endless. I am more excited about this than, honestly, anything that has happened since I broke my neck. Using a sip and puff to ski was amazing but with these controls I could go back to skiing what I used to ski; like difficult, fine controls. Right now, we have just left turn, right turn. With 10 degrees of freedom I could just imagine anything with better controls. It would be a better wheelchair control than this little joystick in front of me. This is ridiculous, absolutely ridiculous. 204 5.3 Acknowledgements Tyler Davis and Christopher Duncan both contributed to collecting the pilot data and creating videos presented in this chapter. 205 Figure 5.1 Schematic example of the sEMG technology applied in two different ways. sEMG electrodes can amplified and digitized by a small, multichannel A-D board. Next, signals are processed and decoded by a miniature computing system. The outputs can then be sent as commands to a prosthetic hand, game controller, computer mouse, etc. 206 Figure 5.2 sEMG configuration used for decoding 3 DOF from a congenital transradial amputee. Here 22 “sticky” electrodes were distributed on the residual stump. 207 Figure 5.3 Image showing the placement of 32 sEMG electrodes on spinal cord injured volunteer. 208 Figure 5.4 Decode output (blue trace) and target distance (black trace) for a 10-DOF online target-touching-task, performed by a C2 level quadriplegic individual. Here, we used signals from 32 sEMG electrodes placed on shoulder, neck, and back muscles and a modified Kalman filter decoder to decode 10 individual and simultaneous DOFs. In this pilot session, three targets for each DOF were sequentially presented. The subject achieved 30/30 successful trials. 209 5.4 References [1] “Open Bionics,” Open Bionics. [Online]. https://www.openbionics.com/about/. [Accessed: 16-Nov-2016]. [2] “Open source EEG/ECG/EMG | CRNL.” [Online]. Available: http://www.mccauslandcenter.sc.edu/crnl/open-source-eegecgemg. [Accessed: 08Aug-2017]. [3] “ADS1298 24-Bit, 8-Ch, Delta-Sigma ADC for ECG | TI.com.” [Online]. Available: http://www.ti.com/product/ADS1298. [Accessed: 08-Aug-2017]. [4] T. A. Kuiken et al., “Targeted reinnervation for enhanced prosthetic arm function in a woman with a proximal amputation: A case study,” The Lancet, vol. 369, no. 9559, pp. 371–380, Feb. 2007. [5] Z. T. Irwin et al., “Chronic recording of hand prosthesis control signals via a regenerative peripheral nerve interface in a rhesus macaque,” J. Neural Eng., vol. 13, no. 4, pp. 46007–46007, 2016. [6] P. Zhou et al., “Decoding a New Neural–Machine Interface for control of artificial limbs,” J. Neurophysiol., vol. 98, no. 5, pp. 2974–2982, Nov. 2007. [7] H. Rehbaum, Ning Jiang, L. Paredes, S. Amsuess, B. Graimann, and D. Farina, “Real time simultaneous and proportional control of multiple degrees of freedom from surface EMG: Preliminary results on subjects with limb deficiency,” in 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2012, pp. 1346–1349. [8] M. L. Homer, A. V. Nurmikko, J. P. Donoghue, and L. R. Hochberg, “Sensors and decoding for intracortical brain computer interfaces,” Annu. Rev. Biomed. Eng., vol. 15, no. 1, pp. 383–405, 2013. [9] L. R. Hochberg et al., “Reach and grasp by people with tetraplegia using a neurally controlled robotic arm,” Nature, vol. 485, no. 7398, pp. 372–375, May 2012. Available: [10] J. P. Donoghue, A. Nurmikko, M. Black, and L. R. Hochberg, “Assistive technology and robotic control using motor cortex ensemble-based neural interface systems in humans with tetraplegia,” J. Physiol., vol. 579, no. 3, pp. 603–611, 2007. [11] C. Bouton, “Decoding neural activity from an intracortical implant in humans with tetraplegia,” in Biomedical Science & Engineering Conference, 2009. BSEC 2009. First Annual ORNL, 2009, pp. 1–1.