Developmentof ComplexCurriculaforMolecularBionicsand InfobionicsProgramswithina consortial* framework** Consortiumleader PETER PAZMANY CATHOLIC UNIVERSITY Consortiummembers SEMMELWEIS UNIVERSITY, DIALOG CAMPUS PUBLISHER The Project has beenrealisedwiththesupportof theEuropean Union and has beenco-financedbytheEuropean SocialFund*** 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 1 **Molekuláris bionika és Infobionika Szakok tananyagának komplex fejlesztése konzorciumi keretben ***A projekt az Európai Unió támogatásával, az Európai Szociális Alap társfinanszírozásával valósul meg. PETER PAZMANY CATHOLIC UNIVERSITY SEMMELWEIS UNIVERSITY sote_logo.jpg dk_fejlec.gif INFOBLOKK 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 2 Peter Pazmany Catholic University Faculty of Information Technology NEURAL INTERFACES AND PROSTHESES ProsthesesWorkingonEEG and SingleCellPrinciples www.itk.ppke.hu Neurális interfészek és protézisek (EEG és egysejtelven működő protézisek) BALÁZS DOMBOVÁRI & GYÖRGY KARMOS LESSON11 Neuralinterfacesand prosthesesProsthesisworkingonEEG and singlecellprinciples www.itk.ppke.hu INTHISLECTUREYOU’LLLEARN: 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 3 Neuralinterfacesand prosthesesProsthesisworkingonEEG and singlecellprinciples www.itk.ppke.hu INTRODUCTION Encouraged by growing recognition of the needs and potentials of people with severe neuromuscular disorders, such as amyotrophic lateral sclerosis, brainstem stroke and spinal cord injury, BCIresearch programs concentratedon developing new communication and control technologies. The originalgoal wasto develop a system which is capable to translate the basic wishes of completely paralyzed (locked-in) patients. Nowadays BCIsdetermine the intents of the patients from several electrophysiological signals (See in the previous lecture: Physiological Basis Of Brain-Computer Interface). Recently the area of BCIapplications increased involving the rehabilitation of stroke patients, control of prosthetic devices. Another new area of BCIapplication is the games and virtual reality. The present day BCIsystems are in experimental phase but some noninvasive BCIsare already commercially available. 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 4 Neuralinterfacesand prosthesesProsthesisworkingonEEG and singlecellprinciples ESSENTIALCOMPONENTSOF A BCISYSTEM 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 5 www.itk.ppke.hu http://www.etsu.edu/cas/bcilab/pictures/Leuthardt.jpg http://www.etsu.edu/cas/bcilab/ BCIsystemscomprisefourmain components(1) electrodestorecordbrainactivity, (2) a ‘decoder’ algorithmthatprocessestheactivitytoextractcontrolsignalsaboutthepresumedintentionsof theBCIuser, (3) an effectortoimplementthedesiredactionextractedbythedecoder, and (4) sensoryfeedbackabouttheresultingeffectoractionthatclosesthecontrolloop. Neural interfaces and prosthesesProsthesis working on EEG and single cell principles www.itk.ppke.hu CLASSIFICATIONOF BCIS We separate non-invasive from invasive BCIs: Non-invasive BCItechniquesuse brain activity recorded with sensors outside the body boundaries. Thesemayworkwith: 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 6 EEG principle:• Slow cortical potentials (SCP), • Oscillatory EEG activity, • Event-related synchronizations (ERS) and desynchronizations(ERD), • Event-related brain potentials (e.gP300, SSVEP), BOLD response in fMRI Near-infrared spectroscopy (NIRS) measuring cortical blood flow (The lasttwotypeof BCIsarenotdiscussedinthiscourse.) www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 7 Neuralinterfacesand prosthesesProsthesisworkingonEEG and singlecellprinciples Invasive BCItechniques use signals recorded by brain implanted micro-or macroelectrodes: Electrocorticogram. Synaptic or extracellular local field potential Action potentials from neurons or neural fibers The spatial resolutions of these methods are different. Schwartz et al., Neuron, 2006 CLASSIFICATIONOF BCIS www.itk.ppke.hu CLASSIFICATIONOF BCIS(CONT.) 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 8 Recordingmethod Advantages Disadvantages Electroencephalogram easyand non-invasive limitedtopographicalresolutionandfrequencyrange; maybecontaminatedbyartifactssuchaselectromyographic(EMG)activityfromcranialmusclesorelectrooculographic(EOG)activity. Electrocorticogram better topographical resolution and frequencyrange requiresimplantationofelectrodearraysonthecorticalsurface,whichhasasyetbeendoneonlyforshortperiods(e.g.,afewdaysorweeks)inHumans Intracorticalrecording provides the highest resolution signals requiresinsertionofmultipleelectrodearrayswithinbraintissueandfacesasyetunresolvedproblemsinminimizingtissuedamageandscarringandensuringlong-termrecordingstability. Neuralinterfacesand prosthesesProsthesisworkingonEEG and singlecellprinciples www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 9 PROSTHESES WORKING ON EEG PRINCIPLE Neuralinterfacesand prosthesesProsthesisworkingonEEG and singlecellprinciples A History of EEG-Based Neuroprosthetics 1963 EEG operates typewriter (Grey Walter 1972 Visual evoked potential used to navigate maze (Vidal) 1980 Slow cortical potentials control virtual rocketship(Birbaumer) 1988 P300evoked potential used to operate simple word processor (Donchin) 1991-94 Sensorimotor rhythms control 1-and 2Dcursor movement (Wolpaw/McFarland) 1998 An ALS patient uses slow cortical potentials to write a letter (Birbaumer) 2000 A SCI Patient uses SMRsto control a simple orthosis(Pfurtscheller 2004 An adaptive SMRBCIprovides better 2-D control (Wolpaw/McFarland) 206 A few ALS patients begin home use of a P300BCI(Vaughan/Wolpaw) http://www.nyas.org/MediaPlayer.aspx?mid=110a9ade-277d-4b7b-a401-b1906fea9afd www.itk.ppke.hu EEG-BASED BCIsGENERATES CONTINUOUS OUTPUT SIGNALS THAT DRIVE COMPUTER CURSORS Several principles of operation have been suggested for such systems. Some groups suggested using slow cortical potentials or mu and beta rhythms. In an attempt to direct the cursor to a particular location on the screen, the subjects often use motor imagery, for example, imagine moving the foot or hand. Computer algorithms were developed that detect event-related desynchronization(ERD) and synchronization (ERS) of the EEGs associated with such imagery. Event-related potentials like P300and visual steady state response are also used as input for BCIs. 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 10 Neuralinterfacesand prosthesesProsthesisworkingonEEG and singlecellprinciples www.itk.ppke.hu Birbaumerand colleagues were the first who developed BCIsystem for ALS (in details see in previous lecture) patients, Their system used slow potential changes. Slow cortical potentials are low frequency potentials (e.g., less than 1 Hz) recorded from the scalp and are associated with various cognitive or sensory-motor events. Decreased cortical activation is associated with scalp positivity and increased activation is associated with negativity. Patients are trained to modify SCPs based on feedback and use this paradigm for BCI-based communication (ThoughtTranslationDevice). 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 11 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles Niels Birbaumer Professor and Director Institute of Medical Psychology and Behavioral Neurobiology, University of Tübingen, Faculty of Medicine. NON-INVASIVEBCIBASEDON SLOWCORTICALPOTENTIAL(SCP) www.itk.ppke.hu THOUGHT TRANSLATION DEVICE (TTD) TTDis a Brain-Computer Interface for the completely paralyzed (LIS) patients using slow-cortical potentials (SCP) to move a cursor on a monitor to select letters. TTDis a feedback device which enables people to respond by voluntary SCP changes. During the training phase, the self-regulation of SCPs is learned through visual-auditory feedback and positive reinforcement. During the spelling phase, patients select letters or words with their SCPs. A psychophysiological system for detection of cognitive functioning in completely paralyzed patients is an integral part of the TTD. 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 12 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles www.itk.ppke.hu THOUGHT TRANSLATION DEVICE (TTD) Inthestandardform,usersreceivevisualfeedbackabouttheirperformancefromacomputerscreenwithtwochoicesonbottomandontop(seeleftpicture).Theuser’scorrespondingSCPresponsesareontherightpicture.Selectiontakesaroundfourseconds.TTDswithauditoryoreventactilefeedbackarealsoavailable. 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 13 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles hamburg smiley2 http://videolectures.net/bbci09_birbaumer_bip/ www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 14 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles 3 THOUGHT TRANSLATION DEVICE (TTD) Time course of a selection cycle. It begins with a warning high-pitchedtone, the feedback phase is marked with a low-pitched tone. http://videolectures.net/bbci09_birbaumer_bip/ www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 15 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles THOUGHT TRANSLATION DEVICE (TTD) E:\BCI2000\SCPBCI_VP3.TIF http://videolectures.net/bbci09_birbaumer_bip/ www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 16 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles LANGUAGESUPPORTPROGRAM Thought translation device makes possible letter selection. The string of letters can be divide into halves by the up or down shift of the SCP. This way the totally paralyzed patient can writeorgivecommands. Image27 http://videolectures.net/bbci09_birbaumer_bip/ www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 17 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles SCHEMATIC STRUCTURE OF THELANGUAGE SUPPORT PROGRAM Boxes show letter sets offered during one trial; solid arrows show the subsequent presentation when a select response is produced; dotted arrows show the presentation following a reject response.At all except the uppermost level, failure to select one of the twochoices results in the presentation of a “go back” option taking the user back to the previous level. http://videolectures.net/bbci09_birbaumer_bip/ A patientinfront of themonitor of theThoughtTranslationDevice. 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 18 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles www.itk.ppke.hu LANGUAGESUPPORTPROGRAM http://videolectures.net/bbci09_birbaumer_bip/ www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 19 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles Letter written by an ALS patient. The acquire of the language supportprogram needs a long training period when the patient learns the letter selection. This letter was written in about eight hours. Birbaumeret al., Nature, 1999 LANGUAGESUPPORTPROGRAM www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 20 Neuralinterfacesand prosthesesProsthesisworkingonEEG and singlecellprinciples dia-0762 QUALITYOF LIFE OF ALSPATIENTS http://videolectures.net/bbci09_birbaumer_bip/ The self ratedquality of life of the ALS patients improvein time and as a result of thepossibility of communicationby BCI. www.itk.ppke.hu ERD is amplitude attenuation and ERS is amplitude enhancement of a certain EEG rhythmtimelockedtoan event. To improve the signal-to-noise ratio simpleaveraging over trials cannot be applied. To overcome this problem Pfurtschellerintroduced a non-stationary analysis method based on an event-related paradigm. The ERD/ERS can be estimated in subject-dependent frequency bands. The outcome of the ERDanalysis resembles in some way an averaged evoked response signal with a temporal profile depending on the mean power ratio of the EEG in the actual measurement interval and a reference interval. 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 21 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles http://bci.tugraz.at/Pfurtscheller.jpg EVENT-RELATED DESYNCHRONIZATION(ERD) AND EVENT-RELATED SYNCHRONIZATION(ERS) GertPfurtscheller Professor Emeritus Institute for Knowledge DiscoveryLab.of Brain-Computer InterfacesGraz University of Technology www.itk.ppke.hu EVENT-RELATED DESYNCHRONIZATIONAND 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 22 C:\Users\BDombovari\Documents\PhD\TÁMOP\BCI\ERD-ERS kép.png LemmS. et al. PLoSComputBiol, 2009 ERDmeasuresthedeviationoftheevent-relateddynamics(bluesolidline)fromaconstantbaselinelevel(blackdashedline)thatisobtainedasaveragedpowerinthereferenceperiodTref. Neural interfaces and prostheses Prosthesis working on EEG and single cell principles www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 23 Neuralinterfacesand prosthesesProsthesisworkingonEEG and singlecellprinciples ERDand ERS appear above the motor area of the neocortex at self-paced movements. The localization of ERDas well as the ERS corresponds to the cortical representation of the given movement. ERDis amplitude attenuation in the alpha band before and at the beginning of the movement; ERS is amplitude enhancement in the beta band after the completion of the movement. The ERDand ESDalso develop at passive movements and movements imagination without the real movement. This means that they can be used in totally paralyzed patients as an input to a BCIsystem. The Pfurtschellergroup used the ERD/ERS to control a hand orthosisin a tetraplegicpatient with high level spinal cord injury. CHARACTERISTICSOF THE ERD/ERS www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 24 Neuralinterfacesand prosthesesProsthesisworkingonEEG and singlecellprinciples (a) Grand average ERDcurves calculated in the alpha and beta bands in a right hand movementtask(left side). Grand average maps calculated for a 125 msinterval during movement (A) and after movement-offset in the recovery period (B) (right side). (b) Maps displaying ERDand ERS for an interval of 125 msduring voluntary movement of the hand(left, upper panel) and movement of the foot (left, lower panel). The motor homunculus with a possible mechanism of cortical activation/deactivation gated by thalamic structures is shown on the right. (c) Superimposed ERDcurves with beta rebound from eight sessions with right motor imageryin one subject. Analyzed frequency band 18±26 Hz, EEG recorded from electrode position C3. In addition to the individual curves also the grand average ERDcurve is plotted (left side). ERDmaps from one session displaying simultaneous contralateral ERDand ipsilateralERS during and contralateral ERS after motor imagery (right side). The scalp electrode positions are marked, (modifiedfrom Pfurtschelleret al., 1997c). `Red' indicates power decrease or ERDand `blue' power increase or ERS. Pfurtschellerand Lopesda Silva, Clin. Neurophysiol. 1999 CHARACTERISTICSOF THE ERD/ERS www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 25 Neuralinterfacesand prosthesesProsthesisworkingonEEG and singlecellprinciples HANDORTHOSISINA TETRAPLEGICPATIENTCONTROLEDBYBRAINOSCILLATION Patient T.S. with hardware components of the portable brain-computer interface.On the patient's left hand, the electrical driven hand orthosiscan be seen. Pfurtschelleret al., NeuroscienceLetters, 2000 www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 26 Neuralinterfacesand prosthesesProsthesisworkingonEEG and singlecellprinciples HANDORTHOSISINA TETRAPLEGICPATIENTCONTROLEDBYBRAINOSCILLATION Pfurtschelleret al., NeuroscienceLetters, 2000 Time courses of the band power in the reactive beta frequency range for three characteristic sessions (# 33, 55 and 62). Classification accuracy (calculated for 160 trials per session) over a time span of 5 months.FB indicatessessions with feedback. T.S. started with a classification accuracy of about 65% and ended with an accuracy of nearly 100%. www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 27 Neuralinterfacesand prosthesesProsthesisworkingonEEG and singlecellprinciples HANDORTHOSISINA TETRAPLEGICPATIENTCONTROLEDBYBRAINOSCILLATION Pfurtschelleret al., NeuroscienceLetters, 2000 C:\Documents and Settings\gkarmos\Desktop\Picture1.png www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 28 Neuralinterfacesand prosthesesProsthesisworkingonEEG and singlecellprinciples THE ROLEOF FEEDBACKINLEARNINGERD/ERSPARADIGM oxford_book Band power (11-13 Hz) time courses ±95% confidence interval displaying ERDand ERS from training session without feedback (left) and session with feedback (right). These data are from one able-bodied subject while he imagined left and right hand movement. Grey areas indicate the time of cue presentation. Sites C3and C4are located over the left and right sensorimotor areas of the brain, respectively. www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 29 Neuralinterfacesand prosthesesProsthesisworkingonEEG and singlecellprinciples CHARACTERISTICSOF THE ERD/ERS Grand average (15 participants) time frequency maps of PM, ME, and MI are plotted for the left and right hand. The ERD/ERS pattern for the different motor tasks show ERDduring movement or movement imagination, especially in .-and ß-frequency bands, which turns to an ERS after termination of movement. The patterns are most pronounced during PMfollowed by ME and weakest during MI. After MI only weak ERS can be observed. (PM) (ME) (MI) Kaiser et al. Front. Neurosci. (2011) 5:86. doi: 10.3389/fnins.2011.00086 More abouttheGraz BCIgroupcanbe findattheirhomepage: http://bci.tugraz.at www.itk.ppke.hu This type of BCIuses the P300component of the event-related brain potential. The P300BCIsystem described by the Donchin’sgroup consists a matrix of letters or other symbols on the monitor. Rows or columns of the matrix are flashing randomly in rapid succession. The flashing letter or symbol that the user wants to select produces a P300potential. By detecting this P300potential, the BCIsystem can determine the user’s choice. 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 30 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles http://psychology.usf.edu/images/people/faculty/edonchin.gif P300-BASED BCI Emanuel Donchin Emeritus Professor Department of Psychology Univ. of IIllinoisin Urbana-Champaign www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 31 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles C:\WINDOWS\TEMP\auto0.bmp CHARACTERISTICSOF THE P300 Johnson 1986 Amplitude of P300is sensitive to stimulus probability, meaning of the stimulus, and the psychological resources allocated to the processing of it. The more complex are the stimuli to be processed the latency of the P300is longer. The P300has a distinct scalp distribution. It is typically largest over the Pzsite. P300BCIsrequire no training. G:\TÁMOP_PROSTH\BCI\ÁBRÁK\Media,79027,en.jpg www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 32 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles d1 The rowsand thecolumnsof thematrixarebrieflyflashedin sequence. The user is asked to count each flash of one element, called the “target,” and ignore other flashes. 6x6matrix, 100 msflash, ISI (interstimulusinterval): 175-250 ms LETTERMATRIXUSEDBYTHEDONCHINGROUP Farwelland Donchin, Electroenceph. clin. Neurophys., 1988 www.itk.ppke.hu DEMONSTRATION OF P300-BASED BCI 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 33 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles H2 V3 K The P300BCIdoes not need long training, different laboratories introduced and it is widely used as protopypeof BCI. http://www.physorg.com/news159453062.html P300BCIcan be used to control robot arm: http://www.youtube.com/watch?v=hs5L6EmOB2M&feature=related Averaged data from a P300BCIat the NextFestExpo (Allison, 2004) http://www.scribd.com/doc/27142772/BCI-Systems www.itk.ppke.hu P300BCICONTROLLEDWHEELCHAIR 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 34 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles A-B: Information represented in the visual display, which is an environment abstraction displayed from the user’s point of view. Stimuli are flashingdots showing the directions. C: Map generated by the autonomous navigation system and the trajectory of the wheelchair in one real experiment. http://webdiis.unizar.es/~jminguez/wheelchairIturrateet al., IEEETrans. Robotics, 2009, 25: 614-627. A B C G:\TÁMOP_PROSTH\BCI\ÁBRÁK\wheelchair\vlcsnap-2011-07-24-09h06m33s3.png www.itk.ppke.hu BCIBASEDONMURHYHTM 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 35 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles Different noninvasive BCImethods were developed in the Wolpawlaboratory. They use the mu rhythm of the EEG. Mu rhythm is a 8-14 Hz “idling” rhythm found over primary motor/somatosensory cortex. These areas are located around the central sulcus. The mu rhythm is largest over central sites such as C3, Cz, or C4. Mu rhythm is often called sensorimotor rhythm (SMR). Mu activity is visible before a voluntary movement, sometimes 2 seconds before it begins. Mu activity reflects movement planning and execution. Mu activity can be voluntarily controlled by movement related imagery. This becomes easier in trained subjects. With continued practice, this control tends to become automatic, as is the case with many motor skills and imagery becomes unnecessary. Jonathan R. Wolpaw, M.D. Chief, Laboratoryof Neural Injury and Repair WadsworthCenter, NYSDept. of Health Albany, NY G:\TÁMOP_PROSTH\BCI\ÁBRÁK\wadworth\Wolpaw2.png www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 36 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles G:\TÁMOP_PROSTH\BCI\ÁBRÁK\nibib_092605_083045 (1).jpg Speller BCIBASEDONMURHYHTM In the early WadswothBCIletter speller the users directed a ball by the mu rhythm. The ball was moving through the screen and the user had to move it to the up or down, pointing the letter string containing the selected letter. The mu rhythm and the ERD/ERS BCIare similar but ERD/ERS is elicited by external stimuli while mu rhythm is modulated spontaneous activity. C:\Documents and Settings\gkarmos\Desktop\Picture1.png Wolpawet al., IEEETrans. onRehab. Enging., 2000 www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 37 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles Ina 2Dversion theuserslearnedto control mu activity over bothhemispheres, adding left-right control. IntheWadsworthlaboratoryfirstdeveloped3DBCIwithnoninvasiveEEG technique. BCIBASEDONMURHYHTM www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 38 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles G:\TÁMOP_PROSTH\BCI\ÁBRÁK\wadworth\vlcsnap-2011-07-24-15h29m27s138.png WADSWORTHBCISYSTEMS G:\TÁMOP_PROSTH\BCI\ÁBRÁK\wadworth\vlcsnap-2011-07-24-15h30m57s18.png The Wolpawgroup further developed the P300letter spelling system with a predictive speller, speeding up the system to 7 selection per min. http://www.wadsworth.org/bci/wm_video.html http://www.npr.org/2011/05/12/135598390/mind-reading-technology-turns-thought-into-action www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 39 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles WADSWORTHHOME BCISYSTEM Recently the Wadsworth group begun to provide severely disabled users with in-home P300-based BCIsystems to use for daily communication and control tasks. A 49-year-old scientist with ALS has used this BCIsystem on a daily basis for approximately three years, finding it superior to his eyegazesystem (a letter-selection device based on eye-gaze direction) and using it from four to six hours per day for email and other communication purposes.http://www.wadsworth.org/bci/wm_video.html http://www.cbsnews.com/video/watch/?id=5228109nVaughanet al., IEEETransactionsonRehab. Engin., 2006, 14: 229–233. www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 40 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles BCIBASED ON STEADY STATE VISUAL EVOKED POTENTIALS (SSVEP) This response is evoked by flickering lights in the range of 4-40 Hz. Steady state response (SSR) is sine wave like oscillation with frequency of the repetitive stimuli. It can be studied in the frequency domain (FFT). It may also be elicited by auditory and somatosensory stimuli. Nearly all subjects seem to have this activity. SSVEPrequires no training. Selective attention can enhance SSVEP. Averaged SSVEPwaveform elicited by a flickering bar in the left visual field. Bold lines show SSVEPwhen the left visual field was attended, while the thin lines show the SSVEPto the same stimulus when the right visual field was attended. Müller and Hillyard, ClinNeurophysiol., 2000 www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 41 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles Two or more light source flashing with different repetition rates can be used to elicit SSVEPswith increased amplitude in the frequency of the attended light source. The earlyBCIliterature stated that it is necessary to shift gaze to the selected light source. According this SSVEPBCIsare “dependent” meaning they depend on muscle activity. If true, SSVEPBCIswould not work in locked in patients. Recent studies indicated that gaze is not an essential factor. BCIBASED ON STEADY STATE VISUAL EVOKED POTENTIALS (SSVEP) Amplitudespectrumof SSVEPto7 Hz stimulation. a: singletrialspectrum b: averageof 40 trials, verticallinesgiveSD www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 42 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles BCIBASED ON STEADY STATE VISUAL EVOKED POTENTIALS (SSVEP) Smallrobot controlledbySSVEPBCI. FourLED lightsourceflashingwithdifferentrepetitionrate. Seevideo at: http://www.gtec.at/content/download/1855/11541/version/4/ http://www.gtec.at/Download www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 43 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles nrn0401_229a_f2 nrn0401_229a_f2 nrn0401_229a_f2 nrn0401_229a_f2 INVASIVEBCITECHNIQUES The limitations of noninvasiveEEG-based BCIsystemsis the distances between the recording electrodeson the scalp and the underlying corticaltissue.Thisis whythesignalamplitudeissmallthebandwidthislimited and S/N ratio is low. The advantageof theinvasivetechniquesis thattherecordingelectrodesareonthesurfaceof theneocortex orintheneuraltissue. The ECoGareoftencalledsemi-invasivebecausetheelectrodesarenotinsertedintothetissue. www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 44 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles BCIBASEDONELECTROCORTICOGRAM(ECOG) IMAGE02 Subduralelectrodegridplacedonthecorticalsurface. BCIsbasedonECoGareworkingonthesameprinciplesastheEEG BCIs. ECoGBCIresearch has almost exclusively been performedon epilepsy patients, in whom subduralgrids are clinically placed over suspected epileptogenicfoci. Advantage of ECoGbased systems is that electrodes are on the cortical surface, yielding a much finer spatial resolution on the order of mm as well as the ability to record higher-frequency (10–200 Hz) content in the signal. Itwasfoundthatpatientscould quickly learn to modulate high-frequency gammarhythms in both motorcortical areas and in Broca’sspeech area to control a one-dimensional computer cursorin real time. www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 45 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles BCITECHNIQUES BASED ON RECORDINGNEURONAL ACTION POTENTIALS Richard A. Andersen James G. Boswell Prof. of Neuroscience Divisionof Biology California Inst. of Technology Andrew B. Schwartz Professor of Neurobiology University of Minnesota Pittsburgh, PA G:\TÁMOP_PROSTH\BCI\ÁBRÁK\SINGLE UNIT\schwartz.jpg Three pioneers of cortical motor prostheses in primates. They developed technology to record cortical single unit activity in primates and transform it to a signal that controls robotic arm. Miguel A. L. Nicolelis, Anne W. Deane Professor Dept. of Neurobiology DukeUniv. MedicalCenter www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 46 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles http://www.nicolelislab.net/ Helms Tilleryet al., Rev. inNeurosci., 2003 Greenand KalaskaTINS, 2011 CORTICAL MOTOR BCIIN PRIMATES A B C A: Schematic arrangementof the experimental setup. B: Delivering food to the mouth usinga cortically controlled robotic arm. The diagram outlines the arrangement, with a piece of orange clipped into the robot end effector. The inset diagram shows 10 minutes worth of cortical control trajectories from a central start location. C:Behavioral performance improvements in monkeys during BCIcontrol(a) Example of single-trial cursor trajectoriesduring initial (left) and late (right) stages of BCIcontrol learning.(b)The success rate across daily training sessions. www.itk.ppke.hu Philip R. Kennedy developed a special glass microelectrode for chronic recording of neuronal activity. Kennedy and colleagues cortically implanted these microelectrodes filled with neurotrophicgrowth factor first into animals. The axon of the neurons targeted by the electrodes grow into them and allows recording of the spike activity. After a series of successful animal experiments they got FDA license to implant the neurotropic electrode into the brains of several ALS patients from 1998. This implantation requires major surgery lasting about 10 hours. Neural activity has beenrecorded with this type of electrodes inALSpatientsfor five years. 2011.09.14.. TÁMOP–4.1.2-08/2/A/KMR-2009-0006 47 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles harnessing1 Philip R. Kennedy M.D. NeuralSignalsInc. Duluth, Ga. NEUROTROPHICELECTRODE Neural interfaces and prosthesesProsthesis working on EEG and single cell principles C:\Documents and Settings\gkarmos\Desktop\Picture2.png NEUROTROPHICELECTRODE The six layers of cortex are diagrammed with the electrode tip in position near layer five that contains corticospinaltractpyramidalcells. Neuritesare shown growing into the tip, through the cone, and out the top end so as to hold the glass cone tip in position. . 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 48 www.itk.ppke.hu Bartelset al., J. Neurosci. Methods, 2008 www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 49 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles currentelectrodesystem NEUROTROPHICELECTRODESYSTEM The wires commingout from the cone is attached to amplifiers and FM transmitters located on the skull, under the scalp. No wires or batteries are used. Power is provided by a power induction system. The neural signals are transmitted to and processed by a computer to activate a switch or drive a cursor and hence provide communication. http://www.nyas.org/MediaPlayer.aspx?mid=915927ef-477a-4830-beff-0fafade6860f www.itk.ppke.hu 2011.09.14.. TÁMOP–4.1.2-08/2/A/KMR-2009-0006 50 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles http://www.plosone.org/article/fetchObject.action?uri=info%3Adoi%2F10.1371%2Fjournal.pone.0008218.g008&representation=PNG_M www.speechprosthesis.org Guentheret al., PLoSONE2009, 4(12): e8218doi:10.1371/journal.pone.0008218 A consortium led by Kennedy developed BCImethod for speech sound production. NeurotrophicElectrode were implanted in the speech-related region of the left precentralgyrusof a human volunteer suffering from locked-in syndrome. Neural signals generated during attempted speech were used to drive a speech synthesizer. Volunteer’s vowel productions with the synthesizer improved quickly with practice, with a 25% improvement. They concluded that neural prostheses may have the potential to provide near-conversational synthetic speech output for individuals with severely impaired speech motor control. A: Left panels: Axial (top) and sagittal (bottom) fMRI slices showing brain activation along the precentralgyrusduring attempted word generation prior to implantation. Red lines denote pre-central sulcus; yellow lines denote central sulcus. Right panels: Corresponding images from a post-implant CT scan showing location of electrode. B: 3DCT image showing electrode wire entering duramater. BCIFOR REAL-TIME SPEECH SYNTHESIS www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 51 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles BRAINGATE SYSTEM JOHN DONOGHUE Henry Merritt WristonProfessorDept. of Neuroscience Brown University Providence, RI BrainGateSystem is a BCIdesigned for human use. It is the result of research and development at premier academic institutions such as Brown University, the Massachusetts Institute of Technology, Emory University, and the University of Utah. The program led by John Donoghue funded by the CyberkineticsInc. Currently, the system consists of a „sensor”(a device implanted in the brain that records signals directly related to imagined limb movement); a „decoder”(a set of computers and embedded software that turns the brain signals into a useful command for an external device); and, the „external device”–which could be a standard computer desktop or other communication device, a powered wheelchair, a prosthetic or robotic limb. http://www.braingate.com http://donoghue.neuro.brown.edu/ http://www.brown.edu/Administration/News_Bureau/2006-07/06-002.html BRAINGATE SENSOR–UTAH ELECTRODEARRAY 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 52 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles www.itk.ppke.hu C:\Documents and Settings\gkarmos\Desktop\Picture1.png The BrainGatesensor, resting on a US penny. It is a 100-electrode Si sensor array developed at the Utah University. The electrodes are 1.0-1.5 mm long, insulated with Parylene. If inserted into the neocortex, the uninsulated, Ptcoated tips reach the fifth large pyramidal cell layer. http://www.cyberkineticsinc.comhttp://www.braingate.com/www.BrainGate2.com 10x10array ~ 4x4mmplatform 1.0-1.5 mm www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 53 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles The first participant in the BrainGatetrial (MN). He is sitting in a wheelchair, mechanically ventilated through a tracheostomy. The grey box (arrow) connected to the percutaneous pedestal contains amplifier and signal conditioning hardware. He is looking at the monitor, directing the neural cursor towards the orange square in this 16-target 'grid' task. At left the MRI slice of his brain with the motor cortex (arrow). The small square represent the place of the electrode array. http://www.braingate.com/meet_the_patients.htmlhttp://www.nature.com/nature/journal/v442/n7099/suppinfo/nature04970.html BRAINGATE SYSTEM C:\Documents and Settings\gkarmos\Desktop\Picture1.png Matthew Nagel (1979 –2007) was the first patient of the BrainGateproject. He suffered a spinal cord injury in 2001 (C4tetraplegic) and the BrainGateelectrode array was implanted into his motor cortexin 2004. The implant was removed after one year. The system decoded the activity of a population of his cortical neurons. This way he could control computer cursor, he could open and close a prosthetic hand, and perform rudimentary actions with a multi-jointed robotic arm. This proves that using these neural signals paralyzed humans can directly and successfully control external devices. BRAINGATE SYSTEM 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 54 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles www.itk.ppke.hu http://upload.wikimedia.org/wikipedia/commons/thumb/f/fd/BrainGate.jpg/256px-BrainGate.jpg Source: WikimediaCommon Left: The present BrainGateSystem uses percutaneous connector between the lead of the electrode array and the preamplifier.This may be a source of infection. Right: In the next generation of the System the brainelectricalsignalswill be transmitted by telemetry. Overview http://nurmikko.engin.brown.edu/?q=node/1 Source: Nanophotonicsand NeuroengineeringLaboratory, Brown Univ. www.itk.ppke.hu REFERENCES Andersen, R.A., Budrick, J.W., Musallam, S., Pesaran, B., Cham, J.G.: Cognitiveneuralprosthetics. TrendsCogn. Sci. 2004, 8: 486–493. Lebedev, M.A., Nicolelis, M.A.: Brain-machine interfaces: past, present and future.TrendsNeurosci.2006, 9: 536-546. Hochberg, L.R., Serruya, M.D., Friehs, G.M., Mukand, J.A., Saleh, M., Caplan,A.H., Branner, A., Chen, D., Penn, R.D., and Donoghue, J.P.:Neuronalensemble control of prosthetic devices by a human with tetraplegia. Nature, 2006, 442:164–171. Schwartz, A.B., Cui, X.T., Weber, D.J., Moran, D.W.: Brain-controlledinterfaces: movementrestorationwithneuralprosthetics. Neuron 2006, 52: 205–220. Donoghue, J.P.: Bridgingthebraintotheworld: a perspectiveonneuralinterfacesystems. Neuron, 2008, 60: 511–521. Song, Y.-K., Borton, D.A., Park, S., Patterson, W.R., Bull, C.W., Laiwalla, F., Mislow, J., Simeral, J.D., Donoghue, J.P., Nurmikko, A.V.: ActiveMicroelectronicNeurosensorArraysforImplantableBrainCommunicationInterfaces. IEEETransNeuralSystRehabilEng. 2009, 17: 339–345. Andersen, R.A., Hwang,E.J., Mulliken, G.H. Cognitiveneuralprosthetics. Annu. Rev. Psychol., 2010, 61: 169-190. Simeral, J.D., Kim, S-P., Black, M.J., Donoghue, J.P., Hochberg, L.R.: Neuralcontrolof cursortrajectoryand clickbya human withtetraplegia1000 daysafterimplantof an intracorticalmicroelectrodearray, J. NeuralEng. 2011, 8: 025027 2011.09.14.. TÁMOP–4.1.2-08/2/A/KMR-2009-0006 55 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles www.itk.ppke.hu http://motorlab.neurobio.pitt.edu/ http://vis.caltech.edu/ http://www.mp.uni-tuebingen.de/mp/index.php?id=137 http://bci.tugraz.at/ http://www.nicolelislab.net/ http://www.neuralsignals.com/ http://www.npr.org/2011/05/12/135598390/mind-reading-technology-turns-thought-into-action http://www.braingate.com http://www.cyberkineticsinc.com/index.htm http://news.brown.edu/pressreleases/2011/03/braingate http://www.bciresearch.org/html/media/video-flash-hq.html http://www.nyas.org/Publications/Ebriefings/Detail.aspx?cid=1914df78-19ec-440e-8d0d-19a8bf81140c http://videolectures.net/bbci09_berlin/ http://future-bnci.org/index.php?option=com_content&view=article&id=133:cognitive-science-160-introduction-to-bci-systems-brendan-allison&catid=50:bci-class&Itemid=61 www.bbci.de 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 56 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles LINKS www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 57 Neural interfaces and prostheses Prosthesis working on EEG and single cell principles REVIEW QUESTIONS: • Which are the essential components of a BCI? • List the brain electrical signals used in noninvasive BCIs. • List the brain electrical signals used in invasive BCIs. • Describe the Thought Translation Device. • How does the patient learn to use slow cortical potentials in BCI? • Describe the Language support program of the TTD • What are the characteristics of the ERD/ERS? • What is the principle of the P300BCI? • Which electrical signal is used inthe Wadsworth BCIsystem? • What are the characteristics of the visual steady state evoked potential? • What is the neurotropic electrode? • What kind of electrode is used inthe Brain Gate System?