2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 1 Development of Complex Curricula for Molecular Bionics and Infobionics Programs within a consortial* framework** Consortium leader PETER PAZMANY CATHOLIC UNIVERSITY Consortium members SEMMELWEIS UNIVERSITY, DIALOG CAMPUS PUBLISHER The Project has been realised with the support of the European Union and has been co-financed by the European Social Fund *** **Molekuláris bionika és Infobionika Szakok tananyagának komplex fejlesztése konzorciumi keretben ***A projekt az Európai Unió támogatásával, az Európai Szociális Alap társfinanszírozásával valósul meg. PETER PAZMANY CATHOLIC UNIVERSITY SEMMELWEIS UNIVERSITY sote_logo.jpg dk_fejlec.gif INFOBLOKK 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 2 Peter Pazmany Catholic University Faculty of Information Technology NEURAL INTERFACES AND PROSTHESES Perspective of Brain-Machine Interface www.itk.ppke.hu Neurális interfészek és protézisek (Agy-gép kapcsolat perspektívája) BALÁZS DOMBOVÁRI & GYÖRGY KARMOS Neural interfaces and prosthesesPerspective of Brain-Machine Interface www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 3 IN THIS LECTURE YOU’LL LEARN: – Future direction of invasive and non-invasive BCIdevelopment – What are the issues to improve neural recordings – What is a hybrid BCI – Today’s commercially available BCIsystems Neural interfaces and prosthesesPerspective of Brain-Machine Interface www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 4 INTRODUCTION BCIresearch became a very popular research area in the recent years. The number of BCIrelated papers has been extremely multiplied thanks to the fast evolution of the technology. Most of the BCIresearch groups are focusing on the restoration of communication and control of severely paralyzed patients. But BCIcommunication is becoming useful for healthy people too. The goal of this lecture is to show some possible opportunities of future BCIresearch. Dornhege,G. (ed.) TowardBrain-ComputerInterfacing, MIT Press, 2007 Neural interfaces and prosthesesPerspective of Brain-Machine Interface www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 5 CRUCIAL ISSUES FOR DEVELOPING BCISIN THE FUTURE BCIindependence from normal neuromuscular communication channels and dependence on internal aspects of normal brain function; Selection of signal acquisition methods, signal features, feature extraction methods, translation algorithms, output devices, and operational protocols; Development of user training strategies; Attention to psychological and behavioral factors that affect motivation and success of the users; Adoption of standard research methods and evaluation criteria; Choice of applications and user groups; Reveal largely unknown capacities and limitations of non-muscular communication channels. Neural interfaces and prosthesesPerspective of Brain-Machine Interface All of the present BCIsare „investigational devices”. According to Miguel Nicolelis(Duke University), in the next 10-20 years brain-computer and brain-machine interfaces (BMIs): include fully implantable recording system• wirelessly transmit multiplestreams of electrical signals, derived from thousands ofneurons capable of decoding spatial and temporalcharacteristics of movements TheseBMIswould utilize acombination of high-order motor commandsand peripherallow-level control signals. 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 6 FUTUREDIRECTIONSOF INVASIVEBCIS Nicolelis, Nature, 2001 Neural interfaces and prosthesesPerspective of Brain-Machine Interface To achieve the ambitious goal of creating a clinically useful invasive BMI for restoring limb mobility, one has to pass the following key bottlenecks: Develop a new type of 3Dprobe array with thousands of channels to obtain stable, very long-term recording of large population of neural ensembles from multiple brain areas. Develop computational efficient algorithms into BCIsoftware which translate the neural activity into command signals for artificial actuators. Implement new type of limb prosthetics which are capable to accept above mentioned command signals. 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 7 FUTUREDIRECTIONSOF INVASIVEBCIS Neural interfaces and prosthesesPerspective of Brain-Machine Interface ISSUESTOIMPROVENEURALRECORDINGS Currently chronically implanted probe arrays offer the best compromise to operate invasive BCIs. Several significant improvements are required before theybecome fully applicable for clinical applications in humans: • Biological compatibility • Wireless headstagesto reduce the risks of infection Many new ideas of how to improve neuronal recordings have been proposed recently. These range from ceramic-based multi-electrode arrays to nanotechnology probes that access the brain through the vascular system. 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 8 Neural interfaces and prosthesesPerspective of Brain-Machine Interface 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 9 HYBRIDBCI ConventionalBCIsrely on only one signal(e.g. ERD/ERS, steady-state evoked potentials, P300evoked potentials,etc). HybridBCIsexploit the advantages of different approaches. Atypical hybrid BCIiscomposed of one BCIand another system (which might be another BCI), and must achieve specific goals better than a conventional system. The hybrid BCIcan either have more than one input and the inputs aretypically processed simultaneously. One of the great challenges in hybrid BCIresearch is identifying the best combinations of signals to accomplish desired goals. www.itk.ppke.hu Neural interfaces and prosthesesPerspective of Brain-Machine Interface 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 10 HYBRIDBCIS www.itk.ppke.hu Hybrid BCIcan have two inputs and the inputs are typically processed simultaneously(B,C), or canoperate two systems sequentially, whereby the first system can act as a “brain switch” (A,D,E) or as “selector” (F,G).Pfurtschelleret al. FrontiersinNeuroscience2010 Neural interfaces and prosthesesPerspective of Brain-Machine Interface 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 11 H-BCICONTROLLED ARM FES-ORTHOSIS www.itk.ppke.hu As an example, a hybrid-Brain Computer Interface (h-BCI), i.e. a combination of a brain switch and a shoulder position sensor, for control of anarm neuroprosthesisbased on Functional Electrical Stimulation (FES) and a mechanical orthosis,can be helpful for spinal cord injured hemipareticpatients. For setup of the brain switch a special pattern in the recorded EEG signals (beta rebound) after imagination of foot movements is being analyzed. When the BCI-output (classifier) exceeds certain thresholds, two different brain-switch signals are generated, which select predefined control modes. They controls the degree of elbow flexion / extension or the degree of hand opening / closing, depending on the movements of the shoulder. TOBI-Newsletter, ISSUE No2 May 2011www.tobi-project.org Neural interfaces and prosthesesPerspectiveof Brain-MachineInterface 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 12 Participantduring the experiment. In the background, thevirtual street with shops and animated avatars is projectedin stereo view so that the participant has the illusion of beingin the street. In reality, the subject is sitting on a comfortablechair, wearing shutter glasses and an electrode cap(see zoomed picture). Pfurtschelleret al., BrainRes., 2006 www.itk.ppke.hu BCI AND VIRTUALREALITY CombiningBCIwithvirtualenvironment(VE) wasfoundtoimproveperformance of thesubjects. Theymade fewererrors, reportedthatBCIswereeasiertolearnand use, andtheyenjoyedBCIusemore. VEsenhancedvividnessand mentaleffort, whichledtomore distinctbrainpatternsand improvedpatternrecognitionperformance.Leebet al., Comput. Intell. Neurosci, 2007Leebet al., IEEERehabilitation, 2007 Neural interfaces and prosthesesPerspective of Brain-Machine Interface 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 13 INTERNATIONAL BCIRESEARCH PROJECTS www.itk.ppke.hu EffectiveBCIresearchand developmentrequireeffortsof scientistsofdifferentfields. A series of internationalcollaborativeprojectsareinprogress: TOBI is a European integrated project which will develop practical technology for brain-computer interaction (BCI). (2008-2012) http://www.tobi-project.org/ DECODERis a European project that will deploy BCIfor the detection of consciousness in non-responsive patients.(2010-2012) http://www.decoderproject.eu/ BETTERis a European project thatwilldevelopa newapproachforgaittraininginwhichassistivetechnologies(ATs) mightbe improvedifcombinedwithnon-invasiveBNCI. (2010-2012). http://www.iai.csic.es/better/ BrainAbleis a 7thFramework Programme„Autonomyand socialinclusionthroughmixed realityBrain-ComputerInterfaces: Connectingthedisabledtotheirphysicaland socialworld” http://www.brainable.org/en/Pages/Home.aspx Walk Again is a multinational collaborative program to develop high performance brain-controlled prosthetic device. http://www.walkagainproject.org/ FutureBNCIis a project fundedbytheSeventhFramework Programme, exploringfuturedirectionswithBCIs. http://future-bnci.org/ Neural interfaces and prosthesesPerspective of Brain-Machine Interface 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 14 COMMERCIALLY AVAILABLE BCISYSTEMS www.itk.ppke.hu In the recent years a series of companies producing electrophysiological instruments developed BCIsystems. These are using the results of the research laboratories. These BCIsystems may be used in further research applications but at the same time they intend to simplify research-grade EEG for everyday use for the people. Since the idea of BCIbecame very popular companies are advertising these systems as tools using thoughts, feeling, emotion to be more creative or to relax. Games are developed using BCIto direct ball or so. Even it is suggested for market research to get true insight about how people respond and feel about material presented to them. The BCItechnology is moving to toys, games. It may get important role in sports as well as in alertness control in cars etc. In this way BCIsare going to become part of our everyday life. Neural interfaces and prosthesesPerspective of Brain-Machine Interface 14 channel high resolution, neuro-signal acquisition and processing wireless neuro-headset. The EmotivSoftware Development Kit includes a TestBenchsoftware which provides real-time display of the Emotivheadset data stream, including EEG, contact quality, FFT, gyro, wireless packet acquisition/loss display, marker events, headset battery level. Affective suitecan reportedly measure the emotional states of the user. Anger, fear, frustration. Emotivputs forth the example that this could be used to have games increase or decrease the difficulty level depending on the player’s state of mind. The Cognitive Suiteis the control mechanism that allows players to control objects, and the Expressive suitewhich measures and interprets facial expressions of the user. The descriptions and demos are vivid, for example: You smile and thus your avatar smiles. EMOTIVEPOCNEUROHEADSET emotiv_BCI.jpg 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 15 www.itk.ppke.hu http://www.emotiv.comhttp://www.emotiv.com/corporate/media.php http://www.emotiv.com/upload/media/4.jpg Neural interfaces and prosthesesPerspective of Brain-Machine Interface BCISPELLEROF G.TECMEDICALENGINEERING TheintendiX®system is based on visually evoked EEG potentials (P300). Itenablesthe user to sequentially select characters from a keyboard-like matrix on the screen just by paying attention to the target for several seconds. Spellingrate of 5 to 10 characters per minute can be achieved by the majority of healthy users at their first trial. To control external devices a specialextension tool called"extendiX"serves as an interface. 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 16 www.itk.ppke.hu intendX.jpg http://www.intendix.com Video here http://www.biosignal.at/Intendix/img/intendiX_system.jpg http://www.gtec.at/ Neural interfaces and prosthesesPerspectiveof Brain-MachineInterface The Neurosycompanydefinesthemissionof thecompanytocreatea paradigmshift inbiosensorsand makethemaccessiblefora broadmarket outsidethelaboratory. Theyintendtosimplifyresearch-gradeEEG foreverydayuseforrealpeople. TheirsystemsarecombiningEEG (brainwaves), EMG (muscleactivity), EOG (eyemovement) and ECG (heart) biosensing. The Neuroskysystemsaredevelopedforresearch, education, sport and entertainmentapplications. There are currently over a dozen games and educational applications available for download. Mattelpartnered withNeuroSkytoreleasetheMindflexgame in2009. THE NEUROSKYBCI 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 17 www.itk.ppke.hu http://www.neurosky.com http://neurosky.com/Images/ProductImage?ProductImageID=6 plxxwave e1311104489868 Just Got an XWave Headset from PLX Company Neural interfaces and prosthesesPerspective of Brain-Machine Interface 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 18 www.itk.ppke.hu REFERENCES Dornhege, G., Millán, J.R., Hinterberger, T., McFarland, D.J., Müller, K.J.: TowardBrain-ComputerInterfacing, The MIT Press, 2007 Miguel ALNicolelis, M.A.L. (ed.):MethodsforNeuralEnsembleRecordings, 2nded. CRCPress; 2008. Graimann, B., Allison, B., Pfurtscheller, G. (eds.) Brain-ComputerInterfaces: RevolutionizingHuman-Computer Interaction, Springer, 2010. Millán JdR, RuppR, Müller-PutzGR, Murray-Smith R, GiugliemmaC, TangermannM, VidaurreC, CincottiF, KüblerA, LeebR, NeuperC, Müller K-R and MattiaD (2010) Combiningbrain–computer interfacesand assistivetechnologies: state-of-the-artandchallenges. Front. Neurosci. 2010, 4:161. doi:10.3389/fnins.2010.00161 Pfurtscheller, G., Allison, B.Z., Bauernfeind, G., Brunner, C., SolisEscalante, T., Scherer, R., Zander, T.O., Müller-Putz, G., Neuper, C., and Birbaumer, N.: The hybridBCI. Front. Neurosci., 2010, 4: 42. doi:10.3389/fnpro.2010.00003. Schalk, G., Mellinger, J.:IntroductiontoBrain-ComputerInterfacingUsingBCI2000, Springer, 2010. Graimann, B., Allison, B., Pfurtscheller, G: Brain-computerInterfaces: Non-invasiveand InvasiveTechnologies, Springer, 2011. Leeb,R., Sagha,H., Chavarriaga,R., MillánJ.R.:A hybridbrain-computerinterfacebasedonthefusionof electroencephalographicand electromyographicactivities. J. Neural Eng.2011, 8:025011. www.itk.ppke.hu 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 19 http://videolectures.net/bbci09_pfurtscheller_thb/ http://www.walkagainproject.org/ http://www.tobiproject.org http://bci.tugraz.at/ http://www.wadsworth.org/bci/index.html http://www.bci2000.org/ http://en.wikipedia.org/wiki/Comparison_of_consumer_brain%E2%80%93computer_interfaces http://www.brain-tuning.de/ http://future-bnci.org/index.php?option=com_content&view=article&id=133:cognitive-science-160-introduction-to-bci-systems-brendan-allison&catid=50:bci-class&Itemid=61 Neural interfaces and prostheses: Prosthesis working on EEG and single cell principles LINKS Neural interfaces and prosthesesPerspective of Brain-Machine Interface 2011.09.14.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 20 www.itk.ppke.hu REVIEW QUESTIONS: WhicharethecrucialissuesfordevelopingfutureBCIs? Whatis hybridBCI? Whatis theadvantageof thehybridBCI? Howcanbe usedvirtualrealityina BCI? WhatarethetrendsinthepresentBCIresearcprojects? Forwhatcanbe usedthecommerciallyavailableBCIs?