10/5/2011. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 1 Development of Complex Curricula for Molecular Bionics and Infobionics Programs within a consortial* framework** Consortium leader PETER PAZMANY CATHOLIC UNIVERSITY Consortium members SEMMELWEIS UNIVERSITY, DIALOG CAMPUS PUBLISHER The Project has been realised with the support of the European Union and has been co-financed by the European Social Fund *** **Molekuláris bionika és Infobionika Szakok tananyagának komplex fejlesztése konzorciumi keretben ***A projekt az Európai Unió támogatásával, az Európai Szociális Alap társfinanszírozásával valósul meg. PETER PAZMANY CATHOLIC UNIVERSITY SEMMELWEIS UNIVERSITY sote_logo.jpg dk_fejlec.gif INFOBLOKK 10/5/2011. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 2 Peter Pazmany Catholic University Faculty of Information Technology Introduction to Neural Processing www.itk.ppke.hu (Bevezetés a Neurális Jelfeldolgozásba) JánosLevendovszky, AndrásOláh, DávidTisza, GergelyTreplán Digitális-neurális-, éskiloprocesszorosarchitektúrákonalapulójelfeldolgozás Digital-and Neural Based Signal Processing & KiloprocessorArrays Signal processing on digital, neural, and kiloprocessor based architectures: Introduction to Neural Processing 10/5/2011. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 3 Outline • Introduction to neural processing • Scope and motivation • Historical overview • Benefits of neural networks • Neurobiological inspiration • Human brain • Network architectures • Applications of neural networks • Summary www.itk.ppke.hu Signal processing on digital, neural, and kiloprocessor based architectures: Introduction to Neural Processing www.itk.ppke.hu 10/5/2011. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 4 Introduction(1) • Inourdays,methodsofartificialintelligence(AI)areincreasinglyappliedinthefieldsofinformationtechnology. • Thesenewapproacheshelpussolveproblemsthatwouldhaveremainunsolvedbytraditionalmethods,-andmoreover-theyusuallyprovideresultswithhigherefficiency. • Lookingatthewell-knowntraditionalareas,likeanalysis,decisionsupport,statistics,control,etc.wecanuseprecise,deterministicmodels.Statisticshelpustoapproximateafunctionortomakeamodel. Signal processing on digital, neural, and kiloprocessor based architectures: Introduction to Neural Processing www.itk.ppke.hu 10/5/2011. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 5 Introduction(2) • Optimizationofasimpleproblemcanbedonebylinearprogramming. • Wealsoknowthatnonlinearordynamicprogrammingarethemainkeymethodstosolveharder,morecomplexproblems. • Butontheotherhand,traditionalthinkingfailsinseveralcases:theproblemcanbesocomplexthatwedonotevenknowtheexactfunction;howcouldweknowitsoptimumthen?! • Predictionusingtimeseriesanalyzingmaynotgiveadequateresults.Inthesecaseswecane.g.usethe”somaAI-method”thatiscalledneuralnetworks. Signal processing on digital, neural, and kiloprocessor based architectures: Introduction to Neural Processing www.itk.ppke.hu 10/5/2011. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 6 Scope and motivation(1) • Whydoweneedtotakeacloserlookatthefunctioningofneuralnetworksandanalogprocessorarrays?Ourmotivationistobeabletocreatemodels-operatingoptimallyunderminimalerrorsignallevels-eveninbarelyknown,highlycomplexITsystems.Theblockdiagramonthenextslideshowsusthenewcomputationalparadigmusingartificialneuralnetworks. Signal processing on digital, neural, and kiloprocessor based architectures: Introduction to Neural Processing 10/5/2011. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 7 Scope and motivation(2) Basedonobservingexamples(inputsanddesiredoutputs)howtolearnthedesiredsignaltransformation? www.itk.ppke.hu stochastic input signal prescribed, unknown transformation desired output error signal Neural network Optimizing the free parameters of the adaptive architecture output signal - Signal processing on digital, neural, and kiloprocessor based architectures: Introduction to Neural Processing www.itk.ppke.hu 10/5/2011. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 8 Scope and motivation(3) • Nowthechallengeistodesignarobust,modelingarchitecturethatcanrepresent,learnandgeneralizeknowledge.Hardenough-butevolutionandbiologyprovidesusamorethanusefulpattern:themammalnervoussystemhashighrepresentationcapability,largescaleadaptation,farreachinggeneralizations,andthestructureismodular. • ThisstudyfocusesonsoftcomputingapproachestosolveITproblems,mimickingthebraininhiscomputationalparadigm. Signal processing on digital, neural, and kiloprocessor based architectures: Introduction to Neural Processing www.itk.ppke.hu 10/5/2011. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 9 Scope and motivation(4) Copying the brain? Signal processing on digital, neural, and kiloprocessor based architectures: Introduction toNeural Processing Historical Notes(1) • Linear analog filters, 20’s • Artificial neuron model, 40’s (McCulloch-Pitts, J. von Neumann); • Perceptron learning rule, 50’s (Rosenblatt); • FFT, 50’s • ADALINE, 60’s (Widrow) • The criticalreview, 70’s (Minsky) • Adaptive linear signal processing (RM, KW algorithms) , 70’s • DSPs and digital filters, 80’s Signal processing on digital, neural, and kiloprocessor based architectures: Introduction to Neural Processing Historical Notes(2) • Feed forward neural nets, 80’s (Cybenko, Hornik, Stinchcombe..) • Back propagation learning, 80’s (Sejnowsky, Grossberg) • Hopfield net, 80’s (Hopfield, Grossberg); • Self organizing feature map, 70’s -80’s (Kohonen) • CNN, 80’s-90’s (Roska, Chua) • PCA networks, 90’s (Oja) • Applications in IT, 90’s -00’s • Kiloprocessor arrays 2005 Signal processing on digital, neural, and kiloprocessor based architectures: Introduction to Neural Processing www.itk.ppke.hu 10/5/2011. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 12 Benefits of neural networks(1) • Computing power of neural networks are based:1. Parallel distributed structure; 2. Ability to learn (generalization); • Use neural network as a part of a system! • Powerful capabilities:1. Nonlinearity:aneuralnetworksisnonlinearinadistributedfashion; 2. Input-outputmapping:neuralnetworksusesparadigmcalledlearningwithateacherorsupervisedlearningbasedontrainingsamplesordesiredresponse; Signal processing on digital, neural, and kiloprocessor based architectures: Introduction to Neural Processing www.itk.ppke.hu 10/5/2011. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 13 Benefits of neural networks(2) 3. Adaptivity:Synapticweightscanbechangedinthesurroundingenvironment.Neuralnetworkscanbeeasilyretrainedwhensmallchangesappearsintheenvironment.Thissynapticweightscanbechangedinrealtimetoadaptinanonstationerenvironment.Torealizethebenefitsofadaptivity,thelearningtimeoftheneuralnetworkshouldbelongenoughtoignorespuriousdisturbances,butshortenoughtoreactimportantchangesoftheenvironment. • Inspiration from neurobiology of human brain which is summarized shortly. Signal processing on digital, neural, and kiloprocessor based architectures: Introduction to Neural Processing www.itk.ppke.hu 10/5/2011. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 14 Biological inspirations(1) Mammal brain properties • High representation capability; • Large scale adaptation; • Far reaching generalizations; • Modular structure (nerve cells, neurons); • Very robust system identification and modeling www.itk.ppke.hu 10/5/2011. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 15 Biological inspirations(2) Modeling architecture Representation Learning Generalization Robustness Modularity Solution provided by evolution and biology: MAMMAL BRAIN Signal processing on digital, neural, and kiloprocessor based architectures: Introduction to Neural Processing Signal processing on digital, neural, and kiloprocessor based architectures: Introduction to Neural Processing www.itk.ppke.hu 10/5/2011. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 16 Biological inspirations(3) Theartificialneuronsweuseinthecourseareveryprimitivein comparisontothosefoundinthebrain. Theprimaryinterestinthiscourseistostudyofartificialneural networksfromanengineeringpointofview! Thesourceofinspirationistheneurobiologyanalogy! Hebb’s Rule: Ifaninputofaneuronisrepeatedlyandpersistentlycausingtheneurontofire,ametabolicchangehappensinthesynapseofthatparticularinputtoreduceitsresistance Signal processing on digital, neural, and kiloprocessor based architectures: Introduction to Neural Processing www.itk.ppke.hu 10/5/2011. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 17 Human brain(1) Human brain as a three-stage signal processing system: Receptors Neural Network Effectors Stimulus Response Signal processing on digital, neural, and kiloprocessor based architectures: Introduction to Neural Processing www.itk.ppke.hu 10/5/2011. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 18 Human brain(2) Thepreviousfigureshowstheblockdiagramrepresentationofnervoussystem.Thebrainisacentralsystemwhichisrepresentedbytheneuralnet.Thisnervenetcontinuallyreceivesinformationandmakesexactdecisionsafterprocessing.Twotypeoftransmissionsexists: • Forward transmission • Feedback Receptors converts the stimulus into electrical signals that contain the information and it is transmitted to neural net (brain). Signal processing on digital, neural, and kiloprocessor based architectures: Introduction to Neural Processing www.itk.ppke.hu 10/5/2011. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 19 Human brain(3) Effectorsconvertelectricalimpulsesgeneratedbythebrain(afterprocessing)intoresponsesassystemoutputssuchasamovement. Theelementaryprocessingunitofthebrainistheneuron. Theprocessingspeedoftheneuronareintherangeofnanosecs,whilethetransistors(silicongates)performsinthemicrosecondrange.Neuronsareslowerthantransistors! Ontheotherhandthenumberoftheinterconnectionsbetween theneuronsareverylarge:60trillionconnections(synapses). Signal processing on digital, neural, and kiloprocessor based architectures: Introduction to Neural Processing www.itk.ppke.hu 10/5/2011. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 20 Human brain(4) Synapses: • The elementary units that controls the interactions between neurons; • Excitation or inhibition type; • Plasticity: creation or modification of existed synapses; • Axons: transmission lines which has smoother surface fewer branches and greater length (output) • Dendrites: transmission lines which has an irregular surface and more branches Signal processing on digital, neural, and kiloprocessor based architectures: Introduction to Neural Processing www.itk.ppke.hu 10/5/2011. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 21 Human brain(5) The figure shows a neuron cell. Neuron Signal processing on digital, neural, and kiloprocessor based architectures: Introduction to Neural Processing www.itk.ppke.hu 2011.10.05.. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 22 Human brain(6) Action potential (spikes): • Majorityofneuronsencodetheiroutputsasaseriesofvoltagesignals. • Thesepulsesaregeneratedclosetothenucleusandpropagatesacrosstheaxonatconstantamplitudeandspeed. • Axonshavehighelectricalresistanceandverylargecapacitance. Signal processing on digital, neural, and kiloprocessor based architectures: Introduction to Neural Processing www.itk.ppke.hu 10/5/2011. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 23 Human brain(7) Figure shows the structural organization of levels in the brain. Neural microcircuits Molecules Synapses Dendritic trees Neurons Local circuits Interregional circuits Brain Signal processing on digital, neural, and kiloprocessor based architectures: Introduction to Neural Processing www.itk.ppke.hu 10/5/2011. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 24 Network architectures(1) NNsincorporatethetwofundamentalcomponentsofbiologicalneuralnets: 1. Neurons (nodes) 2. Synapses (weights) ANN structure Signal processing on digital, neural, and kiloprocessor based architectures: Introduction to Neural Processing www.itk.ppke.hu 10/5/2011. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 25 Network architectures(2) Single neuron bio neurone node node sigmoid Signal processing on digital, neural, and kiloprocessor based architectures: Introduction to Neural Processing www.itk.ppke.hu 10/5/2011. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 26 Network architectures(3) Single neuron Figure shows the analogy between synapse and Weight. bio synapse weight Signal processing on digital, neural, and kiloprocessor based architectures: Introduction to Neural Processing www.itk.ppke.hu 10/5/2011. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 27 Network architectures(4) Single-layer feed forward networks • Figure shows a feed forward (or acyclic) network with a single layer of neurons. • In the case single-layer networks there are only one layer of neuron, which generatethe output(s). Signal processing on digital, neural, and kiloprocessor based architectures: Introduction to Neural Processing www.itk.ppke.hu 10/5/2011. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 28 Network architectures(5) Single-layer feed forward networks • Weight settings and topology determine the behavior of the network. • Given a topology what is the optimal setting? How can we find the right weights? • Single-layer neural networks use the perceptron learning rule to train the network (learning): -Requires training set (input / desired output pairs) -Error is used to adjust weights (supervised learning) -Gradient descent on error landscape Signal processing on digital, neural, and kiloprocessor based architectures: Introduction to Neural Processing ANN structure www.itk.ppke.hu 10/5/2011. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 29 Network architectures(6) Multi-layer feed forward networks • Information flow is unidirectional -Data is presented to Input layer -Passed on to Hidden Layer -Passed on to Output layer • Information is distributed • Information processing is parallel Signal processing on digital, neural, and kiloprocessor based architectures: Introduction to Neural Processing www.itk.ppke.hu 10/5/2011. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 30 Network architectures(7) Recurrent networks • Multidirectional flow of information • Memory / sense of time • Complex temporal dynamics (e.g. CPGs) • Various training methods (Hebbian, evolution) • Often better biological models than FFFNs Signal processing on digital, neural, and kiloprocessor based architectures: Introduction to Neural Processing www.itk.ppke.hu 10/5/2011. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 31 Network architectures(8) Recurrent networks • Hopfieldnetworkisaverygoodexampleofrecurrentnetworks.Weightsettingsandtopologydeterminethebehaviorofthenetwork.Givenatopologywhatistheoptimalsetting?Howcanwefindtherightweights? • HopfieldnetworksusetheHebblearningruletotrainthenetwork(tomemorize). Signal processing on digital, neural, and kiloprocessor based architectures: Introduction to Neural Processing www.itk.ppke.hu 10/5/2011. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 32 Applications(1) Recognition • Pattern recognition (e.g. face recognition in airports) • Character recognition • Handwriting: processing checks Data association • Not only identify the characters that were scanned but identify when the scanner is not working properly Optimization • Travelling salesman problem Signal processing on digital, neural, and kiloprocessor based architectures: Introduction to Neural Processing www.itk.ppke.hu 10/5/2011. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 33 Applications(2) Data Conceptualization • infer grouping relationships: e.g. extract from a database the names of those most likely to buy a particular product. Data Filtering • e.g. take the noise out of a telephone signal, signal smoothing Planning • Unknown environments • Sensor data is noisy • Fairly new approach to planning Digital-and Neural Based Signal Processing & Kiloprocessor Arrays: Introduction to Neural Processing 10/5/2011. TÁMOP –4.1.2-08/2/A/KMR-2009-0006 34 Summary • Fundamentalissues:artificialneuralnetworks,learning,generalization,humanbrain • Collectionofalgorithmstosolvehighlycomplexproblemsinreal-time(inthefieldofIT)byusingclassicalmethodsandnovelcomputationalparadigmsroutedinbiology. • NN’shavesimpleprinciples,verycomplexbehaviors,learningcapability,andhighlyparallelimplementationpossibility.• Informationisprocessedmuchmorelikethebrainthanaserialcomputer. • Neuralnetworksareusedforprediction,classification,dataassociation,dataconceptualization,filteringandplanning. Nextlecture:Signalprocessingbyasingleneuron