Cognitive Neuroscience for AI Developer Summary PDF

Summary Cognitive Neuroscience for AI Developer By Anonymes Brett  Table of Contents Introduction...................................................................................................................1 1.1...

Summary Cognitive Neuroscience for AI Developer By Anonymes Brett  Table of Contents Introduction...................................................................................................................1 1.1 What is Cognitive Science?.........................................................................1 1.2 The cognitive revolution...............................................................................3 Philosophical Approach................................................................................................6 1.3 The Mind Body Problem: What is mind?.....................................................6 Psychological Approach...............................................................................................9 1.4 Psychology and the scientific method......................................................... 9 1.5 Intelligence tests........................................................................................10 1.6 The psychological theories........................................................................11 Neurons and Glia........................................................................................................19 1.7 The Neuron................................................................................................19 1.8 Microglia....................................................................................................23 1.9 Astrocytes..................................................................................................23 1.10 Neural computation................................................................................... 24 Neural Plasticy............................................................................................................28 1.11 Differentiation and Survival of Nerve Cells – and AI use-cases................29 1.12 Synaptic plasticity rules.............................................................................31 Measuring neural activity and connectivity.................................................................33 1.13 Electrophysical methods............................................................................33 1.14 Optical approaches....................................................................................35 1.15 Why this is important to AI Developer?..................................................... 36 1.16 Why is it important to know how the brain is connected?..........................36 Lesioning....................................................................................................................37 1.17 Famous examples of lesion studies and consequences...........................38 1.18 Lesion studies and brain surgeries against epilepsy.................................38 1.19 Non-invasive lesion studies.......................................................................39 1.20 Lesion studies in AI................................................................................... 39 Imaging Techniques...................................................................................................41 1.21 Brain structure...........................................................................................41 1.22 Brain processes / functions....................................................................... 41 1.23 Conclusion.................................................................................................42 Brain Structure and Functional Systems....................................................................43 1.24 Brainstem.................................................................................................. 43 1.25 Cerebellum................................................................................................43  1.26 Diencephalon.............................................................................................44 1.27 Telencephalon (part of cerebrum).............................................................44 1.28 Basal ganglia............................................................................................44 1.29 The Cerebral Cortex..................................................................................45 1.30 Lateralization of Brain Function.................................................................48 Sensation and Perception.......................................................................................... 49 1.31 The Auditory System - Hearing................................................................. 49 1.32 The Vestibular System - Balancing........................................................... 51 1.33 The Olfactory System - Smelling...............................................................51 1.34 The Gustatory System - Tasting................................................................52 Visual System.............................................................................................................53 1.35 The receptive field.....................................................................................55 1.36 The Visual Cortex V1.................................................................................57 1.37 Object recognition......................................................................................57 1.38 Face Recognition.......................................................................................58 Motor system..............................................................................................................59 1.39 Elicit behaviour..........................................................................................59 1.40 Brain regions............................................................................................. 59 1.41 Internal models..........................................................................................61 Language, Attention, Saliency....................................................................................62 1.42 Lesion studies and Apasia.........................................................................63 1.43 Language Comprehension........................................................................63 1.44 Language Production................................................................................ 65 1.45 Attention.................................................................................................... 66 1.46 Fundamental Recent Developments “Language and Attention”.............. 66 Memory, Free Will, GPT 3/4.......................................................................................68 1.47 Anatomy of memory.................................................................................. 68 1.48 Forms of Memory that are Short Term......................................................69 1.49 Forms of Memory that are Long Term.......................................................70 1.50 Free Will and Consciousness....................................................................71 Introduction This is a quick Introduction into the spanning Framework that feeds this Summary actually is! So let’s start and answer the first questions: 1.1 What is Cognitive Science? To bring it on the point. Cognitive Science is the … Scientific, interdisciplinary study of the mind. So, what is Mind? “[…] the complex of faculties involved in perceiving, remembering, considering, evaluating, and deciding. Mind is in some sense reflected in such occurrences as sensations, perceptions, emotions, memory, desires, various types of reasoning, motives, choices, traits of personality, and the unconscious.” And what is Cognition? “higher mental processes such as thinking, perceiving, imagining, speaking, acting, planning” The study of these processes is achieved in modern Cognitive Science, by using the Scientific method. This consists of: · test hypothesis with experiment → update hypothesis → new Experiment (iterative process) One used principle is the one of Occams razor: „Given two explanations of the data, all other things being equal, the simpler explanation is preferable.” This principle is alive in the science of machine learning, as the goal is to discover the simplest hypothesis, that is consistent with the sample data. Now we discussed the “Scientific” and “study of the mind” part of the definition, so we are left with the Multi-disciplinary perspective: 1 The questions Cognitive Science wants to answer are following: How does the human mind work? How does cognition work? How is cognition implemented in the brain? How can cognition be implemented in machines? But these are some of the hardest scientific problems, as the: 1. Brain is hard to observe, measure and manipulate 2. Brain is most complex entity in the known universe An opinion paper by Brown, J. W. “The tale of the neuroscientist and the computer”, In this paper he mocks the different disciplines and how little each discipline knows about there respected field of work in context to the brain. Moral of the story/paper: · Cognitive neuroscience is still in an early stage phase · We need a mechanistic theory to understand cognition in the brain · We have to develop computer models that produce testable hypotheses · We need a multi-disciplinary approach · Neuroscience alone is not enough 2 1.2 The cognitive revolution In the 1950 something happened. Psychology and linguistics were redefining themselves, with a backlash against behaviourism. Computer science and neuroscience came up. As well as personal computer novel brain imaging techniques boosted the development. And in 1960, the researchers started to define and work in this new interdisciplinary field, which at that time had different names, as information- processing psychology, cognitive studies, cognitive science. Cognitive Neuroscience had two views on there problem, the Classical/ Traditional and the Connectionist view, whose switch is in parallel with the process from symbolic AI to Deep Learning. 1.2.1 The classical/ traditional view An information processing System has to represent something and has to do computations on these representations. · Representations are symbolic · Computation in sequential steps 1.2.1.1 What are Representations? Representation is: “something stands for something else”, there are 4 types: 1. Concepts (stands for entity) → „apple“ 2. Propositions (statements about the world) → „Mary has black hair.“ 3. Rules (relationship between propositions) → „If it is raining, I will bring my umbrella. “ 4. Analogies (comparisons between situations) → „Life is a roller coaster.” The traditional opinion was, that representations are symbolic. Symbolic means that a symbol is a surrogate and refers to its referent! Symbols can be assembled into physical symbol system (formal logical system), that means, that these are combined to expressions. These expressions can then be manipulated trough formal processes, to create new expressions. Formal logical systems can allow for intelligence and intelligent behavior (Physical symbol system hypothesis (Newell & Simon, 1976)) 3 Hypothesis is often criticized: · computers use symbols with no meaning as symbols are not connected to the environment (grounding problem) · Computers do not perceive their environment (no bodies) → thus symbols have no meaning Example of formal logic Symbols: all, mammals, …. Expression: all and only mammals nurse their young Processes: rules of deduction derive new, true expressions from known expressions Expression 1: all and only mammals nurse their young Expression 2: whales nurse their young New expression: whales are mammals → Newell and Simon 1976 → intelligence They believed, that everything that was capable of that is intelligent. This view was useful, to develop experts systems, like MYCIN (Shortliffe), as the first expert system, which could diagnose blood infections and meningitis, with a set of if-then rules. 1.2.1.2 Computations We looked at the representations, but what about the Computations? The mind performs computations on representations. e.g.: language: putting a verb into past tense math: adding two numbers etc. → endless list Define broad categories of mental operations according to: Type of operation Type of information that is processed According to the Tri-level hypothesis (Marr, 1982), any information processing can be described at 3 different levels: 1. Computational level: (most abstract) Which problem is the system trying to solve? e.g. to sort a set of numbers. 2. Algorithmic level: How does the system solve this problem? Algorithm? Procedure? e.g. bubble sort, binary tree sort, quicksort 3. Implementational level: How is this algorithm implemented? Code? Physical? e.g. python code, assembler, logical gates, transistors, electron flow 4 1.2.2 The Connectionist view The connectionist view is the more modern interpretation. It describes Representations as: · activation patterns spread over a neural network (Brain) And different to the sequential step theory of the classical view, we now state, that computations are parallel in the network. 5 Philosophical Approach Philosophy is the search for wisdom and knowledge, and is the oldest discipline, reaching back to the ancient Greeks. It plays a critical role in cognitive science, not by producing results as they are theoretical not experimental, but by: · Defining problems · Criticizing models · Suggesting areas for future research · Free to evaluate other disciplines. · Find criteria for intelligence etc. To achieve this, the main method is reasoning a non-scientific method, which can be separated into: · Deductive reasoning: applying rules of logic to statements, in order to derive new statements („College students learn three hours a night“. „Mary is a college student“. → „Mary learns three hours a night“) · Inductive reasoning: draw conclusions based on several observations of specific instances of the world („Whiskers the cat has four legs.“ „Scruffy the cat has four legs.“ →“Cats have four legs“) There are several branches of philosophy, two of them are shortly introduced here: 1. Metaphysics It ask “What is the nature of reality?” · What are the first causes of things? · What is the nature of being? One topic that arises from that is the mind body problem. 2. Epistemology It is the study of knowledge: · What is knowledge? · How is knowledge represented in the mind? · How do we come to acquire knowledge? · … 1.3 The Mind Body Problem: What is mind? How are psychological and mental processes related to physical properties? · Brain: material and physical, measurable · Mind: subjective conscious experiences Mind as non-physical entity inhabiting the brain „Ghost in the machine“ 6 Two fundamental questions: · Is the mind physical or something else? · What is the causal relationship between mind and brain? There evolved three different Theories trying to solve the Problem. 1.3.1 Monism The statement there is only one kind of state or substance in the universe. Aristotle said, mind and body are the same as form and matter! Analogy: “Different shapes of clay are different physical states of brain, no non-physical or spiritual substance.” Monism, can be divided into: · Idealism: The complete universe is mental o e.g. Simulation hypothesis · Materialism (physicalism): All things are made of atoms (Democritus ca. 460- 370 BCE) o mind is the brain o mental states are physical states of the brain 1.3.2 Dualism Mental and physical substances are possible Plato (427-347 BC): mind and body exist in two separate worlds · Mind: ideal world of forms, immaterial, eternal e.g. idea of an ideal circle · Body: material world, extended, perishable e.g. concrete, physical circles 7 Dualism can be divided into: · Substance Dualism (Descartes 1596-1650): o There exist mental and physical substances o Physical substances: world is made of atoms o Mental substances: Unknown o Theory: Minds can do e.g. pattern recognition. No physical substance can do pattern recognition. → Minds are not physical. o Criticism:  How do mental and physical substances interact?  Computers can do pattern recognition and much more! · Property Dualism: o Mind and body are of the same substance but have different properties o Mental states are non-physical properties of the brain Critics on Dualism Dualism violates the principle of Occams razor: two different worlds that interact are needed. Not the simplest explanation! Another problem: Brain damage changes mental states Computer can do a lot of tasks assumed to be impossible (e.g. ChatGPT can write novels) 1.3.3 Functionalism Mind could be implemented in any physical system, artificial or natural, capable of supporting the appropriate computations The same mental state could be realized in quite different ways in two separate physical systems. → Concept of multiple realizability This could generate ethical issues: If mind can be realized in different systems at which point aliens or computers get human rights. Conclusion: Philosophy Allows to ask much broader questions than those of other disciplines → Shows the „bigger picture“ Gives key insights into the relationships between different research areas Plays a very important role in the interdisciplinary endeavor of Cognitive Science Non-empirical approach, in contrast to the scientific method Concepts validated based on logical reasoning and argument Philosophy is better suited to ask questions than to provide answers Close 2-way collaboration between philosophy and science is required 8 Psychological Approach We are coming nearer and nearer to put our feet in current Cognitive Neuroscience, but to build it up correctly, we first need to look at the psychological approaches leading up to the Cognitive Revolution in the 1950 that lead to the creation of Cognitive Neuroscience as a research field. So, what is psychology? Compared to Philosophy it is a rather young discipline, with its roots in the late 19th century. And it is the scientific study of mind and behaviour. Internal mental events: perception, reasoning, language, visual imagery External events: behavior, speech Scientific Method (only modern Psychology, not like Philosophy) There were many competing theorise, that influenced each other or generated counterreactions. The study even of these old theories has a big impact on AI research, e.g. to measure intelligence of AI systems. Voluntarism Structuralism Functionalism Gestalt theory Psychoanalytic psychology Behaviorism 1.4 Psychology and the scientific method Early psychologist relied on introspection and phenomenology, which are problematic methods. The main methods of modern psychology are questionnaires, surveys, case study analysis, recording of behavior. Scientific method (Hypothetic-Deductive-Approach): Experiments to test hypotheses Hypotheses testing to construct or adjust theories Theories to generate new hypotheses Experiments: Independent variable: manipulated by experimenter Dependent variable: what is measured or observed Minimum of two conditions: experimental group vs control group 9 Just by sticking to this graph is not enough, as the experimenter really needs to make sure, only the independent variable should be different between both groups. And should therefore randomize and counterbalance, if not done correctly this can lead to systematic errors, and wrong research results. Beside that the researcher also needs to conduct statistical test/ hypothesis testing, like the t-test) to used to find out if an effect might be caused just by randomness or not, p_value < 5% defined as significant. It is also helpful to calculate the number of minimal subjects you need to test on in order to generate statistically significant results. Major errors in science, that have an effect on the current replicability crisis, as 36%- 68% of published results cannot be reproduced. P-Hacking: Selection of data (e.g. outlier removal) and statistical tests in order to make non-significant results significant HARKing: Hypothesizing after results are known (post-hoc hypothesis) 1.5 Intelligence tests Psychology was not only interested in understanding the mind but started early to measure it. Intelligence test were developed over a century ago, Alfred Binet developed methods to measure intelligence to improve France‘s education system. And in 1920 Binet’s test were adapted by Lewis Terman at Stanford university to measure intelligence of students, which is known as the Stanford-Binet Intelligence Quotient (IQ) and has been influential ever since. One big problem with the test, was the Cultural bias in the test: e.g. task to name different coins → advantage for rich people. That’s why IQ tests were reworked several times, in order to reduce cultural bias. Even with no more bias in the test there is still critiques regarding IQ-Test, as they assume that: a. general intelligence is innate → not true → In twin studies it was shown that IQ can improve when children are moved to intellectual supportive environment b. intelligence can be measured with one number → there are different aspect of intelligence (Howard Gardner (2011)): a. linguistic intelligence, musical intelligence, logical-mathematical intelligence, bodily-kinesthetic intelligence (athletes), interpersonal intelligence (sales persons), intra personal intelligence (self knowledge) … 10 1.6 The psychological theories In this chapter, we will be going through the psychological theorise, based on the timeline, and state their characteristics, main supporter as well as critiques. 1.6.1 Voluntarism The main idea was, that the mind consists of mental elements assembled into higher cognitive components through the power of will (= voluntary effort of the mind). This means, that it takes the voluntary effort of the mind to assemble the mental elements to a conscious whole. The goal was to define a Periodic table of mental elements, and it was hugely inspired by chemistry. As in 1896, Dimitri Mendeleev proposed the periodic table of elements. The method they used was Introspection („inward looking“): Look inward to identify mental elements Presented students colored objects and asked for their experiences Wundt wanted to systemize introspection: students were put in state of attention and experiments were repeated several times The main driver for Voluntarism was Willhelm Wund (1832-1920), and the goal was to study the immediate experience of consciousness, as he divided conscious experiences into two types: Immediate experience: direct awareness of something (we see a red rose) → Wundt’s focus Mediate experience: mental reflection (mental reflections about an object, e.g. tell someone about rose) Voluntarism also characterized all feelings by three dimensions: 1. Pleasure – Displeasure (certain rhythm) 2. Tension – Relaxion (waiting for click) 3. Excitement – Depression (change of tempo) Wundt defined these dimensions by playing his students the metronome and having them introspect. 1.6.1.1 Summary and Critique of Voluntarism Voluntarism was beneficial as it was the first (scientific) attempt to studying the mind. However: Introspection is a problematic method, as o mental experiences change over time o act of introspecting changes experiences And Wundt was never able to find a list of mental elements comparable to the periodic table of elements → far too long list. 11 1.6.2 Structuralism – What the mind is Shares some ideas with voluntarism and hat the same subject matter, the Conscious experience. However, different to Voluntarism, Structuralism defines the Mind as passive agent, with mental elements combining according to mechanistic laws. Titchener (1867-1927) also wanted to avoid the Stimulus Error of “confusing true experience with description of that object based on language and previous experiences”, and therefore believed, that only well-trained observers can accurately introspect. In hindsight was the stimulus error a right observation, but to still rely on introspection a misconception. Structuralism defined three goals of psychology: Describe consciousness in terms of most basic components Discover the laws by which these components associate Understand relation between elements and psychological conditions 1.6.2.1 Mind as passive agent Structuralism states, that the mind is a passive mechanism or substrate within which elements are combined according to a set of laws (mind is a reagent, sometimes participants were called reagents) · A reagent is a substance added to a mixture to produce a chemical reaction. Titchener described a total of 44,000 sensation elements described, and therefore also had the same problem as Voluntarism. According to Titchener sensations can be characterized by four attributes: quality, intensity, duration, clearness (sensation one pays attention to), later also extensity (extent to which sensation fills space → pressure from pencil vs. chair bottom) 1.6.2.2 Summary and Critique of Structuralism Same points of criticism as for Voluntarism, with introspection being a problematic method. However the difference to Voluntarism was the already mentioned the effort of refinement of the experimental procedures. Which included the training of study participants, which however increased the biases responses even more. · Wrong solution for the right problem! He also over-emphasized mental elements and ignore holistic perception. 12 1.6.3 Functionalism – What the mind does Functionalism was a counterreaction to the Voluntarism and Structuralism Theories, which tried to catalogue the “elements” of the mind. In opposition to this the functionalism focuses on what the mind can do, not on mental elements. · Mental processes and functions instead of mental elements. James defined the mind as a stream of consciousness: · mind is a process undergoing continuous change And in that classifies thoughts into: · Substantive thought occurs when mind slows down (focus attention, in contrast to transitive thought) · Transitive thoughts: less focused form of thinking Three major themes of functionalism (proposed by Rowland Angell, 1907): · Mental operations (how mental process operates, what is accomplished under which conditions they occur) · Fundamental utilities of consciousness (role of consciousness for survival of the organism) · Psychophysical relations (relation of the psychological mind and the physical body) Functionalism is strongly influenced by Darwin‘s theory of natural selection, how did the mind develop under evolutionary pressure. And therefore it was the precursor of Evolutionary Psychology: “the study of behavior, thought, and feeling as viewed through the lens of evolutionary biology” 1.6.3.1 Summary and Critique of Functionalism Functionalism was a huge step forward as they used a wide variety of methods: e.g. questionnaires, objective behavioral descriptions, but also introspection. There was no clear definition of the word function: 1. function refers to an process itself (perception and memory) 2. function refers to the usefulness of the process (e.g. how does memory contribute to survive) Another critique is, that it is too practical, and often too focused on usefulness of function (Evolutionary Influence). Structuralism vs. Functionalism, could be defined as the fight which is better basic (Structuralism & Voluntarism) or applied science (Functionalism). 13 1.6.4 Gestalt psychology - „The whole is greater than the sum of its parts“ As functionalism, Gestalt psychology was also a counterreaction to structuralism. With its main message stating: „The whole is greater than the sum of its parts.“ In that context, the “Gestalt” is the integrated whole. In analogy to physics, and countering functionalism, they say, that the Conscious wholes cannot be reduced into parts. As mental parts combined into wholes is the same as particles ordered in a field of force. According to the analogy, you can’t identify the pattern metal particles form in a magnetic field, just by looking at the particles one by one. And according to Gestalt psychology it is the same with “mental parts”. The main method was Phenomenology : · subjective experience, observers describe subjective experience · in contrast to introspection phenomenology focuses on immediate subjective perception Greatest contribution of Gestalt psychology in perception and learning (looser in methodology e.g. observed animals finding solutions) 1.6.4.1 Visual perception – Max Wertheimer Gestalt principles of perceptual organization, formulated by Max Wertheimer, describe the ways in visual parts group to form objects. e.g.: You don’t see a packman on the “Pragnanz” Image. 1.6.4.2 Summary and Critique of Gestalt Psychology · Phenomenology approach lacks scientific rigor · Data was not gained in experimental settings so there was no statistical analysis · Principles of perceptual organization are just descriptive but do not provide explanations 14 1.6.5 Psychoanalytic psychology The famous psychoanalytic psychology was introduced by Sigmund Freud. And states: The mind is made up of „miniature minds“ that compete with each other for control of behaviour. It states that there is a Three-tiered system of consciousness: · Conscious mind (contains thoughts and feelings which we are aware of, home address) · Pre-conscious mind (thoughts we can bring to consciousness with efforts, recall what one did last Friday) · Unconscious mind (thoughts and experiences that can never be brought to consciousness, childhood memories) Freud also defined three other mental structures with different operation modes: These three mental structures, are linked to the three-tired system of consciousness: · Id is completely unconscious and powerful · Super-ego tries to suppress the needs of the id · Ego tries to balance needs of super ego and id · If ego fails to satisfy one → anxiety · Ego constructs defense mechanism to shield itself against anxiety: o Repression: banishing of anxiety arousing thoughts from consciousness o Sublimation: transform of unacceptable impulses in socially valued motivations Freuds model of the mind as machine with interacting parts → he used many terms from mechanics and electronics. 15 1.6.5.1 Summary and critique of Psychoanalytic psychology The approach stimulated further research in the area of unconscious processes and inspired generations in clinical practice. But there is still critique: · Freud overestimated parental and early childhood influence · Scientific shortcomings: theory not based on objective observations but on notes about Freud‘s patients · Freud‘s ideas have no predictive power 1.6.6 Behaviorism – The mind as a black box Behaviorism was the last theory bevor the Cognitive Revolution in the 1950th. And was mainly driven by: They state that the mind is too complex to be studied scientifically, which was true at the time, and therefore behaviorists thought that the scientific method cannot be applied to the mind. This leaded in behavioral experiments. Behaviourism was highly influenced by animal research and Humans were lumped in the same category as animals and it was therefore more general/ natural science. The behaviourists rejected introspection. They stated that the most important stimuli are reward and punishment. To put this in a nutshell, the research strategy of behaviourism is that you have a stimulus that the mind as a “Black Box” and a Result (Behaviour). 16 Two famous experiments are showing the two different types of conditioning. · Pavlovian conditioning/ classical conditioning Ivan Pavlov conducted an experiment, in which he discovered the classical conditioning, in which a reaction is conditioned response is generated by a conditioned stimulus. · The Skinner Box and the operant conditioning 1.6.6.1 Behaviorism – Summary and Criticism The strength was, that they rely on completely objective science of behavior and rigorous scientific methods. Behaviorism was the dominant paradigm until the 1960s and therefore the rise of cognitive psychology. But Edward Chance Tolman (1886-1959) found out that reward and punishment is not necessary for learning and by that he challenged classical doctrine of behaviourism. He underlined that with the finding, that rats can navigate through maze after exploration, with no reward or punishment. → latent learning 17  Conclusion: Psychological approach Historical: many different theoretical positions and schools of thought Psychology is the first discipline to systematically apply experiments to study mind Initially lack of precision and reliance on non-scientific methods Initially no overarching theory or framework Today o Cognitive approach and information-processing perspective  Cognitive Psychology and Cognitive Neuroscience 18 Neurons and Glia Now that we have the history behind us, let’s concentrate on what we all hopefully have inside us. The Brain. And what has your and my brain, as well the one from your pet in common? They all consist of CELLS: For the human brain it could look like this: And let`s check them out one by one. 1.7 The Neuron The Neurons are the major computational unit. They consist of different parts: Blue: A Neuron has ≥ 1 dendrites, which receive the input, and have multiple branches each. They receive input and bring it to the soma/ cell body. Green: The soma or the cell body, is the component in which the computation happens. A main component is the Nucleus, that holds the DNA. Red: Every Neuron has one Axon, which is the component through which the AP travel towards the presynaptic Cell. We will later look at this process in more detail. 19 There are multiple types of Neurons, these are the four, presented in the lecture: · Bipolar cell of the retina: has only one base dendrite with multiple branches on top. · Ganglion cell to dorsal root is a specialized for relaying sensory information from the peripheral nervous system to the central nervous system via the dorsal root. The position of the cell body is special, as it is not the connection between the dendrites and the axon. · Motor neuron of the spinal cord is specialised for relaying efferent motor signals. · The Purkinje cell is a type of neuron found in the cerebellum that plays a crucial role in coordinating motor movements and regulating motor learning and coordination within the central nervous system. Special are the planar, dendritic trees. 1.7.1 The connection between two neurons: the synapse The connection between to neurons is a synapse, which consists of a Presynaptic terminal and a postsynaptic dendrite, as well as the synaptic gap (20nm). Two neurons are coupled chemically, which brings extra complexity, compared to direct electrical coupling. But the reasoning is, that neurotransmitters allow for adjustments. It makes it also possible, to excite and inhibit with the same input signal, based on the neurotransmitter. First, the Voltage gated calcium channels open as soon as an AP is recognised. That leads to the flow of CA2+ inside the presynaptic terminal. This triggers the vesicles containing the neurotransmitter, to bind to the cell membrane and release the Neurotransmitter in the synaptic gap. These bind to the receptor channel and, then open, which leads to a flow of Na+ 20 inside the postsynaptic dendrite. Which results in this case in an excitatory postsynaptic potential. 1.7.2 The theoretical neuron Warren McCulloch and Walter Pitts propose the McCulloch-Pitts neuron in 1943. The goal was to generate a simplified theoretical neuron. They assume that there are no inhibitory inputs, just binary with on (1) and off (0). They defined that the neuron has i-dendrites as the input and a soma with a threshold (Theta). The computation is: 𝑖𝑓 𝑥𝑖 ≥ 𝛩 𝑡ℎ𝑒𝑛 𝑦 = 1 𝑖 𝛩 = 1 → 𝐿𝑜𝑔𝑖𝑐𝑎𝑙 𝑂𝑅 With two inputs: 𝛩 = 2 → 𝐿𝑜𝑔𝑖𝑐𝑎𝑙 𝐴𝑁𝐷 In 1950 shaped Frank Rosenblatt the concept of the theoretical neuron significantly. As he proposed the perceptron, which not only receives binary input but also weights, which enabled to inhibit and excite signals. With the publication of the book perceptron’s, which pointed out the XOR affair, as it was not possible to generate a XOR computation using a perceptron. This XOR affair resulted in a AI Winter, as XOR is a necessary computation. 21 This issue was solved, by realizing that one can combine multiple perceptron’s. By that it is possible to modulate XOR. These are called Multi Level Perceptron’s (MLP). 1.7.3 The Axon and the diverse ways of conduction. Most Axons, especially these who need to conduct the signal as fast as possible are insulated with myelin sheets, which in between the sheath has small gaps, which are called the node of Ranvier. Based on the type of nervous system a neuron is in, there are different cell types realising these insulations: What both of the shown Axons have in common, is that they can “use” the salatory conduction. This means, that the signal “jumps” from one Node of Ranvier to the next. The Nodes of Ranvier are highly sensitive and if they sense a AP in the previous Node, they open there ion channels, and by that the signal jumps. Peripheral nerve fibers are grouped based on the diameter, signal conduction velocity and myelination state. The A group have larger diameter, high conduction velocity, and are myelinated. The B group fibers are myelinated with a small diameter and have a low conduction velocity. The lack of myelin in the C group is the primary cause of their slow conduction velocity. 22 A-delta and C fibers both contribute to the detection of diverse painful stimuli. Because of their higher conduction velocity, A-delta fibers are responsible for the sensation of a sharp, initial pain and respond to a weaker intensity of stimulus. These nerve fibers are associated with acute pain and therefore constitute the afferent portion of the reflex arc that results in pulling away from noxious stimuli. An example is the retraction or your hand from a hot stove. Slowly conducting, unmyelinated C fibers, by contrast, carry slow, longer-lasting pain sensations. 1.8 Microglia Microglia form 10-15 % of the cells in the brain. And are responsible, for maintaining brain activity, terminating a Neurons live, if that does not behave the way it should as well as, taking away the “garbage” (debris and plaques). In Parkinson the plaques can’t be removed, which leads to the neurodegenerative disease. Microglia are resident macrophage cells, and the first and main active immune defence in the CNS. As I said, they are scavenger for plaques and debris. And exists in 3 forms. 1.8.1 Building ANN with glia 1.9 Astrocytes Astrocytes help form the physical structure of the brain and are thought to play a number of active roles, including the secretion or absorption of neural transmitters and maintenance of the blood–brain barrier. They connect to capillaries to sense the amount of nutrients for the neurons (Glucose). The same cell could also connect to different places of the neuron, like, soma, dendrites, Ranvier, synapses. 23 1.9.1 The tripartite synapse This last-mentioned connection point is especially important, as it is part of a system called Tripartite synapse. In these Astrocytes regulate the glutamate biosynthesis. 1.10 Neural computation We spoke all the time about the Action Potential (AP) but never discussed how that actually works. Every Neuron has a resting (membrane) potential in the roam of -60 to -90 mV. It is calculated by: Vm = Vin - Vout. The membrane is like a resistor and a capacitor. Based on the net flow of Ions, this resting state can change, and either depolarize or hyperpolarize. The ion concentrations are highly different between the Extracellular side and the Cytoplasmic side. For example, the potassium (K+) concentration on the cytoplasmic side is way higher than in the Extracellular side. This concentration gradient drives K+ out of the cell. Also relevant are: Sodium (Na+) and Chloride (Cl-) together form salt. For the potassium to, actually drive out of the cell, we need channels. But at the potacium based electrical potential, the electrical potential difference drives K+ into the cell, forming an equilibrium (outwards and inwards flow is the same. 24 To compute this electro statical potential (E) for a given Ion (x), using the Nernst Equation. With: R = gas constant → 8,314 𝐾⋅𝑚𝑜𝑙 𝐽 T = temperature (Kelvin K , at 25°C = 298.15 K) Z = valence of the ion F = Faraday constant (coulomb per mole (C/mol)) Therefore: Ex = J/C = mV And with the constants set in: For potassium (K+) z = +1 and according to the concentration inside and outside the squid axon (table): 1.10.1 Biology explained as electrical circuit. Every involved Ion has a conductance gx, and a statical potential Batery (Ex). The membrane itself wors as a capacitor. You can also model active Pumps, like you see on the image. Since the membrane potential is constant, ther is no net current through the three sets of ion channels: · Ik+ICl+INa=0 25 1.10.1.1 The action potential and its modulation The Action potential can be reconstructed from the properties of sodium and potassium channels. Hodgkin and Huxley were able to measure the Na+ and K+ currents over the entire voltage extent of the action potential. They found out, that Na+ and K+ currents vary as a graded function of the membrane potential. As the membrane voltage is more positive, the: · K+ outward current becomes larger. · Na+ inward current also becomes larger (drive towards depolarisation as equilibrium potential is positive (+55 mV)) · As the voltage becomes more and more positive, the Na+ current declines, and is zero at a membrane potential of +55 mV = equilibrium potential. The process how these gated channels work is shown her: Hodgkin and Huxley explained this by a simple modelm in wich the size of the Na+ and K+ currents is determined by two factors: 1. Magnitude of the Na+ and K+ conductance, gNa or gK, wich reflects the number of open channels for each Ion, at any instant. (Leitfähigkeit) 2. Electrochemical driving force on Na+ ions (Vm – ENa) or K+ (Vm – EK) Thus, the model for e.g. K (potassium) is: Together with Kirchhoff’s law, which models the conservation of electrical charge, Hodgkin and Huxlex found three dynamic variables: · n = depends on potassium (K) · m & h = depend on sodium (Na) Observed Current I = Kirchhoffs law + conductances x variables + leak current (error) To solve this you need to solve the partial differential equations. 26 The drawback of Hodgkin-Huxley, but for a single neuron you need to estimate 20 parameters. That’s the reason why with something like with leaky Integrate-and-Fire-Models, you can ignore the Voltage. Neural coding: Spikes as a rate code= higher Frequency the higher is the activation. Different encodings: 27 Neural Plasticy Let’s start at the beginning, and answer, how our brain emerges. The vertebrate embryo arises from the fertilized egg. Cell division initially form a ball of cells, called the morula, which then hollows out to form the blastula. Next, infoldings and growth generate the gastrula, a structure with polarity and three layers of cells-the endoderm, mesoderm, and ectoderm. Most of the ectoderm gives rise to the skin, but a narrow central strip flattens out to become the neural plate. It is from the neural plate that the central and peripheral nervous system arise. Soon after the neural plate forms, it begins to invaginate, forming the neural groove. The folds then deepen and eventually separate from the rest of the ectoderm to form the neural tube. This neural tube than develops further in sequential stages. Early anteroposterior patterning signals establish distinct transcription factor domains and define the position of the midbrain-hindbrain boundary (MHB) region. 28 Hox genes – Motor neurons The anteroposterior profile of Hox gene expression determines the subtype of motor neurons in the hindbrain (Pons, Cerebellum) and the spinal cord. Different Hox proteins are expressed in discrete but partially overlapping rostrocaudal domains of the hindbrain and the spinal cord. The position of Hox genes on the four mammalian chromosomal clusters roughly corresponds to their domain of expression along the anteroposterior axis of the neural tube. The clustered organization of Hox genes is conserved from flies to vertebrates. 1.11 Differentiation and Survival of Nerve Cells – and AI use-cases The brain cells have a common ancestor. A neuron matures in different steps, a very important step is prunin, as in the spine formation the neuron crates a lot of spines /branches and during pruning just keep what is important for the task. Over the lifetime, after birth there is a wiring phase (new synapses are build), and in adolescence there is a re-wiring phase (net number stay the same). This pruning is also a studied phenomenon in deep neural networks. There are two instances that can be pruned: · Pruning synapses → making network sparse · Pruning neurons → making network dense The human brain starts to prune with the age of 14 and stabilizes as adults. 29 What we stressed, is that the brain is dynamic, during growth as well as re- organisation. This is called neural plasticity. Let’s begin at the start, how can we change the strength of a synapse? How do we change the: · Weights: o Hebbian learning o Delta Rule · Threshold: o Learnable w/ SGD (Stochastic Gradient Decent) · You could also “learn” parts of the activation function. For example a parametric ReLU A connection of ANN and Genetics is the NeuroEvolution of Augmenting Topologies (NEAT). With NEAT there is a Genome, that has the information how the neural circuit is build up. NEAT is Evolutionary, you have multiple candidate networks, see how they perform on a task, and come up with a offspring, etc. The difference between NEAT and the human brain: 30 1.12 Synaptic plasticity rules „Neurons that fire together, wire together.“ Only if they fire together then there is a connection. So the weight depends of the activity. This implies that there are different forms of plasticity, this is called spike timing dependent plasticity (STDP): · Structural plasticity · Synaptic plasticity 1.12.1 The math of synaptic plasticity- Hebb’s rule The activation of v can be described by the multiplication of the weight w and the activation u. → 𝑣 = 𝑤 ⋅ 𝑢 The basic Hebb’s rule says that one wants to change the weight in a time dependant context. As you want to learn something across time. The calculation leads to a Correlation matrix, that’s why Hebb’s rule is also called Correlation-based plasticity rule. And also allows for LTP (long term potentiation). To allow for LTD and more stability. To achieve this, you can subtract a threshold either of the postsynaptic (v) or the presynaptic (u). To tackle the instability process, the BCM rule is used, and it introduces a time dynamic threshold 𝜃 𝑣: 31 You can also stabilize weights through postsynaptic activity, with synaptic normalization. 1.12.2 Supervised and Unsupervised learning Ojas rule is a form how to compute the PCA, and thereby is a form of unsupervised learning. · Unsupervised learning → no target emposed · Supervised learning → target emposed The already mentioned Delta rule, is based on supervised learning: The Hopfield Network can store and recouple and information, like images, with the downside of bad scaling. 32 Measuring neural activity and connectivity We have already discussed, how the action potential etc. forms. But how can you measure this for humans. Therefore, we will divide this into: · Electrophysical · Optical Methods 1.13 Electrophysical methods There are multiple method, that each have different use cases and characteristics: There are multiple ways to measure a single channel/Cell. You use glass capillaries with a diameter of around 2µm, and with a robot/stativ, access a single neuron, based on what the researchers want to measure, there are multiple ways to manipulate the structure of the measured part. 33 If one for example measures single channels, he could manipulate the voltage and measure the current, which is for one channel in the roam of:  5 pA, that is 5 * 10-12 A And only by taking the average over multiple channels a current like we saw while discussing the AP can be detected. To identify functional circuitry, you need paired recordings. The intracellular Measurments, can be also conducted by Multielectrodes, that enable to take measurements along there whole length that is inside the brain. By that reaserchers can measure single cells. Electrocorticography (ECoG), a type of intracranial electroencephalography (iEEG), is a type of electrophysiological monitoring that uses electrodes placed directly on the exposed surface of the brain to record electrical activity from the cerebral cortex, not of single cells. The (EEG) electroencephalography is not invasive, as the electrodes a mounted on the skin on the scull. Thereby the Electrodes are usually placed along the 10-20 System, to map the brain activity. Another method is the Magnetoencephalography (MEG), which introduces a magnetic field in the brain and by that can measure the orthogonal currents. And can reconstruct topoplots that show the regions with brain activity. 1.13.1 The EEG Signal and AI implementations EEG signals are pretty diverse depending on the state of the subject. To get a clear signal, whith no noice, you need to average over hundrets of trails (measurments). This is called White noice averaging: Other EEG signals (noise) are treated as being non- correlated to the stimulus:  Serially uncorrelated random variable  0 mean (!) and finite variance 34   No prior distribution is assumed. Through this averaging, important measurements like the N170 which is part of the event correlated potentials (EKPs) and reflects the neural computing of animal/human faces. 1.13.1.1 Noise2Noise and Noise2Void This assumption, that noice is independent of the training data, and therefore can’t be reconstructed. “This implies that we can, in principle, corrupt the training targets of a neural network with zero-mean noise without changing what the network learns.” 1.14 Optical approaches To “make a cell shine”, you use a special Green fluorescent protein (GFP), with a beta barrel sheet structure. To fluorescence, this structure needs to be excited, by blue light, to then drop a few energy steps (vibrational relaxation → temp conversion) and then, drops from the energy lvl. S1 by emitting green light. This can be used to for example visualise a certain protein, by inserting GFP Gene before the stop code for the certain protein. But to track neural activity, we don’t want it to always fluorescent. That’s why we need:  Activity-dependent fluorescence: Change GFP, so that it only fluorescence when there is a neural AP. As the voltage of the action potential (AP) is small and fast and thereby hard to track, researchers use the principle, that in parallel but over a long period and higher amplitude, the Ca2+ concentration rises and drops, during an AP. That’s why they developed: Calcium imaging using calcium as a proxy to measure neural activity. To realise that they developed a variance from GFP (cpEGFP), with a new element (calmodulin), which is a calcium binding regulatory protein. If Calcium is bound, GFP is tight together and can fluorescent, if not, it is loose. In the bounded state, there is the M13 structure. 35 1.15 Why this is important to AI Developer?  Knowing how the cerebrellum works, you can utelise this for example for robot control  Also the other way around: 1.16 Why is it important to know how the brain is connected? · You can predict the function of a circuit by looking at it. Thid connectivity analysis can be done on different levels: Researchers achieved to map the complete connectome of a worm C. elegans, but researchers found out, that it was not enough to reconstruct its function. 1.16.1.1 Representational Similarity Analysis (RSA) Identify which brain areas are actually active, and are they somehow correlated to each other. We can look at matrices, where we can combine different modalities. How does that work: 36 There are also different types of RDMs= Representational Dissimilarity Matrices (RDMs): · Neural RDM · Behavioural RDM · Conceptual RDM Lesioning Why do we do lesion studies: · draw conclusions about function of a certain part of the brain by studying impairment / functional deficit caused by damage to this brain part! · Some kind of „reverse engineering“→ what can brain do without certain parts Lesions can have different reason: · surgeries to treat e.g. epilepsy. · Brain disorders · Injuries after accidents · Non-invasive lesion studies: o E.g. to treat tinnitus There are multiple brain disorders, that lead to lesion: · Tumours · Vascular disorders – blood supply of the brain: o Can be detected with Angiography. o Reasons can be:  Stroke: occlusion of arteries  Cerebral haemorrhage: breakage of blood vessels · Degenerative disorders: 37  o Alzheimer’s disease:  Atrophy of the cerebral cortex and hippocampus  Reason: unknow o Huntington disease  Reason: Genetic  Atrophie interneurons → hyperexcitable skeletal muscle 1.17 Famous examples of lesion studies and consequences 1.17.1 Phineas Gage: · large iron rod completely driven through his head: o destroying brain's left frontal lobe o dramatic changes in personality and behavior 1.17.2 Pierre Paul Broca (1824-1880): · 1861 he reported impairments in one patient (named ‘Tan’) · Tan could understand language lost the ability to speak after injury to the left posterior inferior frontal gyrus · brain region was named Broca‘s area: o important for speech production 1.17.3 Carl Wernicke (1848-1905): · study of receptive aphasia (1876): o impaired comprehension of written and spoken language after injury to the left superior temporal gyrus:  Subject could speak, but it made no sense 1.17.4 Cortical blindness: · total or partial loss of vision in a normal-appearing eye caused by damage to the occipital cortex. o Inability to report visual stimuli, but however sometimes subjects behave as they have seen the object:  That’s possible because there are 10 identified pathways from eye to the brain. With the most important being V1, the others are evolutionary more ancient, but not removed. 1.18 Lesion studies and brain surgeries against epilepsy What is Epilepsy:  Abnormal hyperactivity in the brain: o Leads to seizures → loss of consciousness, shaking…  Were often treated by lobectomies (resection of cortical lobe)  After lobectomy → scientific evaluation of the effects 38 1.18.1 Patient H.M. (Henry Gustav Molaison) He received in 1953 a bilateral medial temporal lobectomy, a surgical resection of the anterior ⅔ of his hippocampi, parahippocampal and entorhinal cortices.  he became unable to form new explicit memories: o experiences o Only short-term memory of a few minutes  Still able to learn new motor skills: playing an instrument His case was important to:  theories, that explain the link between brain function and memory  development of cognitive neuropsychology 1.18.2 Patient W.J. Another important role played Professor Gazzanniga, as he “created” split-brain patients though a dissection of the corpus callosum to treat epilepsy. One important patient was W.J.:  both hemispheres could not communicate with each other: o stimuli presented to the right visual field:  could verbally report what he had seen o stimuli presented to the left visual field  could not verbally report but press left button (right hemisphere) o conflicts between the hemispheres:  left hand opens a door, right hand tries to stop left hand 1.19 Non-invasive lesion studies 1.19.1 Transcranial magnetic stimulation (TMS): · stimulator generates changing electric current within the coil which induces a transient magnetic field: o magnetic field causes a second inductance of inverted electric charge within the brain itself → hyperactivity → area. o temporally switched off due to refractory period · used clinically to measure function of specific brain circuits in humans: o e.g. used to trat tinnitus but is not so exact and good understood. 1.19.2 Transcranial direct current stimulation (tDCS): · Old Greeks used it. · Today: two small electrodes (1-2mV): o Neurons below the anode become depolarized → more excitable o Neurons below cathode become hyperpolarized → less excitable o Used to treat neurological conditions (e.g. chronic pain) The huge advantage of tDCS and TMS: people are their own control group. 1.20 Lesion studies in AI Could a Neuroscientist Understand a Microprocessor? 39 In contrast to biological brains, AI systems can be completely read out. Location and expansion of lesion can be controlled. Lesioning of individual neurons, connections or layers to reverse-engineer function of an artificial deep neural network. However if lesion studies are the right way to unravel the brain/AI is questionable. And the result is, that only relying on lesion studies to understand the function of the brain is not enough. 40 Imaging Techniques 1.21 Brain structure 1.21.1 Computed Tomography (CT): · Introduced 1983 · Calculate 3D reconstruction from 2D x-ray images · spatial resolution: 0.5-1cm 1.21.2 Magnetic Resonance Imaging (MRI): · Spartial resolution below 1mm 1. Protons align in magnetic field of scanner (up to 7T) 2. Protons are disturbed by radio waves (RF coil) 3. Protons re-align in magnetic field and emit energy via radiation (RF coil switched off) 4. Sensor detects the emitted waves Emitted waves 1.21.3 Diffusion Tensor Imaging (DTI, special form of MRI): · Measure the motion of water contained in axons: o Used to unravel the wiring scheme of the brain (connectome) · Uses a magnetic field gradient 1.21.4 Photo-Acoustic Imaging: · Absorption of laser pulse in tissue: o Thermal expansion → sound wave o Ultra-sound detector receives sound waves · Could already be used to track brain function (blood flow etc) 1.22 Brain processes / functions 1.22.1 EEG, MEG, Single Cell Recordings etc.: · good temporal resolution but bad spatial resolution 1.22.2 Positron Emission Tomography (PET): · Measure local variations in cerebral blood flow: o Radioactive “tracer” is injected (15O) that emits a positron. o This positron reacts with an electron creating a gamma ray o Measured in gamma ray detector, shows line of response (LOR) · Better detectors were developed: o higher efficiency and better spatial resolution o 2D→3D System → coincidence events can be detected across several detector rings · Time of flight analysis (TOF): Where did event occur along the LOR 41 1.22.3 Functional Magnetic Resonance Imaging (fMRI): · Principle is similar as for MRI · Exploits the magnetic properties of the blood pigment (haemoglobin) of the blood cells → Haemoglobin transports oxygen in blood · Haemoglobin without oxygen (Hb) is paramagnetic → Results in a higher T2 value for oxygenated haemoglobin (HbO2) using transverse magnetization · Limitations: Good spatial resolution (~1mm), but slow temporal resolution, as the whole blood oxygenation has a delay after a presented stimulus · Some approaches exist to use fMRI for research on language processing → it is necessary to carefully consider the limitations of that technique 1.22.4 Functional Near Infrared Spectroscopy (fNIRS): · Measures also the fraction of oxygenated and deoxygenated haemoglobin, but not via magnetic properties · Infrared emitter shines near infrared light (700nm - 900nm) through the skull which is refracted at brain o Skull and skin is transparent for this wavelengths o Light is absorbed by haemoglobin depending on oxygenation 1.23 Conclusion 42 Brain Structure and Functional Systems Increases heart rate (fight or flight) Slows heart rate 1.24 Brainstem 1.24.1 Medulla (oblongata) Motor nuclei that innervate the heart, controls respiration, heart rate, etc., relay station for sensory and motor information 1.24.2 Pons connection between brain and cerebellum, important for eye movement (saccades), responsible for generating rapid eye movement (REM) sleep 1.24.3 Midbrain (mesencephalon) large fiber tracs from the telen/diencephalon run through the midbrain to spinal cord or cerebellum 1.25 Cerebellum The cerebellum (pl.: cerebella or cerebellums; Latin for "little brain") is a major feature of the hindbrain of all vertebrates. Temporal coordination of movements (and cognition) Error correction of movements Learning of new motor skills Extremely regular structure: o Climbing fibres, parallel fibres o Contains most neurons of the brain:  69 billion neurons from 86 billion neurons in total 43 1.26 Diencephalon 1.26.1 Thalamus „Gate Keeper“ of cerebral cortex (“gateway to consciousness”) Relay station of sensory input to cerebral cortex “Grand Central Station of the Brain” Several nuclei 1.26.2 Hypothalamus · Connection of the brain and the endocrine system · controls functions of maintaining normal state of body o Hunger and thirst o Body temperature o Controls hormones 1.27 Telencephalon (part of cerebrum) · Evolutionary newer · Limbic system as system of emotional behavior 1.27.1 Hippocampus Spatial Navigation Episodic Memory Memory formation Organization of thoughts 1.27.2 Amygdala Emotional responses Control of fear, anxiety, aggression 1.28 Basal ganglia The basal ganglia (BG) or basal nuclei are a group of subcortical nuclei found in the brains of vertebrates, below the cortex. The basal ganglia are associated with a variety of functions, including regulating voluntary motor movements, procedural learning, habit formation, conditional learning, eye movements, cognition, and emotion. They regulate motor and premotor cortical areas, facilitating smooth voluntary movements. „Servo-mechanism“ for cerebral cortex Shortcuts from sensory association areas to motor control Trained, complex input-output mappings Model-free reinforcement learning 44 1.29 The Cerebral Cortex The Cerebral Cortex consits of approx. 16 billion neurons. It defines the highest level of control, like voluntary movements, conscious perception as well as the highest cognitive functions: Language, Math, Reading It is also responsible for Semantic memory and auto- biographic memory. The cortex is made of large sheets of layered neurons, that are draped and folded for an increased surface and shorter axonal connection (in 3d structure). 1.29.1 Cyto-architectional organization Vertically organized in cortical layers: Nissl stain: cell bodies of neurons Golgi stain: dendrites and axons of a random subset of neurons And horizontally organized in macro- and mini-columns: 1.29.2 Functional organization (Brodmann areas) 1.29.3 Use in AI: Self-organizing feature maps Modify the weight vectors of neurons with the lowest distance to the input data. This generates a map of neurons where nearby outputs share the same properties. 45 1.29.4 Cortical connectivity All layers are interconnected with layers of following columns or areas 46 47 1.30 Lateralization of Brain Function Each part of the brain exists twice: left and right side Cerebral Cortex: left and right hemisphere Lateralization: o tendency for neural functions or cognitive processes to be specialized to one side of the brain o homologue cortex areas at both sides have different functions Left hemisphere represents right side of the body and vice versa Best example is language: Broca‘s and Wernicke‘s area located exclusively in left hemisphere in 95% of right-handers and 70% of left- handers Using Wada tests, that reversibly block one side of the brain it is possible to determine the independent different functions of the brain sides Left hemisphere Right hemisphere - Language: Word detection/generation - Melody, Pitch, Intensity - Verbal - Non-verbal - Reading - Drawing - Problem solving - Visio-spatial tasks - Sequential processing (math...) - Face recognition - Analytic - Parallel processing This asymmetry may be a more efficient and flexible design principle, reduces redundancy across hemispheres and allows for a no-cost extension. 1.30.1 Consciousness in split brain patients Left hemisphere tries to find explanations for actions initiated by right hemisphere. For example the patient gets a “stand up” command, obeys and stands up. On questioning “why did you stand up”, the left hemisphere creates an explanation like “I wanted a coke” as it is not aware of the right hemisphere. Perhaps consciousness is not a single generalized process, but an emergent property out of thousands of modules, that compete for attention and create our “stream of consciousness”. 48 Sensation and Perception 1.31 The Auditory System - Hearing The auditory system transforms and processes sound, which are longitudinal waves (pressure fluctuations). Subjective Loudness rises logarithmic with sound pressure, thus loudness is measured in dB, and is measured with the Weber-Fechner Law. L = loudness P= Sound Pressure P0= 2*10-5 Pa Threshold The best hearing is approx. at 3.4 kHz. The sensory system for the sense of hearing includes the sensory organs (the ears) and the auditory parts of the sensory system. 1.31.1 The outer ear The outer ear functions as a Funnel. The Pinna is a important structure for directional hearing, as the transmission from pinna to tympanic membrane is not linear and has a resonance frequency at 3.4 kHz, which is the frequency of best hearing. 1.31.2 The middle ear The middle ear consists of three structures: Malleus Incus Stapes Pressure fluctuations have to be transmitted to fluid. Normally 98% of the sound would be reflected at the border from air to fluid, but the three 3 ossicles of middle ear enable impedance adjustment, such that only 40% of the sound is reflected, the structure leads to better hearing of 27 dB. 1.31.3 The inner ear - the cochlea The cochlea performs a mechanotransduction: Mechanical signal is transduced to electrical signal. Organ of corti transduces mechanical signal into chemical signal (neurotransmitter release). The inner hair cells are moved by the flow endo lymphe, and the outer hair cells move to further enhance the amplitude of the stimulus. 49 The inner hair cells: Transduce mechanical deflection to electrical/chemical signal, by “Tip links” at top of stereocilia pull on ion channels, that leads to the inner hair cell is depolarized, which then releases Glutamate. The outer hair cell: The outer hair cells are no sensory cells but serve by amplification of travelling wave. This works, as the outer hair cells can contract due to the protein Prestin, when getting a signal from the superior olive. 1.31.4 The auditory pathway The neurotransmitter (glutamate) depolarizes cells of spiral ganglion, which leads to spiking of spiral ganglion neurons. Up to a frequency of 4 kHz, they perform phase locking, which is the tendency of neurons to fire action potentials at particular phases of an ongoing periodic sound waveform. This helps, by better understanding language. Spiking signal is transmitted to the second synapse (dorsal cochlear nucleus, ventral cochlear nucleus) in medulla. Ventral cochlear nucleus extracts temporal a spectral structure of the signal Dorsal cochlear nucleus integrates auditory signal with somatosensory signal, perhaps to detect sound sources. Along the whole auditory pathway, including the auditory cortex, lateral inhibition happens. That means, that cells inhibit neighbouring cells. This leads to sharpening of the frequency selectivity of the auditory system, and by that contrast enhancement. The axons innervate the superior olivary nucleus this results in a shared information from both ears. Birds for example have the logic, that they user inter-aural time differences to map different neurons based on the position of the sound source. 50 The involved areas in the brainstem are: Lateral Lemniscus (Pons): o Inhibitory neurons o Plays a role in source localization Inferior colliculus (Midbrain): o Is used to suppress reflections of sounds from surfaces o This is necessary to perform a source localization Medial Geniculate Body (Thalamus in diencephalon, not brainstem!): o Gate to consciousness o Filters out signal that should not ascend to cortex and thus consciousness The higher processing of the audio system is performed in two streams: Dorsal stream: o Sensori-motor integration o Speech production o „Where stream“ Ventral stream: o Phonological processing o Auditory objects o Speech comprehension o „What stream“ 1.31.5 Tinnitus Tinnitus is related to hearing loss. The synapses in the cochlea are damaged, resulting in less input in the cochlear nucleus. Central Noise and Stochastic Resonance: Neuronal noise is added to lift subthreshold above detection threshold, and activates the feedback-loop implemented in dorsal cochlear nucleus. This results in a maximized information transmission, following an auto-correlation as measure for information. 1.32 The Vestibular System - Balancing For balance, and five vestibular organs are involved (mentioned for completeness). 1.33 The Olfactory System - Smelling The sense of smell, or olfaction, is the special sense through which smells (or odors) are perceived and is a chemical sense like the gustatory sense. Smell is triggered by odor molecules → odorants. Odorants bind to odor receptors/ or small vibrations of odors lead to the sensation o Bipolar cells (receptors) are embedded in olfactory epithelium (1000 different receptors, Receptor cells are bipolar neurons) Bipolar neurons send signal to olfactory bulb. 51 Different combinations of receptor encode different odors, each receptor responds to different odors → unique constellation of receptors. Each glomerulus receives only input from one receptor type, but each odor is recognized by many receptor types → many glomeruli are activated by one odor and closely related odors activate neighboring glomeruli The corresponding signal to the primary olfactory cortex is ipsilateral, meaning there is no crossing, and it does not pass the thalamus. Instead it projects to the amygdala, which is responsible for emotions and autonomous functions. 1.33.1 Fruit fly algorithm Fruit fly generates a tag for every odor: 1. Feed-Forward: become independent from odor concentration 2. Dimensionality expansion via sparse, binary random connection matrix 3. Winner takes it all Outperforms standard locality sensitive hashing for all hash lengths during similarity search! This algorithm might be implemented in different brain areas (e.g. in rat cerebellum). 1.34 The Gustatory System - Tasting The sense of taste, or gustation is the sense through which tastes are perceived. And is as well chemical sense like olfactory sense. Papillae on the tongue contain several taste buds, which contain several taste cells. There are five basic tastes: salty, sour, bitter, sweet and umami And the taste begins when a tastant stimulates a receptor: For bitter, sweet, umami → complicated protein cascade for salty and sour → change of ion concentration (bitter has low detection threshold) Signal transmitted to gustatory cortex (in insular cortex) via brainstem, thalamus and the orbitofrontal cortex is important for processing the pleasantness of stimulus (chocolate). 52 Visual System A visual scene is analysed in 3 levels: Low-level processing, e.g.: o Orientation o Color o Contrast o … Intermediate-level processing, e.g.: o Contour integration o Surface properties o Shape discrimination o Surface depth o … High-level processing the: o Object identification There are different visual fields: Hemifield (right/left): signals from both eyes, but only from one side. X eye visual field (left/right): signals from only one eye (x) Monocular visual field: the regions only one eye perceives. Binocular visual field: the region, both eyes perceive. Full visual field: the combination of the left eye visual field and the right eye visual field. The basic visual pathway: One strange thing is, that the neural communication is against direction of light. 53 There are to different types of photoreceptors, the Rods (Black & White) and Cones (Colour). But how does this work? Also important are bipolar cells, that can bypass signal or invert signal as well as amplify the signal. There are two types: Off bipolar cell ON bipolar cell As well as Ganglion cells, that encode the signal using action potentials, with two types: Off-centre ganglion cell ON-centre ganglion cell There are different cones, for different wavelength, and their computation allows for the perception of colours. Let’s briefly discuss, how a colourful digital image is generated using the Bayer pattern. A Bayer filter mosaic is a color filter array (CFA) for arranging RGB color bandwith filters on a square grid of photosensors. Its particular arrangement of color filters is used in most single-chip digital image sensors used in digital cameras, and camcorders to create a color image. 54 The grid contains 50% green and 25% red and blue bandwidth filters, due to the importance of green, to for example generate contrast and clearness. The result is a Bayer pattern image, which then needs to interpolate towards a set of complete red, green and blue values for each pixel. Let’s get back to the physiological level. Our communication channel: the optic nerve, and where it leaves the eye, there is the Optic disc, with no photoreceptors, normally our brain interpolates over it, but it can be tricked. 1.35 The receptive field The retina has not the same resolution everywhere! But what is a receptive field? In general, it is the group of photoreceptors and horizontal, bipolar and amacrine cells that innervate a retinal ganglion cell. And it has a Centre-surround structure, which is getting bigger and bigger, with rising distance to the Fovea. Based on the way these receptive fields are interconnected, and the characteristics of the bipolar and ganglion cells, different behaviour like contrast enhancement is realised. 55 Another interesting function is lateral inhibition, between neighbouring receptive fields, that causes a larger membrane potential difference, and can therefore enhance the perception between the dark and light and edges. Edge detection on images can be conducted, by calculating the derivative, or with linear filters, like the Prewitt or Sobel Filter. Also, CNN could be trained, to generate a activation map from a colour image. If one applies multiple rounds of filters, he thereby increases the receptive field. In the brain, you can see the same behaviour. 56 Also in CNN, you can see that with increasing depth, the features get to higher level. 1.36 The Visual Cortex V1 Retina produces visuotopic maps in V1, so that you can see a striped excitation pattern to a striped visual stimulus. There are two structures: Ocular dominance colums (Separation of the right and left eye) Orientation Columns (Neurons active if “their” orientation is represented in the visual field. 1.37 Object recognition Now we talk about high-level visual processing, and object recognition. Important to notice is that there are two streams, the dorsal stream (“where”) and the central stream ("what”). 57 1.38 Face Recognition Inferior temporal cortex (IT) is our face detector, as experiments showed, that the activation vastily increases if complete with more and more clear visual stimuli of faces are presented. There are multiple Hypothesis, how people identify different people, one is the Ensemble Coding Hypothesis. It states, that the different features are separated and the Ensemble of them leads to identification. Another study showed, that single neurons spike, for a given face, and therefor suggest that a single neuron represents a single represented human face. Also the voice etc, stimulates the neuron. But what is a face and how can we identify the features: By using dimensionality reduction we get a high-level and low dimensional representation. Methods are: PCA + VQ (Vector Quantization) → Holistic NMF → parts-based CNN can perform Real-Time Object detection, like You Only Look Once (YOLO), and also Face identification with Face Net, this can also be done by DNN (Deep Neural Networks). 58 Motor system The main reason why a brain has evolved in evolution is to control muscles, to impact the surrounding environment. For example, the tunicates are small sea squirts trying to find a rock to stick on for the rest of its life. After finding one they “eat their brain”. 1.39 Elicit behaviour There are three types of muscles: cardiac, smooth and skeletal muscles. Only skeletal muscles can be moved voluntarily. Muscles can only contract in one direction, meaning that there is always a counterpart in the opposite direction. The alpha motor neurons are connected to the ventral root of the spinal cord, with different sized motor neurons for different amounts of muscle fibers having size depending action potentials. Muscle fibers of different motor units are intermingled, so the tension applied to the tendon remains fairly constant, regardless of which motor units are stimulated. On the other side the sensory neurons are connected to the dorsal root of the spinal cord. This can be demonstrated by applying a force at the knee on the quadriceps. This triggers a sensory neuron connected to the spinal cord, that in reply triggers a motor neuron contracting the quadriceps. All without any connection to the brain. Though there are five spinal connections to the brain, divided in the pyramidal and extrapyramidal tract. 1.40 Brain regions The “homunculus” is a representation of the human body based on the proportions of the responsible brain regions 59  Brain regions involved in motor The basal ganglia: generation: 1.40.1 Disorders in basal gangial 60 1.41 Internal models 1.41.1 Inverse model 1.41.2 Forward model The inverse model takes the desired The forward model takes an input trajectory, and generates the motor motor command, predicts the command to achieve this. movement and reduces the error to the desired trajectory while moving. PID-controller – Proportional (now), Integral (past), differential (future) controller with a factor K that needs to be tuned for every application. 1.41.3 Challenges in Brain-Machine Interfaces (BMI) · Invasiveness · Placement of BMI · Motion · Biocompatibility · Durability · Complexity/Bandwidth 61 Language, Attention, Saliency Linguistics is the study of language (no unique tools) and is inter-disciplinary field itself. A contribution of Linguistics in Cognitive Science is, that in 1959 Noam Chomsky’s critique of behaviourism: (Skinner’s book → you can only say things you have heard before) What is language? There is no sophisticated definition for language but some characteristics: Communicative: Transmission and comprehension of information Arbitrary: Symbols that represent things are arbitrary (truck in USA, lorry in UK) Structured: Ordering of symbols is not arbitrary but follows a set of rules Generative: Nearly infinite number of sentences can be build (makes language powerful as any idea can be expressed) Dynamic: Language changes (tweet: sound of a bird → now message on twitter) Language enables us to learn form experiences. Important terms: Phoneme: smallest sound that can change meaning of a word (late, rate) Morphemes: smallest units of spoken language that have meaning (defrost, frost, defroster, morphemes are structured collection of phonemes, stem morpheme: frost, bound morpheme: er) Syntax: rules to arrange words in sentences (related to grammar) Semantics: meaning of words an sentences Pragmatics: meaning of language in social and physical context 62 1.42 Lesion studies and Apasia Aphasia: collective term summarizing deficits in language understanding and production Broca’s aphasia (“tan”): problems with speech production (Broca’s view) but also with grammar (syntax → agrammatic aphasia) Wernicke’s aphasia: problems with speech understanding, produce fluent speech but sentences make no sense (modern view: areas around Wernicke’s area have biggest influence) Conduction aphasia: Damage of white matter tract from Wernicke’s to Broca’s area called arcuate fasciculus → patients understand words and speech errors they hear but cannot correct the own speech errors, problems in producing spontaneous speech and repeating speech, use words incorrectly. 1.43 Language Comprehension Language Comprehension is done in three steps: 1. Perceptual analysis 2. Access to representations in mental lexicon 3. Lexical integration Let’s go through them one by one. 1.43.1 Step 1: Perceptual analysis Auditory or visual input is translated into phonological/orthographic input code. The Spoken Input: The challenge is, that acoustic noise has to be divided from relevant speech signal in order to identify phonemes. There are several problems: 1. phonemes sound different for male and female speakers 2. Auditory speech signals are not clearly separated (even not between words) Prosody (rhythm and intonation) helps to segment the speech stream. The Written Input: The challenge ist, that reading is a recent invention (5500 years old). Therefore, the task is to link arbitrary visual symbols to meaningful words. TODO: Pandemonium model 1.43.2 Step 2: Identifying and storing words in the brain To derive meaning from language input and to produce speech, the brain must store words and concepts that’s the mental lexicon. There are three general functions of mental lexicon: Lexical access: perceptual output activates word-from representations. Lexical selection: lexical representation in mental lexicon which best matches the input is selected. Lexical integration: integrates words into full sentence. 63 The Mental lexicon is the mental store of semantic, syntactic information, spelling, and sound patterns of words. There are two theories: 1. One lexicon for language understanding and production 2. Two different lexica (separate input and output lexicon) Characteristics of the mental lexicon: 1. smallest representation unit in the mental lexicon is the morpheme (Morpheme: smallest meaningful unit in language, frost, defrost, defroster) 2. more frequently used words are accessed more quickly 3. lexical neighbourhood: neighbourhood of words with differ only in one phoneme (smallest unit of sound that changes meaning, late and rate, words with many overlapping phonemes are organized together in the brain 4. Semantic relationship between words In 1975 a influential model by Collins and Loftus, was released. The Idea: Word meanings are represented in a semantic network (words= nodes connected with each other) Distance between words is determined by semantic relations of words (e.g. car is near truck) 1.43.2.1 Word2Vec – Mental Lexicon in Computers? Learn representations of words: each words is mapped to one vector (embedding). Embedding should be useful and efficient and should cover some semantic and syntactic relationships. Using a 1:1 mapping is not very efficient. So there evolved two models. The continuous bag-of-words models predicts the middle word based on surrounding context words and the continuous skip-gram models predicts the surrounding words of certain word. Three classes of models explaining word comprehension: Modular models: normal language comprehension in separate modules (no influence of higher-level representations on lower-level representations, bottom-up data flow) Interactive models: all types of information can participate in word recognition Hybrid models (in between): the context reduces the number of possible word candidates in the mental lexicon o → Modern view: lexical selection is indeed influenced by context 64 1.43.3 Step 3: Integration of words into sentences Semantic information on the words alone is not enough to get the meaning of the whole sentence! Syntactic parsing: The brain cannot store whole sentences → the brain has to assign a syntactic structure to words in sentences The semantic processing: The N400 wave is a negative voltage peak in ERPs (event related potentials) and is sensitive to semantic aspects of linguistic inputs o Semantic violations lead to larger N400 (more negative) wave P600 also called syntactic positive shift o 600 ms after syntactic violation LAN (at 400 ms): left anterior negativity → In case of errors (differences to prediction) there is a higher activity in the brain 1.43.4 Language Comprehension: Conclusion Models of language comprehension include the linking of linguistic input with memorized knowledge. We know where things happen but not what happens! (“Localization is no explanation” David Poeppel). Recently, the group of Friedemann Pulvermüller develops biologically plausible simulations to understand language processing using spiking neuron models. 1.44 Language/Speech Production Model by Willem Levelt (1989) 1) Message Preparation (Conceptualizer) a. Macroplanning: What to say (content) b. Microplanning: How to say (“The park is next to the house.” “The house is next to the park.”) 2) Formulator: Output of micro-and macroplanning is a conceptual message → is given to a formulator → puts message in a phonologically and syntactically correct form 3) Articulator: word syllables are mapped to motor patterns that move our tongue, mouth and vocal apparatus Levelt model is a complete serial (box and arrow) model → no parallel processing 65 1.45 Attention The (Selective) attention, is the ability to prioritize things while ignoring others. There are two versions: Goal-driven control (top-down): (You attend to the lecture because you want to pass the exam) Stimulus-driven control (bottom-up) (You hear a bang and you check out what happened) In contrast to selective attention, is arousal a global state of the organism. (Selective) attention influences how people code sensory inputs, store information in memory, act on to survive. Mechanisms that determine where and on what our attention is focused → attentional control mechanisms. The Cortical and subcortical areas important, and there is Neglect (brain attention network is damaged in one hemisphere): attention bias in the direction of the lesion. lesion right → left visual field is ignored. The cocktail party effect, where one follow conversations in loud environments, shows, that the Information processing system has limited capacity. 1.46 Fundamental Recent Developments “Language and Attention” TODO The skill of language processing might be enough to develop general intelligence. That means, that potentially we do not need brain like architecture to reach general intelligence. A danger is that LLMs are biased due to bias in training data. And that the system does not know when it is just guessing and when it is knowing. Game Changer in computational linguistics, also known as natural language processing are Transformer Networks. Main advantage: No recurrent neural network o training can be parallelized Main principles: o Positional encoding: vector is added containing the information o