Week 3 - Unit II - Neural Network Fundamentals - I.pptx.pdf

Unit II: Neural Network Fundamentals Deep learning, biological and artificial neurons, perceptron CSA301 Deep Learning 1 Learning Outcomes Explain the concepts of artificial intelligence and deep learning Compare and contrast the biological a...

Unit II: Neural Network Fundamentals Deep learning, biological and artificial neurons, perceptron CSA301 Deep Learning 1 Learning Outcomes Explain the concepts of artificial intelligence and deep learning Compare and contrast the biological and artificial neurons Explain and implement the concept of perceptron and multi-layer perceptron 2 Artificial Made or produced by human beings rather than occurring naturally, especially as a copy of something natural. However, artificiality does not necessarily have a negative connotation, as it may also reflect the ability of humans to replicate forms or functions arising nature, as with an artificial heart or artificial intelligence. 3 Artificial Intelligence Artificial intelligence (AI) is the intelligence exhibited by machines or software. It is an academic field of study which studies the goal of creating intelligence. A branch of computer science dealing with the simulation of intelligent behaviour in computers. Source: https://www.datacatchup.com/wp-content/uploads/2019/05/image.png 4  What is machine learning? Machine learning is a type of artificial intelligence that provide computer with the ability to learn without being explicitly programmed. Source: https://www.edureka.co/blog/deep-learning-tutorial 5 Limitations of Machine Learning One of the big challenges with traditional machine learning model is a process called feature extraction. For complex problems such as object recognition, this is a huge challenge Deep Learning to the rescue Deep learning models are capable to focus on the right features by themselves, requiring little guidance from the programmer. The idea behind deep learning is to build learning algorithm that mimic brain 6 Deep Learning Picture from: https://www.xenonstack.com/blog/static/public/uploads/media/machine-learning-vs-deep-learning.png 7 Deep Learning Deep Learning is a subfield of machine learning concerned with algorithms inspired by the structure and function of the brain called artificial neural networks. The term neural network is a reference to neurobiology, but although some of the central concepts in deep learning were developed in part by drawing inspiration from our understanding of the brain, deep-learning Source: https://levity.ai/blog/difference-machine-learning-deep-learning models are not models of the brain. 8 Biological Neurons Source: https://prod-mpc.upgrad.com/blog/biological-neural-network/ 9 Biological Neurons Neurons connected to another neuron Source: https://commons.wikimedia.org/wiki/File:Two_neurons_connected.svg 10 Biological Neurons The cell body of the neuron is called the soma, where inputs (dendrites) and output (axons) connect soma to other soma. Soma process the information Each neuron receives electrochemical inputs from other neurons at their dendrites. If these electrical inputs are sufficiently powerful to activate the neuron, then the activated neuron transmits the signal along its axon, passing it along to the dendrites of other neurons. These attached neurons may also fire, thus continuing the process of passing the message along. Supplementary Reading: https://cs.stanford.edu/people/eroberts/courses/soco/projects/neural-networks/Biology/index.html 11 Biological Neurons Assume you're watching Friends. The information your brain receives is now processed by the "laugh or not" set of neurons, which will assist you in deciding whether or not to laugh. Each neuron is only fired/activated when its respective criteria are met, as shown below. https://towardsdatascience.com/mcculloch-pitts-model-5fdf65ac5dd1 12 Biological Neurons In reality, it is not just a couple of neurons which do the decision making. Our brain contains a massively parallel interconnected network of 1011 neurons (100 billion), and their connections are not as straightforward. https://towardsdatascience.com/mcculloch-pitts-model-5fdf65ac5dd1 13 Biological Neurons This massively parallel network also ensures that there is a division of work. Each neuron only fires when its intended criteria is met i.e., a neuron may perform a certain role to a certain stimulus, as shown below. https://towardsdatascience.com/mcculloch-pitts-model-5fdf65ac5dd1 14 Artificial Neurons Artificial neurons mimic the basic function of biological neurons, and much like their biological counterparts they only become useful when connected in a larger network, called Artificial Neural Networks. The concept of ANN was first introduced back in 1943 by the neurophysiologist Warren McCulloch and the mathematician Walter Pitts. 15 16 Artificial Neurons The early successes of ANNs until 1960s led to the widespread belief that we would soon be conversing with truly intelligent machines. When it became clear that this promise would go unfulfilled (at least for quite a while), funding flew elsewhere and ANNs entered a long winter. By 1990s, powerful machine learning techniques such as support vector machines was invented. These techniques seemed to offer better results and stronger theoretical foundations than ANNs, so once again the study of neural networks entered a long winter. During that time, there were not much data and computing power available to harness the power of the ANN. 17 Artificial Neurons However, we are yet gain witnessing another wave of interest in ANNs and will this wave die out like the previous ones did? There are few good reasons to believe that this wave is different and that it will have a much more profound impact on our lives: ○ Huge quantity of data available to train neural networks and ANN outperform other ML algorithms ○ Increase in computing power -> train large neural networks ○ Entered a virtuous circle of funding and progress 18 McCulloch-Pitts Model of Neuron The McCulloch Pitt's Model of Neuron is the earliest logical simulation of a biological neuron, developed by Warren McCulloch and Walter Pitts in 1943 and hence, the name McCulloch Pitt’s model. The motivation behind the McCulloh Pitt’s Model is a biological neuron. A biological neuron takes an input signal from the dendrites and after processing it passes onto other connected neurons as the output if the signal is received positively, through axons and synapses. This is the basic working of a biological neuron which is interpreted and mimicked using the McCulloh Pitt’s Model. 19 McCulloch-Pitts Model of Neuron McCulloch and Pitts proposed a very simple model (McCulloch-Pitts Neuron Model) of the biological neuron, when later became known as an artificial neuron: it has one or more binary (on/off) inputs and one binary output. The artificial neuron simply activates its output when more than a certain number of its inputs are active. Mostly used in logical functions 20 McCulloch-Pitts Model of Neuron To visualise, let’s build a few ANNs that perform various logical computations assuming that a neuron is activated when at least two of its inputs are active. ANNs performing simple logical computations [McCulloch-Pitts Neuron] 21 McCulloch-Pitts Model of Neuron The first network on the left is simply the identity function: if neuron A is activated, then neuron C gets activated as well (since it receives two input signals from neuron A), but if neuron A is off, then neuron C is off as well. 22 McCulloch-Pitts Model of Neuron The second network performs a logical AND: neuron C is activated only when both neurons A and B are activated (a single input signal is not enough to activate neuron C). 23 McCulloch-Pitts Model of Neuron The third network performs a logical OR: neuron C gets activated if either neuron A or neuron B is activated (or both). 24 McCulloch-Pitts Model of Neuron Divided into two parts (g & f) G takes an input (dendrite), perform an aggregation and based on the aggregated value, the second part, f makes decision. f is a threshold value which decides the activation of the input value. If the value of the function is equal to or Source: greater than the threshold value, it gives a https://towardsdatascience.com/mcculloch-pitts-model-5fdf65ac5dd1 positive output and vice versa. 25 McCulloch-Pitts Model of Neuron A bank wants to decide if it can sanction a loan or not. There are two parameters to decide Salary and Credit Score. So there can be 4 scenarios to assess- High Salary and Good Credit Score High Salary and Bad Credit Score Low Salary and Good Credit Score Low Salary and Bad Credit Score Let X1 = 1 denote high salary X1 = 0 denote low salary X2 = 1 denote good credit score X2 = 0 denote bad credit score 26 McCulloch-Pitts Model of Neuron Let the threshold value be 2. The truth table is as follows: The truth table shows when the loan should be X1 X2 X1 + X2 Loan Approved approved considering all the varying scenarios. In 1 1 2 1 this case, the loan is only approved if the salary is 1 0 1 0 high and the credit score is good. The McCulloch 0 1 1 0 Pitt's model of neuron was mankind’s first attempt 0 0 0 0 at mimicking the human brain. 27 McCulloch-Pitts Model of Neuron Let’s take the task of LBW predictions using ML. Source: https://tinyurl.com/3uufaze2 28 McCulloch-Pitts Model of Neuron In this case, the data could be represented in the Boolean format as below: Source: https://tinyurl.com/3uufaze2 29 McCulloch-Pitts Model of Neuron There might be other factors affecting LBW decision but here we are considering only these three features in this case. We need to find the value of b in such a way that when all the x values are plugged in the equation along with the b value, the predicted value of y must match the true output. Here b can be considered as the threshold value. Source: https://tinyurl.com/3uufaze2 30 Biological Neuron vs Artificial Neuron Biological Neuron Artificial Neuron Cell Nucleus (Soma) Node Dendrites Input Axon Output 31 The Perceptron (Single Layer Perceptron) A computer model or computerized machine devised to represent or simulate the ability of the brain to recognize and discriminate. The perceptron is one of the simplest ANN architectures, invented in 1957 by Frank Rosenblatt. Slightly tweaked version of the McCulloch-Pitts Neuron. Here the neurons are called Threshold Logic Unit (TLU) ○ Inputs and outputs are numbers instead of binary on/off values and each input connection is associated with a weight. The perceptron is a mathematical model of a biological neuron. It is an algorithm for supervised learning of binary classifiers. The perceptron can work on non-boolean values where each input connection get associated with a weight. 32  The Perceptron (Single Layer Perceptron) Architecture of the perceptron network The Perceptron outputs either a 0 or a 1, thus, in its original form, the Perceptron is simply a binary, two class classifier. 33 The Perceptron (Single Layer Perceptron) Perceptron has weight and bias that could be determined by the learning algorithm. Learning is achieved by an iterative process of weights adjustments until the exact output response is produced. TLU computes a weighted sum of its inputs: { z = w1 x1 + w2 x2+ … + wn xn = xTw } + bias After computing the weighted sum, the step function (activation function) is applied to the sum and outputs the result: hw(x) = step(z), where z = xTw 34 MP Neuron Model vs Perceptron Model Both, MP Neuron Model as well as the Perceptron model work on linearly separable data. MP Neuron Model only accepts boolean input whereas Perceptron Model can process any real input. Inputs aren’t weighted in MP Neuron Model, which makes this model less flexible. On the other hand, Perceptron model can take weights with respective to inputs provided. 35 Bias and Weights The bias allow you to shift the activation function by adding a constant to the input. Bias in Neural Networks can be thought of as analogous to the role of a constant in a linear function, whereby the line is effectively transposed by the constant value. Whereas, weights shows the strength of the connection between the node (neurons). The weight may bring down the importance of the input value or it may elevate it. 36 Bias - Need of bias The above figure can be written as : y = mx+c where m = weight and c = bias 37 Bias - Need of bias Suppose, if c was absent, the graph will be formed like this: Y = mx Y = mx + c Due to absence of bias, model will train over point passing through origin only, which is not in accordance with real-world scenario. With the introduction of bias, the model will become more flexible. Bais: https://www.youtube.com/watch?v=e570k28sCoc 38 Examples of Weights and Biases Neural networks were designed to mimic how the human brain differentiates and organizes inputs. For example, to train an AI model to identify the letters A, B, and C, the neural network will need to understand the shapes that make up each letter. For the letter C, the model will need to detect three shapes: a top curve, a slightly bent line to the left, and a bottom curve. If a top curve is detected, it will propagate forward to the next layer. However, neurons can accidentally be triggered by miscalculating data. It might see the top curve of B or the left line of A and misclassify those letters as C. Weights and biases further define the importance of signals and unidentified data features. These additions will help eliminate neural network errors. Inputting a bias into hidden layers will place a data characteristic that was missed in a previous iteration. Shifting the necessity of a signal using weights can help a machine learning model define the importance of calculated data. 39 Implementing Logic Gates with Perceptron Activation Function def activation_function(y): if y