Neural Network PDF

Learning from Data Elshimaa Elgendi, PhD Operations Research and Decision Support Faculty of Computers and Artificial Intelligence Cairo University Supervised Learning Neural Network Textbooks Zurada, J.M. (1992) Introduction to Artificial Neural Systems. West P...

Learning from Data Elshimaa Elgendi, PhD Operations Research and Decision Support Faculty of Computers and Artificial Intelligence Cairo University Supervised Learning Neural Network Textbooks Zurada, J.M. (1992) Introduction to Artificial Neural Systems. West Publishing Company, St. Paul. Others… Jacek M. Zurada serves as a Professor of Electrical and Computer Engineering Department at the University of Louisville, History Of Neural Networks ANN is a computing technique designed to simulate the human brain’s method in problem-solving. In 1943, McCulloch, a neurobiologist, and Pitts, a statistician, published a seminal paper titled “A logical calculus of ideas immanent in nervous activity” in Bulletin of Mathematical Biophysics, where they explained the way how brain works and how simple processing units—neurons—work together in parallel to make a decision based on the input signals. The similarity between artificial neural networks and Warren McCulloch and Walter Pitts the human brain is that both acquire the skills in processing data and finding solutions through training  History Of Neural Networks The perceptron, a revolutionary algorithm created to handle complicated recognition tasks, was invented by Cornell University's Frank Rosenblatt in 1957, accelerating work in the field. The lack of computer Frank Rosenblatt capacity required to analyze vast volumes of data slowed progress in the four decades that followed. In the 2000s, computer scientists finally got what they needed, owing to increased processing power and sophisticated hardware, as well as the availability of enormous data sets from which to draw, and neural networks and AI took off, with no end in sight. Consider that 90% of internet data has been produced since 2016. Because of the Internet of Things' rapid expansion, this rate will continue to rise (IoT).  From biological neuron to artificial neuron (perceptron) The human brain consists of about 1011 Artificial neural network consists of simple computing units “neurons” working in computing units “artificial neurons,” and parallel and exchanging information each unit is connected to the other units through their connectors “synapses”; these via weight connectors; then, these units neurons sum up all information coming calculate the weighted sum of the coming into them, and if the result is higher than inputs and find out the output using the given potential called action potential squashing function or activation function. (threshold), they send a pulse via axon to the next stage. Synapse Synapse Dendrites Axon Axon Soma Soma Dendrites Synapse + 1, if X   n X =  xi wi Y = i =1 − 1, if X   Brain Vs. ANN Consider humans: Neuron switching time ~ 0.001 second Number of neurons ~ 1011 Connections per neuron ~ 104-5 Scene recognition time ~ 0.1 second 100 inference steps doesn't seem like enough → much parallel computation Properties of artificial neural nets (ANN) Many neuron-like threshold switching units Many weighted interconnections among units Highly parallel, distributed processes Artificial Neural Networks Artificial Neural Network (ANN): is a machine learning approach that models human brain and consists of a number of artificial neurons that are linked together according to a specific network architecture. Neuron in ANNs tend to have fewer connections than biological neurons. each neuron in ANN receives a number of inputs. An activation function is applied to these inputs which results in activation level of neuron (output value of the neuron). Knowledge about the learning task is given in the form of examples called training examples.  Applications of Artificial Neural Networks Some tasks to be solved by Artificial Neural Networks: ❖ Classification: Linear, non-linear. ❖ Recognition: Spoken words, Handwriting. Also recognizing a visual object: Face recognition. ❖ Controlling: Movements of a robot based on self perception and other information. ❖ Predicting: Where a moving object goes, when a robot wants to catch it. ❖ Optimization: Find the shortest path for the TSP.  Artificial Neural Networks Before using ANN, we have to define: 1. Artificial Neuron Model. 2. ANN Architecture. Single-layer feedforward neural network Multilayer feedforward neural network Associtative memory Radial Basis NN … 3. Learning Mode. Perceptron d y =  w j x j + w0 = w T x j =1 w = w 0 , w1 ,...,wd T x = 1, x1 ,..., xd  T (Rosenblatt, 1962) 13 What a Perceptron Does Regression: y=wx+w0 Classification:y=1(wx+w0>0) y y s y w0 w0 w w x w0 x x x0=+1 1 y = sigmoid (o ) =  1 + exp − w T x  18 Two Classes g(x ) = g1 (x ) − g2 (x ) = (w1T x + w10 ) − (w T2 x + w 20 ) = (w1 − w 2 )T x + (w10 − w 20 ) = w T x + w0 C1 if g(x )  0 choose  C 2 otherwise 19 Geometry 20 Multiple Classes gi (x | w i , w i 0 ) = w Ti x + w i 0 Choose C i if K gi (x ) = max g j (x ) j =1 Classes are linearly separable 21 Pairwise Separation gij (x | w ij , w ij 0 ) = w Tij x + w ij 0  0 if x  C i  gij (x ) =  0 if x  C j don' t care otherwise  choose C i if j  i , gij (x )  0 22 An Illustrative Example Activation Functions Usually, do not just use weighted sum directly. Apply some function to the weighted sum before it is used (e.g., as output). → Call this the activation function.  Activation Functions ▪ The choice of activation function determines the Neuron Model.  Computing with Neural Units Incoming signals to a unit are presented as inputs. How do we generate outputs? One idea: Summed Weighted Inputs. Input vector: XT = [3 1 0 -2]T Weight vector: WT = [0.3 -0.1 2.1 -1.1]T Processing with linear activation function X. W = 3(0.3) + 1(-0.1) + 0(2.1) + -2(-1.1) = 0.9 + (-0.1) + 0 + 2.2 = 3 Output: 3 Example (1): Step Function Example (2): Another Step Function Example (3): Sigmoid Function The math of some neural nets requires that the activation function be continuously differentiable. → A sigmoidal function often used to approximate the step function. Example (3): Sigmoid Function Example Calculate the output from the neuron below assuming a threshold of 0.5: Sum = (0.1 x 0.5) + (0.5 x 0.2) + (0.3 x 0.1) = 0.05 + 0.1 + 0.03 = 0.18 Since 0.18 is less than the threshold, the Output = 0 Repeat the above calculation assuming that the neuron has a sigmoid output function: Single Layer Neural Network Multilayer Neural Network Multi Layer Neural Network More general network architecture, where there are hidden layers between input and output layers. Hidden nodes do not directly receive inputs nor send outputs to the external environment. Multi Layer NN overcome the limitation of Single-Layer NN, they can handle non-linearly separable learning tasks.  XOR No w0, w1, w2 satisfy: w0 0 (Minsky and Papert, 1969) w 2 + w0 0 w1 + w0 0 w1 + w 2 + w 0 0 41 x1 XOR x2 = (x1 AND ~x2) OR (~x1 AND x2) 42 Neural Network Architectures Even for a basic Neural Network, there are many design decisions to make: 1. # of hidden layers (depth) 2. # of units per hidden layer (width) 3. Type of activation function (nonlinearity) 4. Form of objective function  ANN Capabilities & Limitations Main capabilities of ANN includes: ✓ Learning. ✓ Generalization capability. ✓ Noise filtering. ✓ Parallel processing. ✓ Distributed knowledge base. ✓ Fault tolerance. Main problems includes: ❖ Learning sometimes difficult/slow. ❖ Limited storage capability. Questions Thank you