Smart Systems Lec 7 PDF

# Smart Systems and Computational Intelligence ## Lecture 7 Smart Systems and Computational Intelligence Course ### Neural Networks - The brain is a collection of about 10 billion interconnected neurons. Each neuron is a cell that uses biochemical reactions to receive, process, and transmit infor...

# Smart Systems and Computational Intelligence ## Lecture 7 Smart Systems and Computational Intelligence Course ### Neural Networks - The brain is a collection of about 10 billion interconnected neurons. Each neuron is a cell that uses biochemical reactions to receive, process, and transmit information. - Each terminal button is connected to other neurons across a small gap called a synapse. - A neuron's dendritic tree is connected to a thousand neighboring neurons. When one of those neurons fires, a positive or negative change is received by one of the dendrites. The strengths of all the received charges are added together through the processes of spatial and temporal summation. ### The Biological Neuron - A schematic of a biological neuron is shown. It includes the axon hillock, soma, dendrites and nucleus. The synapse is also shown, which consists of the terminal button, synaptic gap and dendrites. ### How Neural Networks are Arranged - A diagram of how a neural network is arranged is shown with dendrites, axon and a cell body. ### What are Neural Networks? - **Neural Networks (NNs)** are networks of neurons, as found in real (i.e. biological) brains. - **Artificial Neurons** are crude approximations of the neurons found in brains. They may be physical devices or mathematical constructs. ### Artificial Neural Networks (ANN) - It can be said that the ANN is a nature-inspired structure, so it does have similarities with the neural network of the human brain. - The artificial neuron has a similar structure, in which it contains a processing unit (nucleus), inputs (dendrites), and an output (axon). - From a practical point of view, an ANN is just a parallel computational system consisting of many simple processing elements connected together in a specific way to perform a particular task. - ANN can help to automate complex decision-making. - ANN learn from past experience and improve their performance levels. ### What are Artificial Neural Networks used for? - **Brain modeling:** The scientific goal of building models of how real brains work. This can potentially help us understand the nature of human intelligence, and formulate better teaching strategies, or better remedial actions for brain-damaged patients. - **Artificial System Building:** The engineering goal of building efficient systems for real-world applications, may make machines more powerful, relieve humans of tedious tasks, and may even improve upon human performance. ### Elements of Artificial Neural Networks - **Processing Units:** The form in which the neurons and layers are calculated together - **Topology:** The form in which the neurons and layers are connected together - **Learning Algorithm:** The way the connection weights are adapted to reflected the intended behaviour based on sample data ### Single-Input Neuron - A diagram representing the inputs, weights and output of a single-input neuron is shown. The equation for the neuron output, $a = f(wp+b)$, is also shown. An example is given using a specific value for the input, weight and bias. ### Multiple-Input Neuron - Neural computing requires a number of neurons, to be connected together into a neural network. Neurons are arranged in layers. - A diagram is shown with a multiple-input neuron, including the inputs ($P_1, P_2, P_3$), weights ($W_1, W_2, W_3$) and output ($a$). The equation for the neuron output is also shown: $a = f(p_1w_1 + p_2w_2 + p_3w_3 + b)$. - Each neuron within the network is usually a simple processing unit which takes one or more inputs and produces an output. At each neuron, every input has an associated **weight** which modifies the strength of each input. The neuron simply adds together all the inputs and calculates an output to be passed on. ### Neuron with R Inputs (Abbreviated Notation) - A diagram is shown with a neuron with R inputs, including the input ($p_{Rx1}$ with R rows), weights ($W_{1xR}$ with R columns) and output ($a_{1x1})$. The equation for the neuron output is also shown: $a=f(Wp+b)$. ### A Layer of Neurons - A diagram of a layer of neurons is shown. ### Layer of S Neurons (Abbreviated Notation) - A diagram is shown of a layer of S neurons with inputs ($p_{Rx1}$ with R rows), weights ($W_{SXR}$ with R columns) and output ($a_{Sx1}$ with S rows). The equation for the neuron output is also shown: $a = f(Wp+b)$. ### Topology - Diagram of the topology of neural networks is shown. It shows the different types of networks: Neural networks, Feed forward networks, Recurrent / Feedback networks, Single-layer perceptron, Multilayer Preceptron , Radial Basis Function Nets, Competitive networks, Kohonen's SOM, Hopfield Network and ART models. ### Learning Algorithm - Learn the connection weights from a set of training examples. - **Supervised Learning:** The network is provided with a correct answer (output) for every input pattern; weights are determined to allow the network to produce answers as close as possible to the known correct answers; the back-propagation algorithm belongs to this category. - **Unsupervised Learning:** Does not require a correct answer associated with each input pattern in the training set; explores the underlying structure in the data or the correlations between patterns in the data; organizes patterns into categories from these correlations; the Kohonen SOM algorithm belongs into this category. ### Monolayer and Multilayer Networks - Diagram shows a monolayer network with inputs and outputs. - Diagram shows a multilayer network with input layer, hidden layers, and output layer. ### A single-layer neural net - Diagram of a single-layer neural net is shown with the input units, weights and output units. ### The Training Algorithm of the SLP Model - **Step 1:** Initialize all weights and thresholds Set Wij(0) and Θj to small values, where Wij(0) is the weight from input node *i* to output node *j* at time 0 and Θj is the threshold at output node *j*. - **Step 2:** Present new input *X* and desired output *D* X = (X1, ..., Xn), D = (D1, ..., Dm). - **Step 3:** Calculate actual output *Y* as follows: $Y_j(t) = f(\sum_{i=1}^{n} W_{ij}(t)X_i - \Theta_j), 1 \leq j \leq m$. ­­­­ - **Step 4:** Adapt weights as follows $W_{ij}(t + 1) = W_{ij}(t) + \eta [D_j(t) - Y_j(t)]X_i(t)$. Where: η is a positive gain factor less than 1. - **Step 5:** If the current example is not classified, go to Step 3. - **Step 6:** Repeat by going to Step 2 for all examples. ### A multilayer neural net. - Diagram shows the input and output layers of a multilayer neural net. ### Multiple Layers of Neurons - Diagram shows multiple layers of neurons. The inputs and outputs of the neurons are shown. The equation for the neuron output is also shown in each layer. ### Feedforward Networks - The flow of the signals in neural networks can be either in only one direction or in recurrence. - In the first case, we call the neural network architecture feedforward, since the input signals are fed into the input layer; then, after being processed, they are forwarded to the next layer. - Multilayer perceptron nets and radial basis functions nets are also good examples of feedforward networks. ### Feedback Networks - When the neural network has some kind of internal recurrence, it means that the signals are fed back in a neuron or layer that has already received and processed that signal, the network is of the type feedback. A diagram of a feedback network is shown with input, hidden and output layers. ### The Back-Propagation Training Algorithm - **Step 1:** Initialize all weights and thresholds Set Wij(0), Θj to small values, where Wij (0) is the weight from input node *i* to output node *j* at time 0, Θj is the threshold in output node *j*. - **Step 2:** Present new input *X* and desired output *O* X = (X1, ..., Xn), O = (O1, ..., Om) - **Step 3:** Calculate actual output *Y* as follows: $Y_j (t) = f(\sum_{i=1}^{n} W_{ij}(t)X_i - \Theta_j), 1 \leq j \leq m$. - **Step 4:** Adapt weights as follows $W_{ij}(t + 1) = W_{ij}(t) + \eta Y_j(1-Y_j)(O_j-Y_j)X_i$ - **Step 5:** Repeat by going to Step 3 if *E* is still large. - **Step 6:** Go to Step 2 for the next example. ### The Learning Process - Neural networks learn by adjusting the connections between the neurons, namely the weights. - Weights represent the neural network knowledge and different weights cause the network to produce different results for the same inputs. - So, a neural network can improve its results by adapting its weights according to a learning rule. A diagram is shown of a neural network with inputs, outputs and weights, showing the learning process. ### Designing a Neural Network Application - **Steps to follow:** 1. Choose the architecture: number of layers, number of units on each layer, activation functions, interconnection style. 2. Train the network: compute the values of the weights using the training set and a *learning algorithm*. 3. Validate/test the network: analyze the network behavior for data that do not belong to the training set. ### Models of Neural Networks - Models of neural networks are varied and numerous, and at least 100 different types are being explored in research or being developed for applications. ### Common Models of Neural Networks | Model | Training Mode | Limitations | Primary Use | |--------------------------------------|----------------|-------------------------------------------|-------------------------------| | Single-Layer Perceptrons (SLP) | Supervised | No complex mapping between i/o pairs | To distinguish hyper plane separable patterns | | Multi-Layer Perceptrons (MLP) | Supervised | Requires large number of iterations to converge | When correct i/o examples are abundant | | Self Organizing Map (SOM) | Unsupervised | Requires extensive training | Clustering of input patterns | | Hopfield | No Training | Weights must be set in advance using training patterns | Used as an associative memory | | Generalized Regression (GRNN) | Supervised (one pass) | A pattern-layer node is needed for each example | Performs a general regression statistical model | ### Neural Networks - Neural Networks are often classified as single-layer or multilayer. In determining the number of layers, the input units are not counted as a layer. Equivalently, the number of layers in the network can be defined to be the number of layers of weighted interconnected links between neurons. - Single-layer network has one layer of connection weight. Often, the units can be distinguished as input units, which receive signals from the outside world, and output units. - The input units are fully connected to output units but are not connected to other input units. - The output units are not connected to other output units. ### Neural Network as a Classifier - **Strength:** - High tolerance to noisy data - Ability to classify untrained patterns - Well-suited for continuous-valued inputs and outputs - Successful on an array of real-world data, e.g., hand-written letters - Algorithms are inherently parallel - Techniques have recently been developed for the extraction of rules from trained neural networks - **Weakness:** - Long training time - Require a number of parameters typically best determined empirically, e.g., the network topology or "structure". - Poor interpretability: Difficult to interpret the symbolic meaning behind the learned weights and of "hidden units" in the network ## Quiz Suppose the inputs are X1=0.6, X2=0.1, and the weights are set as in the picture. Calculate the output $O_6$ and $O_7$ assuming a Sigmoid Squashing Function. Hint: Use sigmoid Function (1/(1+e^(-X))) to get each X. - Diagram shows a neural network ## Quiz Answer - **Activations of the hidden units:** Put X1=0.6 & X2=0.1 $X_3=X_1*0.1+X_2*-0.2+0.1=0.14$ $X_4=X_1*0+X_2*0.2+0.2=0.22$ $X_5=X_1*0.3+X_2*-0.4+0.5=0.64$ - **Use sigmoid Function (1/(1+e^(-X))) to get each X:** New $X_3=0.53$ New $X_4=0.55$ New $X_5=0.65$ - **Activations of the output units:** Put New $X_3=0.53$, New $X_4=0.55$ & New $X_5=0.65$ $X_6=New X_3*-0.4+New X_4*0.1+New X_5*0.6+(-1) = 0.13$ $X_7= New X_3*0.2+New X_4*-0.1+New X_5*-0.2+0.6=0.52$ - **Use sigmoid Function (1/(1+e^(-X))) to get each X:** New $X_6=0.53$ New $X_7=0.63$ ## Any Questions??? ## Thanks