Lec 01.pdf

Introduction to Soft Computing and Neural Networks IT3071 – Machine Learning and Optimization Methods Introduction to the Module This is a compulsory unit for the 3rd year students who follow BSc (Hons) in IT specializing in the Data Science program. The objective of t...

Introduction to Soft Computing and Neural Networks IT3071 – Machine Learning and Optimization Methods Introduction to the Module This is a compulsory unit for the 3rd year students who follow BSc (Hons) in IT specializing in the Data Science program. The objective of this unit is to get an in-depth understanding of both the theory and practices in the field of Neural Networks and Optimization methods. In this unit, the students will learn: Fundamentals of Neural Networks Multilayer Neural Networks Recurrent Networks Convolutional Neural Networks Support vector machines Linear Programming Genetic Algorithms Assessments Assignment = 50% Based on practical sessions and independent research Final exam = 50% A written examination based on the lectures, tutorials, reading materials and practical work covered during the semester. Primary references F. Chollet, "Deep learning with Python”,Simon and Schuster, 2021. J. Sen, S. Mehtab, and A. Engelbrecht, "Machine Learning: Algorithms, Models and Applications," BoD–Books on Demand, 2021. P. E. Andries, "Computational intelligence: an introduction," John Wiley, 2020. Soft vs Hard Computing Soft computing and hard computing are two different paradigms for problem-solving in computer science and artificial intelligence. They each have distinct characteristics and are suited to different types of problems. Applications Hard Computing: Suitable for tasks requiring high precision and clear logical solutions, such as scientific computing, cryptography, and traditional software development. Soft Computing: Ideal for tasks where human-like reasoning and adaptability are needed, such as autonomous driving, Soft Computing The use of inexact solutions to computationally hard tasks, when there are no known algorithms that can compute the exact solution in polynomial time. Tolerant of imprecision, uncertainty, partial truth, and approximation. Fuzzy Logic (FL), Evolutionary Computation (EC), Machine Learning (ML) and Probabilistic Reasoning (PR)  Domain called “Artificial Neural Networks” motivated by how the human brain What is a computes in an entirely different way Neural from the conventional digital computer. Network? The brain is a highly complex, nonlinear, and parallel computer It has the capability to organize its structural constituents, known as neurons Neurons perform certain computations many times faster than the fastest digital computer in existence today. e.g., recognizing a familiar face embedded in an unfamiliar scene in approximately 100–200 ms, whereas tasks of much lesser complexity take a great deal longer on a powerful computer  A neural network is a massively parallel distributed processor made up of simple processing units What is a It has a natural propensity for storing Neural experiential knowledge and making it available for use. Network? It resembles the brain in two respects: cont. Knowledge is acquired by the network from its environment through a learning process. Interneuron connection strengths, known as synaptic weights, are used to store the acquired knowledge. The procedure used to perform the learning process is called a learning algorithm Learning algorithm is a function of which is to modify the synaptic weights of the network in an orderly fashion to attain the desired design objective  The Human Nervous System Brain is represented by the Arrows from left to right The receptors convert The effectors convert neural (nerve) net, which indicate the forward stimuli from the human electrical impulses continually receives transmission of body or the external generated by the neural information, perceives it, information-bearing environment into net into discernible and makes appropriate signals through the system. electrical impulses that responses as system decisions. The arrows from right to convey information to the outputs left (shown in blue) signify neural net (brain). the presence of feedback in the system.  Neurons are the structural constituents of the brain. Neurons are five to six orders of The magnitude slower than silicon logic Human gates. Brain However, the brain makes up for the relatively slow rate of operation of a neuron by having a truly staggering number of neurons (nerve cells) with massive interconnections between them. It is estimated that there are approximately 10 billion neurons in the human cortex, and 60 trillion synapses or connections. The net result is that the brain is an enormously efficient structure.  Synapses, or nerve endings, are elementary structural and functional The units that mediate the interactions between neurons. Human Brain In an adult brain, plasticity may be accounted by two mechanisms: the creation of new synaptic connections between neurons, and the modification of existing synapses. Axons - the transmission lines Dendrites - the receptive zones Neurons come in a wide variety of shapes and sizes in different parts of the brain. Biological Neuron Biological Neuron  Nonlinearity Input-output mapping Some Adaptivity Properties of Neural Fault tolerance Networks VLSI implementability Uniformity of analysis and design Model of a Neuron m uk =  wkj xk yk =  (uk + bk ) j =1 Model of a Neuron cont.  Types of Activation Function Threshold Function Sigmoid Function The sigmoid function, whose graph is “S”-shaped, is by far the most common form of activation function used in the construction of neural networks. It is defined as a strictly increasing function that exhibits a graceful balance between linear and nonlinear behavior.  Network Architectures The manner in which the neurons of a neural network are structured is intimately linked with the learning algorithm used to train the network. We may therefore speak of learning algorithms (rules) used in the design of neural networks as being structured. In general, there are three fundamentally different classes of network architectures: Single-Layer Feedforward Networks Multilayer Feedforward Networks Recurrent Networks  Single-Layer Feedforward Networks In a layered neural network, the neurons are organized in the form of layers. In the simplest form of a layered network, we have an input layer of source nodes that projects directly onto an output layer of neurons (computation nodes), but not vice versa. In other words, this network is strictly of a feedforward type. “single-layer” refers to the output layer of computation nodes (neurons). We do not count the input layer of source nodes because no computation is performed there. In the network shown in the diagram, there ae 4 input nodes and 4 computational nodes (neurons).  Multilayer Feedforward Networks One or more hidden layers, whose computation nodes are correspondingly called hidden neurons or hidden units are present. The function of hidden neurons is to intervene between the external input and the network output in some useful manner. By adding one or more hidden layers, the network is enabled to extract higher-order statistics from its input.  Multilayer Feedforward Networks In a rather loose sense, the network acquires a global perspective despite its local connectivity, due to the extra set of synaptic connections and the extra dimension of neural interactions The network shown in the figure is a 10-4-2 network  Recurrent Networks Contains feedback loops. The presence of feedback loops has a profound impact on the learning capability of the network and on its performance. Nonlinear dynamic behavior, assuming that the neural network contains nonlinear units.  Knowledge Representation Knowledge refers to stored information or models used by a person or machine to interpret, predict, and appropriately respond to the outside world A major task for a neural network is to learn a model of the world (environment) in which it is embedded and to maintain the model sufficiently consistent with the real world so as to achieve the specified goals of the application of interest. Knowledge of the world consists of two kinds of information: Prior information Observations (measurements) Knowledge Representation Observations are inherently noisy, being subject to errors due to sensor noise and system imperfections. In any event, the observations so obtained provide the pool of information, from which the examples used to train the neural network are drawn. Examples may be labelled or unlabelled Labelled examples are expensive as they require a “teacher”  Discussion Consider the handwritten-digit recognition problem. In this problem, the input signal consists of an image with black or white pixels, with each image representing one of 10 digits. Typically, the training sample consists of a large variety of handwritten digits that are representative of a real-world situation. Given such a set of examples, what are the steps involved in designing a neural network to recognize hand written digits? Learning Processes Learning with a teacher (supervised learning) Learning Processes cont. Learning without a teacher there is no teacher to oversee the learning process. That is to say, there are no labeled examples of the function to be learned by the network. Under this second paradigm, two subcategories are identified: Reinforcement learning Unsupervised learning