Multi-Layer Neural Networks

• This lecture covers Artificial Neural Networks (ANNs), focusing on multi-layer neural networks, covering the limitations of perceptrons, deep neural networks, and training with gradient descent.
• The instructor is Siman Giri, the Module Leader for 6CS012.

Learning Objectives

• Review the limitations of perceptrons discussed in the previous week.
• Overcome the limitations of perceptrons and the Perceptron Learning Algorithm.
• Discuss Deeper Neural Networks (also called Multi-Layer Perceptrons).
• Understand the basics and architecture of Deep Neural Networks.
• Train a Deep Neural Network with Gradient Descent for multi-class classification problems.

Recap of Challenges with Simple Perceptrons

• Challenge 1: Automating feature extraction.
  • High dimensionality results when using pixel values directly (e.g., 784 columns for 28x28 images).
  • Datasets can become very large for larger images.
• Challenge 2: Non-linear decision boundaries.
  • Logistic Regression or Softmax Regression are better at separating linearly separable data.
  • Standard perceptrons fail when classes/labels are not linearly separable

Perceptron and its Limitations

• The perceptron is a fundamental component of modern neural network architectures, often referred to as neurons.
• Perceptrons are often single hidden units connected to some input x.
• It computes the dot product of a weight vector w and input x, adding a bias term, and passing the result through an activation function.
• The activation function determines if the perceptron/neuron activates based on the sum of weighted inputs.
• The original perceptron using unit step function as activation function is not ideal for most modern problems.

Why Not a Unit Step Function

• The unit step function is defined as f(z) = 1 if z ≥ 0, else f(z) = 0, where z = wo + (wTx).
• Unit step functions do not capture subtleties of the data and create a sharp boundary at 0.
  • It implies all positive sums classify as class 1, and all negative sums classify as class 0.
  • It does not accommodate cases where negative values can belong to class 1.
• The output is binary only, either 0 or 1, limiting nuanced decision boundaries.
• The small changes in input do not affect output, limiting the model's ability to learn complex patterns.
• The non-differentiability of the function is incompatible with gradient-based optimization, such as backpropagation in neural networks.

Activation Function

• Introduces non-linearity to the output of neurons.
• Good activation function properties include:
  • Non-linearity to learn complex patterns and decision boundaries.
  • Differentiability to enable gradient-based optimizations (i.e., gradient descent).
  • Computational efficiency for real-time applications.
  • Gradient behavior to avoid vanishing or exploding gradients.

Sigmoid Neurons

• A type of artificial neuron using the sigmoid activation function to model the output of the neuron.
• The sigmoid function is a smooth, S-shaped curve that maps any real-valued input to a value between 0 and 1.
  • It makes it useful for problems involving probabilities or binary classification.
• The sigmoid activation function is defined as σ(z) = 1 / (1 + e^(-z)), where z = wo + Σ Wixi is the weighted sum of the inputs to the neuron.

Limitations of the Perceptron

• Perceptrons are binary classifiers that work well when data is linearly separable.
• Performance can be lower when data isn't linearly separable. -Arranging perceptrons in multiple layers makes XOR and similar problems solvable.
• The "Theory of Universal Approximation" backs up the idea that multiple perceptrons/neurons in layers can learn non-linearly separable functions, such as XOR.

Theory of Universal Approximation

• States that by arranging neurons in layers and using nonlinear activation functions, neural networks can represent any non-linear function.
• With this in mind, neural networks become powerful for function approximation, pattern recognition, and deep learning applications.
• Any boolean function can be implemented using a one-hidden-layer perceptron network with 2^n neurons, where n is the number of inputs.

Challenges of Single Layer Neural Networks

• While a single-layer perceptron network can represent any function, real-world functions require many neurons for a good approximation, creating inefficiency.
• A single layer would require millions of neurons: making computing inefficient in image recognition (e.g. dog vs cat classification).
• Multi-layer (aka deeper) neural networks are preferred in practice.

Architecture of One Hidden Layer Neural Network

• The input layer functions as a placeholder and involves no computation. Understanding the Architecture x = [x1, x2, x3, x4].
• The computation starts in the first hidden layer, propagating forward to the next layer.
• The hidden layer computation is z¹ = (W¹x) + b¹ and a¹= (f¹(z¹)).
  • W1 is a weight vector, and there are 4 neurons per layer.
• Computed output after activation function f¹ yields a¹, using sigmoid (σ).

Architecture of One Hidden Layer Neural Network - Output

• Neurons in the output layer can also have different activation functions: depending on the classification task.
• Binary uses a sigmoid activation function, and multi-class uses a softmax activation function.
• For demonstration, the setup utilizes a single neuron with a sigmoid activation function.

Limitations of Perceptron Learning Algorithm

• Perceptron Learning Algorithm is designed for single-layer perceptrons or one neuron.
  • It applies to problems that are linearly separable only.
• The algorithm adjusts weights iteratively based on the prediction error, updating when the model is incorrect.
• A key limitation of the perceptron learning algorithm is the inability to propagate errors to middle (hidden) layers: preventing it from solving nonlinear classification issues.

Towards Multi-Layer (Deep) Neural Networks

• A solution requires an updated learning algorithm that can both feedforward and backpropagate the adjusted weights through the hidden layers.
• This raises the question: Can Gradient Descent algorithm be used?

From Single Layer to Shallow Network

• Single Layer Neural Networks
  • Networks with one hidden layer and 2n neurons.
  • Theoretically represent any function, but require many neurons: less computationally efficient.
• Shallow Neural Network Concept
  • Aims to avoid exponentially large number of neurons by distributing the computation across multiple hidden layers with fewer neurons per layer.
• A shallow neural network typically consists of one or a few hidden layers, reducing the need for 2^n neurons in a single layer while capturing complex patterns

From Shallow to Deep Neural Network

• Limitations of Shallow Neural Networks
  • Universality: While shallow multi-layer perceptrons (MLP) can approximate any continuous function with enough neurons, neuron quantity can be impractically large.
  • Expressivity: A small number of neurons might not capture complex, non-linear decision boundaries needed for classification problems.
  • Learnability: There's no guarantee that the learning method will find the optimal solution.

Deep Neural Networks (DNN)

• Definition: Any network with more than 2 hidden layers.
• Motivation:
  • Deeper networks can represent more complex patterns. Statistical: Inherently compositional; combine simple patterns recursively. Computational and Expressiveness: More expressive than shallow ones; learn more complex patterns with the same total parameters.

Understanding the Architecture of DNN

• Multi-layer Fully Connected Neural Network with some weight to every edge. -Every neurons are connected to each other.
• Individual neurons conduct a weighted sum and apply an activation function The Number of Neurons
• The quantity of neurons determined by the size of input image
  • The input layer would be just a placeholder. Number of neurones for MNIST with image with shape (28 × 28 × 1) have 784 neurons (28281 = 784),
• The number of hidden layers {Lk} and neurons per layer {k₁: kth layer n neurons} are hyperparameters determined by the designer.
  • {It is important to note increasing the neurons helps improve capacity, but cause overfitting}
  • {Decreasing the neurons underfit}

Output Layer Number of Neurons

• Depends on classification task and activation function used
  • Binary Classification Neuron= 1 with sigmoid function applied -Multilcass Classification Number of Neurons= The number of classes/
  • A Softmax Activation function is applied

Training the DNN

• Key Question: How to learn the weights and biases in Deep Neural Networks (DNNs)?
• Training is done via an Empirical Risk Minimization (ERM) Framework.
  • Requires both a Decision Process, Loss Function, Empirical Risk and an Optimization method for training.

Defining the DNN as a Model

• Unlike traditional machine learning models, DNNs are mathematical representations of deep neural networks with multiple layers.
• The models vary in the number of layers, number of neurons and general structure.
• The architecture mainly depends on the nature of the task and the nature of the data.
• For MNIST digit classification, the first layer has input placeholders, hidden layers can have 64, 128 and 512 neurons, and the output layer has 10 neurons for prediction.

Training the DNN within the ERM Framework

• Objective of DNNs with Softmax Output
  • Minimize average loss over the training dataset that can be formulated as L(W, b) = −(1/n) Σ Σ yik log(ŷik). Finding weight factors w and b that minimizes average-Log Loss over the data

Loss Function: Categorical Cross Entropy

• Categorical Cross-Entropy Loss measures how well the predicted probability distribution matches the true class labels in classification tasks.
  • It is used in multi-class classification problems.
• Goal: To maximize the probability of the correct class.
• For a single sample x₁ with true label yi and y^(predicted probabilities): l(yi, y^) = −Σ yik log(y^ik)

Computing Gradient

• The weights in Multi Layer networks are learned with the combinations of forward and backward propagations.
  • Forward Propagation (activation) and backward propagations (error determination for weight changes)

Backpropogation

• Backpropagation is a technique used by deep layer networks to find the error of the network through calculation and comparison
• There are three conditions (dataset, feedforware and Error function)
• The output from the input layer will be the dataset used in the Error function
• The derivatives will be calculated going forward- back and updates accordingly

Variants of the Gradient Decent

Here are three main types (dependent on data computation)

• Batch gradient decent
• Stochastic gradient decent
• Mini-Batch Stochastic Gradient Decent

Common Issue With The Gradiant Decent

• Same learning rate applies to all parameter updates.
• Getting trapped into suboptimal local minima or saddle points.
• Choosing a Proper Learning
• Note reducing and learning has to adapt to characteristics