Neural Network Fundamentals

Fundamentals of Neural Networks (NN)

• A NN, or Neural Net, constitutes interconnected processing units called neurons.
• Artificial Neural Networks (ANNs) form an integral part of Artificial Intelligence and provide a foundation for Deep Learning.
• An ANN presents a computational architecture of neurons, which mathematically represents how a biological neural network identifies and recognizes relationships in data.
• ANNs are designed to mimic and simulate the functioning of the human brain and are built to replicate biological neurons using mathematical structures.
• ANNs simulate the way the human brain analyzes and processes information and solve complex problems.
• The concept of ANNs imitates the processes of natural neural networks.
• They aim to enable machines to understand, imitate, and act upon decisions the way a human brain does.
• The network fundamentals are connected via neurons or nodes, similar to the human brain.

Biological Neural Network (BNN) vs. Artificial Neural Network (ANN)

Biological Neural Network (BNN)

• Receives input through the five senses.
• Dendrites pass input.
• Neurons or nodes carry electrical impulses and transmit information to other nerve cells.
• Synapses facilitate neuron communication.
• Axons channel nerve impulses away from the cell body and output the information.

Artificial Neural Network (ANN)

• Input is collected from sources of data.
• "Wires," or connections, pass received inputs.
• Neurons consolidate information and make decisions.
• Weights, or interconnections, transform the input data within hidden layers.
• Output is produced by the neural network.

Components of a Neural Network

• The neural network structure is based on the problem's specifications and is configured for each specific application.

Layers

• Key layers include input, hidden, and output layers.
• The input layer gathers data such as text, images, audio, or video files, known as predictors.
• The output layer derives numerical values or classifications based on its input.
• The hidden layer derives features for the model, and can be a single layer or multiple layers.

Types of Neural Networks

• Single Layer Perceptron: A neural network with a single hidden layer.
• Multilayer Perceptron: A neural net with more than one hidden layer that connects each layer.

Neurons

• Neurons are the primary processing unit in neural networks that receive data and perform calculations.
• Neurons reside in the hidden and output layers, but aren't present in the input layer.
• The number of neurons in the hidden layers is chosen for the appropriate architecture.

Weights and Bias

• Inputs have connections with first layer of hidden neurons and continue to connect with the subsequent layers’ neurons.
• Weights and biases are applied during the transmission between the layers.
• Outputs of a layer become the inputs of another layer.
• Weights control the strength of connections, like synapses in biological networks.
• Bias is an additional, constant input independent of the preceding layer activating the model with a default value.
• Weights and biases are learnable parameters that are set up and optimized to minimize loss or error.

Activation Function

• This is applied to bring non-linearity and is also known as the ‘Squashing function.’

Activation Function Types

• Sign, Step, and Sigmoid are functions.
• Activation functions assist problem statements to create an output.

Perceptron Model

• Perceptron consists of input values (nodes), weights/bias, net sum, and activation functions in a single-layer neural network.
• The process involves multiplying input values/weights, totaling values to create the weighted sum, and applying this to an activation function to derive the desired output.

Single Layer Perceptron Model

• This is a simple Artificial Neural Network (ANN) with a feed-forward network and a threshold transfer function.
• The goal analyzes linearly separable objects with binary outcomes.
• Algorithms begin with allocated input, sum inputs (weight), activate model, and show the output value as +1 if the total exceeds the pre-determined value.
• Performance is satisfactory if the outcome aligns with the set threshold, with no alteration in weight demand.
• The model has discrepancies when multiple weight inputs are given.
• Changes is made for desired outputs.
• This type of perceptron learns linearly separable patterns exclusively.

Multi-Layered Perceptron Model

• It has the same model structure but with more hidden layers.

Multi-Layered Perceptron Stages

• Forward Stage: activation starts in the input layer and ends on the output layer.
• Backward Stage: Weights and bias values are modified.

Advantages

• Can be used to solve complex non-linear problems.
• Can train with small and large input data.
• Provides quick predictions and has a similar accuracy ratio with data.

Disadvantages

• Computations are difficult and time-consuming.
• Difficult to predict how dependent variables affect independent variables.
• Model functions depend on the quality of training.

Characteristics of Perceptron

• Algorithms learn through supervised learning for binary classifiers that automatically learn the weight coefficient.
• Neuron behavior is determined by multiplying weights with input features and made with several rules.
• Step rule applies if the weight function is positive.
• Added inputs over the threshold show an output signal.
• Otherwise, no output is shown.

Limitations of Perceptron Models

• Binary outputs are due to the hard limit transfer functions and are used to classify separable sets of input vectors.
• Hard too classify, if input vectors are non-linear.

Learning Methods

• A learning rule is a method or mathematical logic that improves performance through iteration.

Hebbian Learning Rule

• When neighboring neurons operate in phase, the weight increases, but decreases if out of phase.
• The weight remains unchanged when there is no correlation based on the sign of the input.
• Positive/negative inputs result in strong positive weights, while opposite inputs result in strong negative weights.
• This is a single-layer neural network that constantly updates based on training sets.

Hebbian Learning Algorithm Steps

• Set initial weights and biases to zero.
• Repeat activation setting and output for each input vector and pair.
• Update weight and bias by hebb rule.

Delta Hebbian Learning

• It depends on supervised learning and is a common rule.
• Modification in sympatric node weight is a product of error and input.
• For a given input vector, no learning happens with correct output; otherwise, weights are adjusted to reduce differences.
• The change in weight from neuron to neuron involves learning rate (ai) activations.
• The difference between expected and actual outputs is (ej).
• Error squared vs. weight graphs are parabolas with least values when used in networks.

Competitive Learning

• Competitive learning is unsupervised and known as the Winner-takes-All rule.
• Output nodes compete to represent the input pattern, with the winner given 1 and losers 0.
• Activation functions are then applied to a subset with only the winner updating its weight.

Error Correction Learning

• A technique of comparing the system output to the desired output with error to determine training.
• Error signals attempt to minimize error signals, most popularly being the backpropagation algorithm.

Boltzmann Learning

• Statistical and from thermodynamics that include individual neuron states during supervised training.
• Slower than error correction learning using "Boltzmann machines" and trains the systems iteratively by accounting for probability distributions.
• They use recurrent structure.

Boltzmann Machine Features

• Stochastic neurons have one of two states, either 1 or 0.
• Some neurons are adaptive (free state), while others are clamped (frozen).
• Discrete Hopfield Network would become a Boltzmann Machine if simulated annealing is applied.
• They have bidirectional connections with fixed weights, Wij, where connections exist if Wij != 0.
• There also exists a symmetry in weighted interconnection, i.e. Wij = Wji.
• Self-connections are implemented, and unit states are either at 1 or 0.
• Maximizing the Consensus Function (CF) and is the goal is defined between iterations i and j.