Artificial Neural Networks (ANNs)

• Artificial Neural Networks (ANNs) are computational models inspired by the structure and function of biological neural networks.
• ANNs consist of interconnected nodes called artificial neurons, which mimic the behavior of neurons in the human brain.
• Mathematical models of ANNs describe the relationships between these neurons and how they process information.

Neuron Model

• A single artificial neuron performs a weighted sum of its inputs, applies an activation function, and produces an output.
• The input to a neuron consists of signals from other neurons or external sources, represented as a vector x = [x1, x2, ..., xn].
• Each input xi is associated with a weight wi, representing the strength of the connection between the input and the neuron.
• The weighted sum of the inputs is calculated as net = Σ(xi * wi) for i = 1 to n.
• A bias term b is often added to the weighted sum to adjust the neuron's activation threshold: net = Σ(xi * wi) + b.
• The activation function f(net) introduces non-linearity, enabling the neuron to model complex relationships.
• Common activation functions include sigmoid (logistic), ReLU (Rectified Linear Unit), tanh (hyperbolic tangent), and step function.
• The output of the neuron is given by a = f(net), where 'a' represents the neuron's activation or output.

Network Architecture

• ANNs are organized into layers: an input layer, one or more hidden layers, and an output layer.
• The input layer receives the initial data. Each neuron in the input layer corresponds to one input feature.
• Hidden layers perform intermediate computations to extract relevant features and patterns from the input data.
• The output layer produces the final result or prediction of the network.
• Neurons in adjacent layers are connected by weighted connections.
• The connections between neurons can be feedforward (unidirectional) or recurrent (bidirectional).
• Feedforward networks, such as Multi-Layer Perceptrons (MLPs), have connections only in one direction (from input to output).
• Recurrent Neural Networks (RNNs) have feedback connections, allowing them to process sequential data and maintain memory.

Mathematical Representation of a Layer

• The output of a layer can be represented mathematically using matrix notation.
• Let X be the input matrix to a layer, where each row represents a sample and each column represents a feature.
• Let W be the weight matrix for that layer, where Wij represents the weight connecting the j-th neuron in the previous layer to the i-th neuron in the current layer.
• Let b be the bias vector for the layer.
• The net input to the layer is calculated as Net = X * W^T + b, where W^T is the transpose of the weight matrix.
• The output of the layer is given by A = f(Net), where f is the activation function applied element-wise.

Training Process

• Training an ANN involves adjusting the weights and biases to minimize the difference between the network's predictions and the actual target values.
• This optimization process is typically done using iterative algorithms like gradient descent.
• The loss function quantifies the error between the predicted output and the actual target.
• Common loss functions include Mean Squared Error (MSE) for regression tasks and Cross-Entropy Loss for classification tasks.
• The gradient of the loss function with respect to the weights and biases is calculated using backpropagation.
• Backpropagation involves propagating the error signal backward through the network to update the weights and biases.
• The update rule for the weights is given by W = W - α * dL/dW, where α is the learning rate and dL/dW is the gradient of the loss function with respect to the weights.
• Similarly, the update rule for the biases is given by b = b - α * dL/db, where dL/db is the gradient of the loss function with respect to the biases.
• The learning rate α controls the step size during optimization. Too large a learning rate can cause the algorithm to diverge, while too small a learning rate can lead to slow convergence.
• The training process continues until the loss function converges to a minimum or until a predefined stopping criterion is met (e.g., maximum number of iterations, desired accuracy).

Activation Functions

• Activation functions introduce non-linearity into the network, enabling it to model complex relationships between inputs and outputs.
• Sigmoid Function: f(x) = 1 / (1 + e^-x). It outputs values between 0 and 1, making it suitable for binary classification tasks and suffers from vanishing gradient problem.
• Tanh Function: f(x) = (e^x - e^-x) / (e^x + e^-x). It outputs values between -1 and 1, and is also susceptible to the vanishing gradient problem.
• ReLU (Rectified Linear Unit): f(x) = max(0, x). It outputs x if x > 0 and 0 otherwise, addresses the vanishing gradient problem to some extent, but can suffer from the "dying ReLU" problem where neurons become inactive.
• Leaky ReLU: f(x) = x if x > 0 and αx if x ≤ 0, where α is a small positive constant and attempts to address the dying ReLU problem by allowing a small gradient when the neuron is inactive.
• Softmax Function: Used in the output layer for multi-class classification problems. It converts a vector of real values into a probability distribution, where each value represents the probability of belonging to a specific class.

Key parameters

• Weights: Represent the strength of the connection between neurons which adjusted during training to minimize the loss function.
• Biases: Adjust the activation threshold of a neuron & also adjusted during training.
• Learning Rate: Controls the step size during the optimization process.
• Number of Layers: Determines the depth of the network.
• Number of Neurons per Layer: Determines the width of the network.
• Activation Function: Introduces non-linearity.
• Loss Function: Quantifies the error between predicted and actual values.

Advantages

• ANNs can model complex non-linear relationships.
• They can learn directly from data without explicit programming.
• They are capable of generalization, meaning they can make predictions on new, unseen data.
• They can handle noisy or incomplete data.

Disadvantages

• ANNs can be computationally expensive to train, especially for large networks and datasets.
• They can be prone to overfitting, where the network learns the training data too well and fails to generalize to new data.
• They can be difficult to interpret, making it challenging to understand why a network makes a particular prediction.
• They require careful selection of hyperparameters, such as learning rate, network architecture, and activation functions.