Deep Learning and Neural Networks

• Artificial neural networks are used to process and analyze information in deep learning, a subset of machine learning.
• Neural networks consist of computational nodes within deep learning algorithms.

Deep Learning vs Machine Learning

• Deep learning automates feature extraction reducing human intervention.
• Traditional machine learning often requires engineers to manually identify features, classifiers, and adjust algorithms.

Advantages of Deep Learning

• Achieves state-of-the-art performance in tasks like image recognition and natural language processing.
• Automatically identifies relevant features from data, eliminating manual feature engineering.
• Deep learning models can handle large and complex datasets, scaling effectively and learning from massive data amounts.
• Can be applied to a wide range of data types, including images, text, and speech.
• Continuously improves its performance as more data becomes available.

Challenges in Deep Learning

• Deep learning requires large amounts of data.
• Training requires specialized and expensive hardware like GPUs and TPUs.
• Processing sequential data is time-consuming, potentially taking days or months to complete.
• The complex nature of deep learning models makes it difficult to interpret results.
• Overfitting will cause poor performance on new data.

Neural Networks

• They mirrors the structure and function of human neurons and are also referred to as "neural nets."

Perceptron

• A mathematical representation of a biological neuron where electrical signals are represented as numerical values, in contrast to actual neurons where dendrites receive electrical signals from axons.
• At the synapses, electrical signals are modulated; this is modeled in the perceptron by multiplying each input value by a weight.
• Fires an output signal only when the strength of the input signals surpasses a threshold. Modeled by calculating the weighted sum of inputs and applying a step function to determine output.
• The output of a perceptron feeds into other perceptrons, mirroring biological neural networks.

Components of a Perceptron

• Consists of various inputs (x1, x2,..., xn).
• Each input is associated with a weight (w1, w2,..., wn).
• A bias term (b) is included to adjust the decision boundary.
• Uses a step function to determine if the weighted sum of inputs plus the bias is above or below a certain threshold.
• The mathematical representation is y = 1 if ∑(wixi) + b > 0 or 0 if ∑(wixi) + b ≤ 0.
• Delivers a binary output (1 or 0), suitable for linearly separable classification problems.

Types of Activation Functions

• Binary Step Function: Activation depends on whether a threshold value is reached.
• Linear Activation Function: Also known as "no activation" aka an "identity function", activation is proportional to the input.
• Non-linear Activation Functions: Solve the limitations of linear functions.
  • Enable backpropagation by relating derivative function to the input
  • Enable stacking of multiple layers of neurons, and any output can be represented as a functional computation in a neural network.

Shallow Neural Network

• Composed of an input layer, hidden layer, and output layer.
• Computations occur for all neurons, with each pass known as forward propagation.
• The output layer compares its results to the ground truth labels, adjusting weights based on differences, in a backward pass called backpropagation.

Backpropagation

• The network minimizes an objective function, such as the error across all points in a data sample.
• At the output layer, the network calculates error, and its derivative with respect to weights, which is known as the gradient of that layer.
• Weights for that layer are updated based on the gradient.
• The learning rate impacts the step size to the weights.
• The process is repeated until the first layer is reached.
• Values of gradients from previous layers can be reused to efficiently compute the gradient.

Parameters vs Hyperparameters

• Parameters are adjusted during training, representing relationships in the data used for predictions.
• Hyperparameters are set before training, governing algorithm behavior, such as the learning rate or regularization strength.

Loss Function

• Compares target and predicted outputs, measuring training data modeling by the neural network.
• The training aims to minimize the loss between predicted and target outputs.
• A local minimum is a point where the loss function is minimized within a neighborhood.
• A global minimum represents a point where the loss function is minimized across the entire parameter space.

Optimizers

• Essential algorithms that fine-tune a model’s parameters during training to minimize a function.
• These algorithms enable learning by refining weights and biases based on data feedback.
• Common optimizers include Stochastic Gradient Descent (SGD), Adam, and RMSprop, and each has unique update rules, learning rates, and momentum strategies.

Batches

• Batch size is a hyperparameter controlling how many training samples are worked through before the model updates its parameters.
• A training dataset is divided into batches.
• Batch Gradient Descent: Batch Size equals the size of the Training Set.
• Stochastic Gradient Descent: Batch Size equals 1.
• Mini-Batch Gradient Descent: Batch Size is between 1 and the size of the Training Set.

Epochs

• A hyperparameter that specifies the number of complete passes through the training dataset during gradient descent.
• One epoch means dataset samples have had the chance to update model parameters.