Neural Networks and Machine Learning Concepts

Introduction to Machine Learning

• Machine learning is a field of computer science that gives computers the ability to learn without explicit programming.
• Neural networks gained popularity in the 1980s.
• Support Vector Machines (SVMs), Random Forests, and Boosting became popular in the 1990s.
• Neural networks took a back seat until 2010, re-emerging as Deep Learning.
• Computing power, larger training sets, and improved software (e.g., TensorFlow and PyTorch) contributed to Deep Learning's success.
• Yann LeCun, Geoffrey Hinton, and Yoshua Bengio received the 2019 ACM Turing Award for their work in neural networks.

Machine Learning Basics

• Machine learning algorithms learn from labeled data to build a model.
• This model then makes predictions on new, unseen data.
• Labeled data is crucial in training machine learning models.

ML vs. Deep Learning

• Most machine learning methods rely on human-designed input features.
• Machine learning often involves optimizing weights to maximize prediction accuracy.
• Deep learning excels in automatically learning relevant features from data.

What is Deep Learning (DL)?

• Deep learning is a subfield of machine learning that focuses on learning representations of data.
• It's effective at learning patterns.
• Deep learning algorithms use a hierarchy of multiple layers to learn different levels of representation.
• Deep learning models can understand complex data and provide useful responses when fed with tons of information.

Why is DL useful?

• Manually designed features can be over-specified or incomplete, requiring significant time and resources to design and validate.
• Deep learning features are easily adaptable and fast to learn.
• Deep learning frameworks can be used to learn representations of the world, visuals, and languages.
• Deep learning enables the use of end-to-end joint system learning.
• The use of large training data sets is essential for effective deep learning models.
• Deep learning techniques have outperformed other approaches in fields like speech, vision, and natural language processing (NLP).

Representational Power

• Neural networks with at least one hidden layer can approximate any complex continuous function.
• Deep neural networks have the same representational power as a single-layer network, but deep networks often perform better.

Perceptron

• The perceptron is the basic processing element in a neural network.
• Perceptrons have multiple inputs and output a single output based on weighted sums of inputs.
• A perceptron's output is dependent on the weights applied to its inputs.

Single Layer Neural Network

• A single-layer neural network has one input layer and one output layer with no intervening layers.
• The output of the network is calculated using a linear transformation.
• A single-layered neural network only can perform linear separations and cannot function efficiently and effectively for complex separations.

Sigmoid Function

• This is a sigmoid activation function, used in neural networks.
• The function maps the input range to output between 0 and 1.

Neural Network Example

• This is an example of a simple neural network's configuration.
• It illustrates how calculations proceed forward through a network.
• It shows the network performing a calculation using weights between corresponding neurons.

Matrix Operation

• Neural networks perform calculations using matrix operations to accelerate computational speed.
• Matrix operations allow for efficient calculations.

Neural Network

• Neural networks consist of multiple layers.
• They use weight matrices to combine inputs from previous layers.
• Each layer generally has bias values.

Softmax Layer

• In multi-class tasks, softmax is often the final layer of a neural network.
• It maps network output values to a probability distribution, making predictions more interpretable.

Activation: Sigmoid

• Sigmoid functions map real-valued input to a range between 0 and 1.
• Sigmoid is less common in current NNs due to the vanishing gradient problem.
• The gradient near 0 or 1 becomes close to zero slowing down training.

Activation: Tanh

• Tanh function maps real values to a range of -1 to +1.
• Tanh functions are zero-centered and preferred to sigmoid in many applications.

Activation: ReLU

• ReLU is a rectified linear unit that thresholds values at zero, making it computationally efficient.
• ReLU is a popular choice in modern NNs.
• ReLU activation functions speed up gradient descent.

Activation: Leaky ReLU

• Leaky ReLU is a variant of ReLU that fixes the “dying ReLU” problem.
• A small negative slope ensures some signal flow when the input is less than zero.

Activation: Linear Function

• Linear activation functions output a signal directly proportional to the input signal.
• Linear activation is commonly used in regression tasks, as output values are real numbers.

Training NNs

• Training NNs involves properly setting parameters (θ) for optimal performance based on a defined criterion.
• Initializing parameters with random values influences training outcomes.
• Data preprocessing techniques (like mean subtraction and normalization) for input data improve network convergence, allowing the use of larger learning rates.

Training NNs - Loss Functions

• A loss function calculates the difference between the model predictions and true labels
• Mean square error and cross-entropy are common examples of loss functions
• Total loss is calculated over the entire training set.

Training NNs - Gradient Descent

• Training NNs usually uses gradient descent, either mini-batch or stochastic variants.

Gradient Descent Algorithm

• A method of updating model parameters to minimize a loss function via calculated gradients.
• The algorithm involves iterative refinements using the gradients to improve the model parameters.

Problems with Gradient Descent

• GD can be slow at plateaus or stuck at saddle points.
• It's often highly complex (and non-convex) and may not find the global minima of the loss function.

Gradient Descent with Momentum

• The algorithm takes into account the momentum from prior iterations.
• This approach can aid in avoiding slow convergence at plateaus or local minima.

Adam

• An optimization algorithm that adaptively adjusts the learning rate for each parameter to improve training speed and stability.
• Adam employs momentum to update parameters.

Learning Rate

• The learning rate, or step size, dictates how much to adjust the parameters per iteration
• Selecting a suitable learning rate is a crucial process for neural network training.
• Poor choices can lead to slow training or instability.
• Learning rate schedules are crucial for choosing learning rates at different stages of the training process.

Vanishing Gradient Problem

• Vanishing gradients occur when gradients, while moving down a gradient, diminish, making updates minuscule, slowing training.
• Solutions include adjusting learning rates appropriately or using alternative activation functions like ReLU.

Generalization

• Generalization involves a model's ability to perform well on unseen data, unrelated to the training dataset
• This includes dealing with overfitting (when a model fits training data too well but poorly on new data) and underfitting (when the fitting of the model is not good enough).

Regularization: Weight Decay

• Weight decay adds a regularization term to the loss function to prevent large weights and overfitting the model.
• The weight decay coefficient controls the regularization’s strength during training.

Regularization: Dropout

• Dropout randomly turns off a portion of nodes during training to prevent co-dependencies between nodes and increase training stability.
• It is a form of ensemble learning.

Regularization: Early Stopping

• Helps prevent overfitting by monitoring the model's performance on a validation set.

Batch Normalization

• By normalizing data in each minibatch, it is more effective for training the model.
• Results in faster convergence, and larger learning rates are achievable.
• It mitigates the internal covariate drift problem.

Hyper-parameter Tuning

• Hyper-parameters are parameters controlled by the user and not learned during the training process.
• This includes learning rates, the number of layers and nodes, optimizer type, regularization parameters, batch size, activation function, and loss function.
• Several methods are used for selecting optimal hyper-parameter values.

k-Fold Cross-Validation

• A technique of splitting the data to validate different parameter settings and estimate the model's performance on unseen data
• It involves a process of repeatedly training and validating models across different splits of the data, averaging the result to obtain a more precise and robust evaluation of a prediction model.

Ensemble Learning

• Ensemble learning involves combining output from multiple different models that can work better together

Deep vs. Shallow Networks

• Deeper networks often perform better than shallow networks, but there is a limit to how deep a network should be.

Convolutional Neural Networks (CNNs)

• CNNs are specialized neural networks that are particularly adapted to image data.
• They can efficiently extract features and are robust against changes in image placement.
• CNNs do not use fully-connected layers, making them faster and less prone to overfitting.
• Convolution and pooling are the core layers used.

Pooling Layer

• Pooling layers reduce the spatial dimensions of features from previous convolutional layers.
• They reduce parameters through downsampling.

Fully Connected Layer (FC layer)

• FC layers are the final layers of a CNN where nodes connect across the entire feature map.

CNN Summary

• Modern CNNs are a combination of convolution, pooling, and fully connected layers with increasing depth and smaller filter sizes.