Introduction to Deep Neural Networks

Introduction to Deep Neural Networks

• Deep neural networks are complex sets of interconnected nodes that progressively process data, making them powerful tools.
• Models can be adjusted with methods like gradient descent to find the ideal structure (or model parameters) of the network.

Supervised Learning

• A supervised learning model uses input data (x) and a target value (y) to predict y given x.
• Two types exist; regression (predicting a numeric value) and classification (predicting a categorical value).

Nobel Prize and AI

• The 2024 Nobel Prize in Physics went to scientists who helped develop the core of artificial intelligence or specifically, machine learning.
• The 2024 Nobel Prize in Chemistry went to scientists who uncovered the secrets of proteins by using AI.

Protein Structures via AI

• Predicting the 3D structure of proteins using AI has been a significant challenge.
• Advances in machine learning methodologies like AlphaFold drastically improved protein structure prediction accuracy.

Machine Learning Example

• Input x is processed by the machine learning algorithm to determine a prediction y.
• The example showcases various input data types, such as:
  • Protein amino acid sequence
  • Medical X-Ray images
  • Images of various types

Machine Learning and AI

• Machine Learning (ML) is a subset of artificial intelligence (AI).
• ML algorithms learn from data to make predictions or decisions without explicit programming.

Basics of Machine Learning

• Given data points (xᵢ, yᵢ), the aim is to find a function (f(x)) that best fits that data.
• Models like linear or polynomial functions are common in determining the function's structure or model parameters.
• The models are adjusted and trained to reduce error, or loss, in the predictions.

Deep Learning

• Deep learning uses multiple layers of interconnected nodes to transform input features or representations into useful structures for predictions.
• Deep learning structures, including convolutional neural networks, recurrent neural networks, and transformers, are designed for various data types.

Deep Neural Network Architecture

• Deep neural networks consist of interconnected processing units arranged in layers to transform data progressively.

Recap: Linear Regression

• A simple linear model, output = input features times weights plus a bias term, is demonstrated.

Logistic Regression

• A supervised machine learning method that estimates the probability of a categorical outcome (e.g. binary).
• It uses a sigmoid function as a non-linear activation function.

Softmax Regression

• An extension of logistic regression for handling multi-class classification problems.
• Uses the softmax function to predict the probabilities of each category.

Artificial Neuron

• A building block of deep learning models that calculates a weighted sum of input features plus a bias term and then applies a non-linear activation function.

Layer: Parallelized Weighted Sums

• A layer of a neural network that performs a weighted sum of input features.
• Applies a non-linear activation function like sigmoid or ReLU to those sums.
• Weights and biases (or offsets) are adjusted during training.

Network: Sequence of Parallelized Weighted Sums

• Neural networks process data sequentially with multiple layers.
• Weights, activations, and biases are adjusted to optimize an output.

Activation Functions

• Various activation functions are used in deep learning.
• Some examples include sigmoid, ReLU, and hyperbolic tangent (tanh).

Pop Quiz

• Determining the number of parameters involved in a simple multi-layer perceptron (MLP).

MLP Example

• Demonstrates how to design neural network architectures in Keras.

Activation at Output Layer

• The choice of activation function depends on the predicted outcome type.
• For regression, an identity function maps directly to the output.
• Softmax is frequently used for predictions of probabilities of multiple classes.

Training Deep Neural Networks

• Gradient descent methods help minimize the loss function in neural networks.

Training Neural Network Parameters

• Defines and minimizes the loss function, which measures the differences between the calculated output and the true value.

Loss Function for Classification Problems

• Cross-entropy is a common loss function for classification problems.
• It assesses the difference between predicted and true probability distributions.

Learning as Optimization: Gradient Descent

• Gradient descent is an optimization technique for determining the model's parameters (like weights and biases) that minimize a particular cost/loss function.

Large-Scale Learning

• Gradient descent algorithms, like stochastic gradient descent (SGD), are used to train models when the datasets are large.

Mini-Batch Gradient Descent

• A compromise between batch and stochastic gradient descent, mini-batch gradient descent uses subsets of the training data.

Learning Rate (LR)

• The learning rate in gradient descent determines the extent of adjustment to model parameters with each iteration.

Adaptive Learning Rate

• Learning rate adjustment mechanisms, such as exponential decay or step decay, modify the learning rate throughout training.

GD for Neural Networks

• Gradient descent methods are used to train neural networks, but the non-convexity of neural network loss functions leads to several training challenges like gradient instability.
• Gradient vanishing/exploding are issues that arise when training very deep neural networks.

Parameter Update Rules: Optimizers

• Techniques for efficiently updating parameters in large neural networks during training and improving learning stability, like SGD, Momentum, RMSprop, and Adam.

Computing Gradients: Backpropagation

• Backpropagation uses the chain rule of calculus to efficiently determine the gradient of the cost function, enabling effective training of neural networks.

Backpropagation

• Backpropagation is an algorithm to compute the gradients that enables the training of weights and biases in neural networks.

Vanishing Gradient Problem

• In very deep neural networks, the gradients become very small throughout the many layers, hindering or making training very difficult

Regularization Techniques

• Techniques like dropout, regularization batch normalization, and early stopping regulate the training of neural networks.

Dropout

• A regularization method for neural networks where neurons are randomly deactivated during training.

Batch Normalization

• A technique to normalize inputs within a mini-batch that helps combat internal covariate shift (changes in input distribution) for faster model training and improved performance.

Norm Penalties

• L1 and L2 penalties are regularization techniques used to encourage smaller weights and sparser connections in neural networks, which aids generalization.

Early Stopping

• A regularization method to prevent overfitting by stopping training the neural network at the point where performance on a validation set begins to degrade.

Dataset Augmentation

• Creating more data for training the model (increasing the size of the training dataset) that improves generalization.

Deep Learning Approach in General

• Deep learning is applicable to unstructured data (images, text, and audio) requiring sophisticated architectures.

Specialized Deep Learning Architectures

• Specific types of deep learning architectures (CNNs, RNNs, LSTMs, GRUs, and Transformers) are designed for handling various data types and tasks.

Summary of Topics Covered

• Summarizes essential topics, such as architecture, training methods, and techniques to improve deep neural network performance.

How to Combat Overfitting

• Describes various regularization methods to reduce overfitting issues. Various methods like dropout, batch normalization, early stopping, and dataset augmentation were discussed.

This quiz explores key concepts in deep neural networks and supervised learning. Learn about the architectures, methodologies like gradient descent, and the significance of neural networks in AI advancements, including Nobel Prize achievements in the field. Test your understanding of how these technologies impact protein structure prediction using AI.