Neural Networks concepts

Artificial Intelligence

• Encompasses programs performing tasks that typically require human intelligence
• Definitions involve programs that can think, reason, and learn

Artificial General Intelligence (AGI) vs. Narrow Artificial Intelligence

• AGI solves any problem a human can, even those not specifically trained on
• AGI has understanding and learning comparable to humans
• Narrow AI solves specific problems without generalizing

Training Mode vs. Inference Mode

• In training mode, intermediate results are calculated and stored for backpropagation
• Updating models occurs in training mode
• Inference mode yields actual predictions without updating the mode

Training vs. Production

• Training aims to improve the model
• Training mode is used to update models while inference mode validates them
• Production uses fully trained models for real-world tasks
• Production models operate in inference mode

Relationship Between AI, Machine Learning, Neural Networks, and Deep Learning

• Artificial Intelligence > Machine Learning > Neural Networks > Deep learning
• Artificial Intelligence encompasses all sub-fields
• Modern AI work is typically Machine Learning-based
• Neural networks are all Machine Learning-based
• Modern Neural Networks typically fall into Deep Learning

Symbolic AI vs. Subsymbolic AI

• Symbolic AI represents information as symbols or rules, making it easier to understand
• Subsymbolic AI stores sentences as numeric values in a neural network
• In subsymbolic AI, specific concept locations cannot be pinpointed

Machine Learning and Modern AI

• Machine learning programs improve behavior without direct human modification
• Almost all modern AI programs depend on machine learning

Predictive, Descriptive, and Generative AI Systems

• Predictive AI uses supervised learning on labeled datasets to predict data
• Descriptive AI uses unsupervised learning to identify patterns or structures in data
• Generative AI uses self-supervised learning to generate its own labels and create new content

Supervised Learning vs. Unsupervised Learning

• Supervised learning uses labeled data
• Unsupervised learning uses unlabeled data

Self-Supervised Learning

• LLMs (ex: Chat GPT) often a system that generates its own labels using defined algorithms
• Useful for large datasets that are difficult to manually label
• Examples include next-word prediction and masked-word predictio

Datasets, Samples, Features, and Labels

• Datasets is a collection of samples
• Each sample contains a set of features used as inputs
• Labels are the expected output for each sample

Classical Machine Learning vs. Neural Networks

• Classical methods are easier to understand
• Classical methods is less computationally expensive
• Classical methods work better with less data
• Classical methods less prone to overfitting

Foundational Mathematical Specialties for Machine Learning

• Statistics helps discover the relationships between inputs and expected outputs
• Calculus is the study of how things change at a rate
• Linear algebra is the study of linear systems in multidimensional space

Linear Algebra

• The study of linear systems in a multidimensional space
• 2nd space is line
• 3rd space is a plane.
• n-space is a hyperplane

Tensors

• Provides a theoretical framework for understanding our data
• Used commonly in AI
• Can be used to modify data and come up with representations that are useful
• Used to carry out much of the operations of neural networks
• Multidimensional arrays used to store data
• Can be input data, output results, or the neural networks themselves

N-Dimensional Space

• Used to represent a sample
• n represents the number of dimensions
• Common to also associate n with input and m with output

Models Mapping Dimensions

• Describing how a model transforms data from one original space
• Output space has 2 dimensions corresponding to the labels per sample

Jupyter Notebook

• Interactive notebook system
• It's valued for presenting findings and prototyping
• Supports languages like Julia, Python, and R

Google Colab

• Cloud-based platform
• Used for running Jupyter notebooks
• Offers free access to GPUs and TPUs

Advantages of Neural Networks vs. Classical Machine Learning

• Neural networks can automatically extract complex features from raw data

Parameters in Neural Networks

• Parameters are the weights and biases
• Learned during training to optimize model predictions

Weights and Biases

• Weights adjust the input to each node
• Biases are constants added to the weighted sum to help the model learn

Activation Functions in Neural Networks

• Introduce non-linearity which enable networks to model complex patterns
• Removing them would make the network equivalent to a linear model

Deep Neural Network

• Has many hidden layers
• Allows the model to learn hierarchical representations of data
• Allows for complex data modelling

Dense or Fully-Connected Neural Network

• Every node in a layer is connected to every node in the subsequent layer

Neural Network Capabilities

• Can simulate any continuous function with non-linear relationships

Deeper, Higher, and Lower

• Deeper means closer to the output, deeper into it
• Higher means closer to the output, more abstract
• Lower means closer to the input

Network Capacity

• High Network capacity refers to the ability to learn greater detail from the dataset
• If your dataset is too small, then overfitting can occur

Role of features

• If there are insufficient samples it leads to overfitting
• More samples help the model generalize better

Input Order in Dense Neural Networks

• Dense neural networks do not rely on the order of inputs
• Each input processes independently by each node

One-Hot Encoding

• Used for categorical data
• Converts categories into binary vectors to be used in machine learning models

Activation Functions

• They are used to scale the output values to a desired range
• Often used for probabilities for classification tasks

Training loop

• Steps: Model processes the batch, computes the loss, adjusts weights with back propagation, and updates parameters

Epochs

• An epoch is one complete pass through the entire training dataset

Batch

• A batch consists of a sample of data

Stochastic vs Mini batch

• Stochastic updates parameters after each sample
• Mini-Batch uses small batches
• Full-Batch uses the entire dataset for each update

Stochastic Definition

• Stochastic refers to using a single data sample at a time for parameter updates
• Introduces randomness into the training process

Over fitting

• Occurs when a model learns noise or irrelevant patterns in the training data
• Usually due to too much complexity or insufficient data

Validation

• Validation Dataset helps tune hyperparameters and prevent overfitting
• helps maintain training for better and unseen data sets

Model Relationships

• In real and unsimplified systems, data relationships become complex, and we focus more on optimizing model
• In simple systems performance through metrics helps visualize raw inputs and outputs.
• In advanced real world systems metrics is how we maintain data training

Loss vs Output

• Loss has more data useful for training and is the preferred function for training loops
• Output graphs are preferred at the end so you avoid having to do fine-tuning

Gradients

• Used for increasing accuracy and optimization for gradient vector models.
• It is less useful for parameters and performance

Epoch and Datasets

• Epochs represent complete passes through the dataset and its subsets
• Batches are processed for any one set of data and are used to update in gradients descents

Model Training

• Gradient descent stops when the loss no longer improves
• Gradient descent may be used up to the point that the model does not get more accuract convergence criteria is met

Accuracy

• Loss measures how the mode's predictions are accurate to the actual values
• Accuracy measures the percentage of that predictions

Models

• The Forward Pass computes the output
• The Backward Pass updates the weights based on the error calculated during the forward pass.

Samples

• Training samples are used to update model parameters
• Validation samples help evaluate the model's performance without influencing parameters

Model Building

• In each epoch, batches are processed with each sample
• Comparison occurs when true prediction values is compared to our model
• Then after weights are updated models and weights are updated.

Gradients Impact

• Vanishing gradients occur when gradients become too small for effective weight updates. leads to slow or stalled learning in deep networks.

Steps for AI system Development

• Pre-development stage: Problem definition, data gathering;
• Development stage: Train the model, validate model;
• Deployment stage: Monitor performance, models must be updated, system maintenance.

Data Variations

• Structured data is easier to work with
• Complex data is costly due to the requirements for training

Ethical Issues

• Issues include bias, privacy, consent
• Potential harm to individuals or communities.

Evaluation

• The evaluation should evaluate the real world performance of fair, unbiased model

Thresholds

• Can allow for inflated data
• Can allow for mislead stakeholders about the model's capability.

Data Preperation

• 2 stages: Data normalization and Data missing values

Steps for model selection:

• Performance of the model for test and use it, and evaluate and test the model for long term functionality

Long Term Model Deployment

• Optimize the model, and adapt to any new data

GPU vs CPU

• GPUs are optimized for single-threaded performance while CPUs use faster calculations and are less efficient.

AI Advantages for Machine Learning

• Faster data speeds and improved processing power

Variable Issues

• Weather patterns and more data sources can cause irredeemable errors

Chaotic Patterns

• High level changes can cause major faults

Real Life Models

• The challenge faced difficulties due to complex and noisy data with many variables, making predictions unreliable.

Al Difficulty Level

• Geopolitical events involve complex, unpredictable human behavior and a wide range of influencing factors, making them difficult to model effectively

Al for models

• Certain events, like natural disasters or economic trends, could be modeled with Al based on historical data and patterns.

Systems and Models

• Meaning is interpretability but the models are more efficient

Unfair Data

• Online gaming that can exploit those weaknesses for unfair advantage and for the model

Data usage

• Can be model optimized
• Can allow for better for real purposes built for specific task

Questions cover neural network depth, capacity, feature impact, input order, one-hot encoding, activation functions, cost functions, and gradient descent variations. Also covers sample representation in n-dimensional space.