ELE888 - Midterm (Theory)

• Machine learning performs tasks based on patterns, not explicit instructions

Benefits of Machine Learning

• Shorter

• More accurate

• Easier to maintain

• Machine learning includes spam filters

• Spam filters detect unusual words/phrases and labels them as spam

Machine Learning Branches

• Supervised learning is more successful than unsupervised learning
  • Classification predicts categorical values
  • Used to Predict numerical values
• Unsupervised learning groups data based on similar features
  • Clustering groups/clusters points without labels, using similar features
  • Dimensionality reduction reduces features by obtaining principal variables, and not all features are useful
• Reinforcement learning involves a system learning through its actions
• Favored actions yield positive rewards
• Unfavored actions yield negative rewards/punishment
• Actions are remembered for future decisions
• Agents select the best action to achieve a task and improve model performance

Instance Based Learning

• Instance based learning memorizes examples

• Instance based learning uses similar features to identify other examples.

• Using word count to detect spam emails is an example

• Emails shares similar word counts with known spam emails will be flagged

• K-NN is an example

• Datasets are useful for supervised and unsupervised learning only

• Data for Reinforcement learning cannot be tabulated

Target Columns

• Supervised machine learning model has a target column

• Presence of a target column with numerical values indicates regression

• Presence of target column containing categorical values indicates classification

• Model based learning uses the example models to make predictions

Decision Boundary

• Decision boundaries distinguish two classes
• Point (x,y) can be plugged into a function to define the boundary
  • Positive output: one side of the boundary
  • Negative output: the other side

Cost (AKA Loss) Function

• The measure how well a model predicts values

• The goal is to minimize cost for accurate predictions

• The function improves model accuracy

• Cross validation maximizes the amount of data using in training

• Cross validation is crucial when evaluating the model

• Regularization is a technique preventing a model from overlearning, and improves the model's generalizability

• Overlearning occurs when a model learns too much training data, causing poor performance

Ridge Regression

• Ridge regression adds a penalty to all coefficients to reduce them equally
• Impact of less important features is reduced
• Alpha = 0: reverts to linear regression cost function and disregards overfitting
• High Alpha: penalizes large coefficients, leading to a simple, under-fitted model

Lasso Regression

• Some coefficients are reduced to 0

• All values that are zero are not taken into consideration in the model

• Coefficients with a non-zero value are considered important features of the data set

• Overfitting describes models that learns too closely with training data, and cannot generalize

• Underfitting describes models that has not learned enough from the data, and cannot make any accurate predictions

Linear Regression

• Linear Regression is a method to predict value using relationship between two variables
• Attempts to find the line of best fit and is known as the regression line
• Used for future predictions
• Independent Variable: variable used for making predictions
• Dependent Variable: is the variable being predicted

Contour Plots

• 3D Representation depicting relationship between two variables
• Variables involved include theta0, theta1 and the cost function

Ellipses

• Shows different levels of cost function through ellipses
• Each ellipse corresponds to specific cost function
• Every point on the same elliptical ring has J(theta) or cost function
• Cost will decrease near the ellipse center which indicates accurate predictions from the model
• The ellipse's center represents the optimal theta0, theta1 values, helping minimize cost function

Ways to Minimize Cost Function

• Gradient descent
• Normal equations

Gradient Descent

• Repeatedly adjusts model parameters to achieve task or improve model's performance

Normal Equations

• They use direct solution and formula to find the exact values

Benefits of Using Normal Equations

• Its not iterative
• It works well with small datasets
• Gives exact solution

Negatives of Using Normal Equations

• Associated computational costs are high

• Doesn't work well with large datasets

• Learning rate controls how much a model's weights are updated during training

Large Learning Rate

• Learns much faster/reaches optimal values quicker
• Overshooting is possible because big steps miss optimal values
• Models can oscillate wildly causing divergence

Small Learning Rate

• Tune the process of reaching optimal values (achieves accurate results) because of small steps
• Since the changes are small it takes Model takes longer to learn since updates are small
• Model may get stuck in a sub-optimal value, unable to reach the global optimal alue

Logistic Regression

• Used in binary classification to predict one of two outcomes
• Predicts a continuous probability value, implements a decision threshold to classify outcome of two classes

Stochastic Gradient Descent (SGD)

• An iterative learning algorithm performs updates to model after one data point
• Makes model faster, but increases noise
• Reduces memory usage and computation time

Batch Size

• When training mode, dataset is split into batches/groups

• Small batch size indicates faster but less stable training

• Large training size: more stable, but requires more memory

• Mini Batch: splits data into small portions, updates model after batches are passed through

• With 10 samples and batch size of 2, model updates after all 5 batches are completed

• Batch set or batch gradient uses the entire dataset at once to compute gradients

• Batch methods need more memory and are slow for large datasets

• Epochs are a full pass through the entire data set during training

• Learning curve is a graph that shows how wel a model is learning overtime

• Improving curves mean that the model is learning

• Flat curves mean that the model has stopped learning

Model Hyper-Parameters and Parameters

• Model hyper-parameters are selected before model training and control how the model learns
• Model parameters are values learned from the dataset or can change during training
• Normalizing features before running gradient descent is crucial for a variety of reasons

Normalizing Features

• Faster convergence allows gradient descents to update model parameters based on partial derivatives of cost function

• When features have vastly different scales, cost function becomes elongated which makes gradient descent steps inefficient and slow

• Normalization ensures that the minimum is reached faster

• Spherical Contour refers to the cost function

• Avoiding Vanishing or Explosoding Gradients

• If features are on different scales, the gradients can either become to small or to large leading to poor updates in learning

• Better Numerical Stability

• Large value features can cause numerical overflow errors, particulary when dealing with matrix operations

• Equal contribution of Features

• Features with large magnitudes dominate learning process, making model biased toward those features

• Normalization ensures each feature contributes equality

• Hypothesis (also called the model) is used when machine learning used mathematical functions to make predictions based on input features

• Represents relationship between input values and outputs

Overfitting

• Overfitting occurs when machine learning models capture noise in the training data instead of underlying pattern
• Models perform well on training data but poorly on new data
• Simpler terms, they are to complex and use memorization instead of generalization

Signs of Overfitting

• High accuracy on training data, but poor performance on validation/test data
• Large gap between training and validation loss
• Models perform well on known data, but fails on new inputs

How to Reduce Overfitting

• More Training Data
• Collect more samples to help model generalize better
• Helps reduce models dependency on noise
• Feature Selection and Engineering
• Remove irrelevant features the introducee noise
• Use domain knowledge to create meaningful features
• Regularization
• Adds absolute weight penalties, leading to feature selection
• L2 Regularization: Adds Squared weight penalties, reducing coefficients and making model more simple
• Cross Validation
• Use techniques to ensure model is evaluated on multiple data subsets

Skewed Learning

• Skewed learning refers to situations where dataset has imbalanced distribution
• Some classes appear more than others
• It is common in calssification problems where one class dominates, and leading to biased data

Examples of Skewed Learning

• Fraud Detection
• 99% of transactions are legitimate and 1% are fraudulent
• Models predicting legit all time would be 99%accurate
• Models would be useless for detecting fraud
• Medical Diagnosis (Rare Diseases)
• Datasets are made of 98% healthy patients and 2% who have a disease
• If model always predicts “healthy“ it will have high accuracy but fail in detecting disease
• Spam Detection
• 95% of emails are non spam, and 5% that are spam
• Models struggle to correctly spam messages

Similarities Between Logistic Regression and Neural Networks

• Share similarities in their fundamental concepts and math formulations
• Both use a weighted sum of inputs
• Both use sigmod activation function
• Both are used for classification
• Both use loss functions based on Log Likelihood
• Both use a gradeient descent for Optimization

Sigmoid va Hyperbolic Tangent Differences

• Both sigmoid and hyperbolic tangent functions are S shaped, but have key differences in their properties and neural networks
• Their equations are not the same

Range of Output Values

• Sigmoid outputs value between 0 and 1, which can lead to problems in models with centering data
• Tanh outputs values between negative 1 and 1, which helps activations stay closer to zero, improving learning stability

Zero-Centering and Effect on Learning

• Tanh is zero centered, meaning outputs are evenly and distrubted around data
• Helps in Training
• Sigmoids is not Zero Centered
• Can cause bias shifts in activations, leading to slower convergence

Gradient Vanishing Problem

• Both Sigmoid and Tanh both suffer from vanishing gradients when inputs are too large or to small