Machine Learning Introduction

Introduction to Machine Learning

• Kylie Ying, a physicist and engineer with experience at MIT, CERN, and Free Code Camp, introduces machine learning for beginners.
• The video covers supervised and unsupervised learning models, the logic and math behind them, and programming on Google CoLab.
• The UCI machine learning repository is used, specifically the "magic gamma telescope data set".
• The data set involves using properties of patterns recorded by a gamma telescope to predict the type of particle that caused the radiation (gamma particle or hadron).
• The attributes of the patterns collected in the camera include length, width, size, and asymmetry.

Setting Up the Environment

• Import necessary libraries such as NumPy, pandas, and matplotlib.
• The data set can be found at a specified URL.
• Upload the downloaded file to Google CoLab.
• Read the CSV file into a pandas data frame.
• Assign column labels to the data frame using a list of attribute names from the data set description.

Data Preprocessing and Understanding

• The class labels in the data set are "G" and "H," representing gammas and hadrons, respectively.
• Convert the class labels to numerical values (0 and 1) for computer understanding.
• Each row in the data frame represents a sample or data point.
• Each sample has values for different features and a class label.
• The goal is to predict the class (gamma or hadron) based on the features, which is a classification task.
• The features are the properties used to predict the label, in this case, the class column.
• The overall process is an example of supervised learning.

Machine Learning Fundamentals

• Machine learning is a subset of computer science focused on algorithms that allow computers to learn from data without explicit programming.
• AI (artificial intelligence) aims to enable computers to perform human-like tasks.
• Machine learning is a subset of AI focused on making predictions using data.
• Data science finds patterns and draws insights from data, possibly using machine learning.

Types of Machine Learning

• Supervised learning uses labeled inputs to train models and predict outputs for new inputs.
• Unsupervised learning uses unlabeled data to learn patterns in the data.
• Reinforcement learning involves an agent learning in an interactive environment based on rewards and penalties.

Supervised Learning in Detail

• A machine learning model takes inputs (feature vector) and produces an output (prediction).
• Qualitative features are categorical data with a finite number of categories or groups.
  • Nominal data: Categorical data without inherent order (e.g., gender, nationality).
  • Ordinal data: Categorical data with inherent order (e.g., age groups, ratings).
  • One-hot encoding is used to represent nominal data for computers.
• Quantitative features are numerical valued data.
  • Discrete (integers)
  • Continuous (real numbers).
• Examples include length and temperature.

Supervised Learning Tasks

• Classification tasks predict discrete classes.
  • Multi class classification: Predicts one of several different classes
  • Binary classification: Predicts between two classes (e.g., hot dog or not hot dog).
• Regression tasks predict continuous values.

Model Evaluation and Training

• A Pima Indian diabetes data set contains features like pregnancies, glucose levels, and the outcome (diabetes or not).
• Features matrix (X) contains the input features.
• Labels/targets vector (y) contains the output values.
• The model makes a prediction based on the input features.
• The prediction is compared to the actual value to assess the model's performance.
• The difference between the prdeiction and the actual value is referred to as loss.
• Training involves adjusting the model based on this comparison.
• The data is split into training, validation, and testing data sets to assess how well the model can generalize to new, unseen data.
• The training data set is used to train the model.
• The model generates a vector of predictions corresponding to the training data samples.
• The difference between the predictions and the true values is calculated as a loss.
• Adjustments are made to reduce the loss, improving the model's accuracy.

Validation Set

• Used as a reality check of a model during or after training.
• Checks if the model can handle unseen data.
• After each training iteration and after training is over, the validation set is used to assess loss.
• Loss from the validation set is not fed back into the model (no closed feedback loop).

Loss

• Represents the difference between a model's prediction and the actual label.
• A smaller loss indicates better model performance.
• Model C, among the examples, had the smallest loss, indicating the best performance.

Test Set

• Used as a final check on a chosen model to see how generalizable it is.
• Assesses model performance on data it has never seen during the training process.
• The loss on the test set is the final reported performance of the model.

Loss Functions

• Used to quantify the difference between prediction and actual label.
• Aim to provide a formulaic way to describe the loss and come up with numbers.
• L1 Loss:
  • Calculates the absolute value of the difference between the real value and the predicted value.
  • Loss increases linearly as the difference between the predicted and real value grows in either direction
• L2 Loss:
  • Squares the difference between the real and predicted values.
  • Provides a quadratic loss function, where small differences result in minimal penalty and larger differences incur a much higher penalty.
• Binary Cross Entropy Loss:
  • Used for binary classification problems.
  • Loss decreases as the model's performance improves.

Accuracy

• A measure of performance, such as the percentage of correct predictions out of the total predictions.
• Example: If a model predicts four items as "apple, orange, orange, apple" and the actual values are "apple, orange, apple, apple," the accuracy is 75% (3 out of 4 correct).

Data Preparation in Colab Notebook

• Classes are converted to numerical values (0s and 1s) for computer understanding.
• Features are plotted as histograms to understand their relationship with the class (gamma or hadron).
• Training, validation, and test data sets are created by splitting the data frame using "NumPy.split".
• Data is shuffled using the "sample" method.
• Splitting occurs at 60% for training data, between 60% and 80% for validation, and from 80% to 100% for test data.

Data Scaling

• Scaling adjusts the values in the dataset so they are relative to the mean and standard deviation of their respective columns.
• Scaling can be important to ensure features do not disproportionately impact model training due to differing scales.
• A function named "scale data set" created to scale the data
• Standard Scalar:
  • Imported from the "scikit-learn library".
  • Used fit and transform the x values.
  • "H stack" stacks arrays horizontally, placing them side by side.
  • "NumPy reshape" reshapes the array ensuring compatibility for stacking.
  • The function returns scaled data and corresponding y values.

Oversampling

• Used when there is an imbalance in the dataset, where one class has significantly fewer samples than the other.
• Addresses unequal representation by increasing the number of samples in the minority class to match the majority class.
• Random oversampler:
  • Imported from the "scikit-learn library" helps perform the oversampling.
  • Validation and test sets were not oversampled, maintaining their original distribution for unbiased evaluation

K-Nearest Neighbors (KNN)

• A classification algorithm that assigns a label to a new data point based on the labels of its nearest neighbors.
• The algorithm relies on a distance metric (e.g., Euclidean distance) to determine the proximity of data points.
• Euclidean distance:
  • A straight-line distance
  • Common distance function used to measure distance.
  • Formula is the square root of one point x minus the other points x squared plus the same thing for y
• "K" represents the number of neighbors considered when determining the label of a new point.
• The label is determined by looking at what is around the point.
• The appropriate number of neighbors will vary depending on a particular dataset.
• The majority label among the k-nearest neighbors is assigned to the new data point.
• The algorithm can be extended to higher dimensions.

KNN Model Training and Prediction

• KNN is implemented using scikit-learn (SKlearn)
• SKlearn packages avoid manual coding, reducing bugs and improving speed
• The KneighborsClassifier is imported from sklearn.neighbors for classification tasks
• KNN model is initialized with a specified number of neighbors
• The .fit method trains the model using the x_train and y_train data
• x_train represents the training data features
• y_train represents the training data labels
• The .predict method generates predictions using the x test data

Evaluating KNN Model Performance

• classification_report is used to assessed performance
• It provides key metrics such as precision, recall, and F1-score
• Accuracy measures the overall correctness of the model's predictions
• Precision measures how many of the points labeled as positive by the algorithms are actually positive
• Recall measures how many of the truly positive points were correctly labeled as positive by the algorithm
• The F1-score balances precision and recall, useful for unbalanced datasets

Naive Bayes

• Conditional probability and Bayes rule are fundamental concepts

Conditional Probability

• Probability of having COVID given a positive test is written as P(COVID | Positive Test)
• It's calculated by dividing the number of people with COVID who tested positive by the total number of people who tested positive

Bayes' Rule

• Bayes' Rule is used when the probability of A given B is unknown.
• The formula accounts for probability of B given A, probability of A, and probability of B

Applying Bayes' Rule to Disease Statistics

• The probability of a false positive is the probability of testing positive given no disease
• The probability of a false negative is the probability of testing negative given the disease
• The probability of disease is the likelihood of having the disease in the general population

Expanding Bayes Rule

• Posterior: The probability of a sample belonging to a certain class, given the evidence
• Likelihood: The probability of observing the features, assuming the sample belongs to a certain class
• Prior: The probability of a class in the overall population of samples.
• Evidence: The overall probability of the features.
• Bayes rule is the probability of being in some class K, given all the data

Naive Bayes Assumption

• Naive Bayes assumes all features are independent when calculating probabilities
• Makes computation easier, but may sacrifice accuracy

Maximum a Posteriori (MAP)

• MAP selects the most probable class for a given instance
• It minimizes the probability of misclassification by maximizing the posterior probability

Gaussian Naive Bayes Implementation

• GaussianNB is imported from the sklearn.naive_bayes
• Used the same way as the KNN model above

Model Comparison

• Naive Bayes model performs worse than the KNN model above even if not "too shabby"

Regression vs. Classification

• Linear regression may not be suitable for classification problems
• A regression line might not accurately predict class types
• Linear regression estimates probability, ranging from 0 to 1, for class 0 or class 1

Addressing Probability Range Limitations

• The equation p = mx + b can range from negative infinity to infinity
• Probability values must be between 0 and 1
• Setting odds equal to mx + b addresses the infinite value issue, where odds = p / (1 - p)
• Taking the log of the odds allows for negative values, resolving the negative range issue

Solving for Probability

• Removing the log by taking e to the power of both sides: p / (1 - p) = e^(mx + b)
• Multiplying out: p = (1 - p) * e^(mx + b)
• Simplifying: p = e^(mx + b) - p * e^(mx + b)
• Moving like terms: p * (1 + e^(mx + b)) = e^(mx + b)
• Solving for p: p = e^(mx + b) / (1 + e^(mx + b))
• Rewriting with a numerator of 1: p = 1 / (1 + e^(-mx - b))

Sigmoid Function

• Sigmoid function: s(x) = 1 / (1 + e^(-x))
• Logistic regression fits data to the sigmoid function
• The sigmoid function ranges from 0 to 1
• Sigmoid function can goes in between zero and one, which fits the expected shape for classification models.
• Rewriting the probability equation in terms of the sigmoid function helps fitting the data

Types of Logistic Regression

• Simple logistic regression uses one feature (x0)
• Multiple logistic regression uses multiple features (x0, x1,..., xn)

Implementation in scikit-learn

• Logistic regression can be imported from sklearn.linear_model
• Different penalties, such as L2 (quadratic formula), can be used
• The best parameters to pass into the model should be determined based on validation data

Support Vector Machines (SVM)

• SVM aims to find the line or hyperplane that best differentiates classes
• In 2D, this is a line; in 3D, it's a plane

Finding the Best Divider

• The best divider is the one that clearly separates the data
• The goal is to maximize the margin, which is the boundary between the points and the dividing line

Margin and Support Vectors

• Margin: The boundary between the points in the classes and the dividing line
• Support vectors: Data points that lie on the margin lines and help define the divider

Robustness to Outliers

• SVMs may not be robust to outliers, as outliers can significantly alter the position of the support vector

Kernel Trick

• The kernel trick involves creating a projection to make data separable
• Example: Transforming x to x and x^2
• Applying a kernel transforms the data to separate class

Implementation in scikit-learn

• SVC (Support Vector Classifier) can be imported from sklearn.svm
• SVM performance :accuracy often jumps with SVM due to kernel trick
• SVM may perform better than logistic regression, Naive Bayes, and k-NN

Neural Networks

• Neural networks consist of an input layer, hidden layers, and an output layer
• Each layer contains neurons

Neurons

• Neurons receive inputs that are weighted by some value (w)
• The sum of the weighted inputs, along with a bias term, goes into the neuron
• The output of the neuron is determined by an activation function

Activation Functions

• Activation functions introduce non-linearity to the model
• Without activation functions, the neural network becomes a linear model
• Examples of activation functions: sigmoid, tanh, ReLU

Training

• Training involves feeding the loss back into the model and making adjustments to improve the predicted output

Gradient Descent

• Gradient descent follows the slope of the loss function to reduce the loss
• The loss with respect to different weights (w0, w1,..., wn) may vary
• The change in weights can be calculated using calculus
• New value: w_new = w_old + alpha * (arrow), where arrow suggests direction

Learning Rate

• Alpha (α) is the learning rate, which determines the size of the step taken in the direction of reducing the loss
• A smaller learning rate prevents overshooting and ensures stable convergence