Untitled Quiz

Machine Learning Lecture Notes

• Lecture 12 Introduction to Machine Learning
  • Instructor: Keerthana Vinod Kumar
  • PMRF Scholar, Koita Centre for Digital health
  • Indian Institute of Technology Bombay
• Gamma Hyperparameter
  • Higher gamma value: Decision boundary closer to points, more jagged, prone to overfitting
  • Lower gamma value: Smoother, less prone to overfitting
  • Applicable to all hyperplanes; Easier to observe in higher dimensions
  • Sometimes referred to as sigma
• Assignment 4
  • Use diabetes dataset (onedrive link provided)
  • Perform Exploratory Data Analysis (EDA): Overall info, dataset description, check for null values, data visualization, data scaling
  • Separate features and target variable
  • Perform train-test split (80:20)
  • Perform classification using SVM, Random Forest, KNN, and Decision Tree
  • Calculate evaluation metrics: Confusion matrix, Accuracy, Precision, F1-score, Recall, TPR, FPR, ROC-AUC
  • Store results in a DataFrame
  • Evaluate which algorithm performs best and give reasons
  • Upload assignment

Model Evaluation Metrics

• Confusion Matrix
  • Summarizes model performance on test data
  • Shows accurate/inaccurate instances
  • Used to evaluate classification models
  • Displays instances produced by the model
  • TP (True Positive): Correctly predicts positive
  • TN (True Negative): Correctly predicts negative
  • FP (False Positive): Incorrectly predicts positive
  • FN (False Negative): Incorrectly predicts negative
• AUC-ROC
  • ROC (Receiver Operating Characteristic) curve
  • Shows classification model performance for different thresholds.
  • Plotted between True Positive Rate (TPR) and False Positive Rate (FPR)
    • TPR = TP / (TP + FN)
    • FPR = FP / (FP + TN)
    • ROC curve analysis: Higher AUC implies better classification performance
  • Perfect performance corresponds to (TPR=1, FPR=0)
• Accuracy, Precision, Recall and F1
  • Accuracy
  • Proportion correct predictions to total predictions
  • Precision
  • Proportion of true positives among all positive predictions
  • Precision = TP / (TP + FP)
  • Recall (Sensitivity)
  • Proportion of true positive predictions among all actual positives
  • Recall = TP / (TP + FN)
  • F1-Score
  • Harmonic mean of precision and recall
  • F1-score balances precision and recall

Hyperparameter Tuning

• GridSearchCV
  • Used for finding the best hyperparameter combination
• Hyperparameters: Settings set before training; Non-learned from data.
  • Example: Number of trees in random forest, kernel type, parameter in SVM
• Kernel Function: Determines the similarity measure between data points
  • Used in higher-dimensional spaces for classification
• C (Hyperparameter): Affects margin size
  • Larger C: Smaller margin, to avoid misclassifying training samples (regularization)
  • Smaller C: Larger margin, allowing more misclassifications
• Gamma (Hyperparameter): Also related to the distance of a point from decision boundary
  • Higher gamma: Closer points considered for boundary, increasing potential for overfitting
  • Lower gamma: Further points considered, smoother boundary, decreasing overfitting chance

Kernel Hyperparameters

• Linear Kernel: Straight line to separate data points (works well for linearly separable data).
• Polynomial Kernel: Creates curved or complex shapes for non-linear but not too complex relationships between data points.
• Radial Basis Function Kernel (RBF): (Gaussian kernel) Determines similarity based on the distance between points; good for complex relationships in data.
• Sigmoid Kernel: Based on the hyperbolic tangent function; less common in practice.

Classification Algorithms

• Logistic Regression: Predicts categorical variables using probabilistic values (between 0 to 1) to represent the probability of either output (0 or 1).
• Support Vector Machines (SVM): Creates the best line (hyperplane) that separates the data points into their respective categories in order to accurately group new data.
• K-Nearest Neighbors (KNN): Assigns new data points to the class or category which the closest neighbors belong to based on a similarity function (e.g., Euclidean distance) and the closest 'K' neighbors to the new data points.
• Decision Trees: Tree-structured classifier; internal nodes represent feature values, branches represent decision rules, and each leaf node represents the outcome
• Random Forest: Building numerous decision trees using random subsets of the dataset to minimize the risk of overfitting while improving prediction performance.

Regression Algorithm

• Linear Regression: Predicts continuous variables based on a relationship between the dependent variable and one or more independent variables.
• Describes a linear relationship between variables.

Clustering

• K-Means Clustering: Groups unlabeled data points into distinct clusters based on the similar characteristics of the data points.

Types of Datasets/Problems

• Classification: Predicting a class or discrete value (e.g., male/female, spam/not spam)
• Regression: Predicting a quantity or continuous value (e.g., salary, house price)

Exercises

• Classification goal: Dividing data into distinct categories
• Regression model: Linear Regression
• Regression scenarios: Estimating time, predicting sales.
• Binary Classification: Predicting pass/fail, spam/not spam.
• Problem with continuous target variable: Regression.

Probability

• Probability: Measure of the likelihood that an event will occur.
• Ranges from 0 to 1.

Statistics

• Central Tendency: Mean, Median, Mode
  • Mean: Average of values
  • Median: Middle value
  • Mode: Most frequent value
• Variability: Range, Variance, Standard Deviation
  • Range: Difference between highest and lowest values
  • Variance: Average of squared deviations from the mean
  • Standard Deviation: Square root of the variance
• Correlation: Relationship between two variables; does not imply causation.
• Causation: Relationship between cause and effect.
• Statistical Study Designs:
  • Sample Study: Selecting a subset to study the entire population; calculate parameters from the sample.
  • Observational Study: Observing without manipulating variables; assesses association, not causation.
  • Experimental Study: Manipulating variables to assess causation.