Machine Learning Concepts and Techniques Quiz

Supervised Learning

• Supervised learning is a machine learning method where a model is trained on labeled data.
• The labeled data provides input-output pairs, guiding the model to learn the relationship between them.
• This allows the model to predict outputs for new inputs.
• An example is given using apples, where the model learns that the input "apples" results in the output "It's Apples."

Machine Learning Process

• Data acquisition: Gathering the relevant data.
• Data pre-processing: Cleaning and preparing the data for modeling.
• Model training and building: Training the model on the data.
• Model evaluation: Assessing the model's performance.
• Model testing/deployment: Evaluating the model with new data and deploying it for use.

Traditional Machine Learning Models

• Linear and Logistic Regression
• K-Nearest Neighbor (k-NN)
• Naïve Bayes
• Support Vector Machine (SVM)
• Artificial Neural Network (ANN)
• Decision Tree

Ensemble Classifiers

• Bagging
• Boosting
• Stacking

Course Outcomes

• Students will understand concepts of regression and classifications in solving machine learning problems.
• Students will understand the practical applications and real-world examples of using regression and classification.
• Students will understand uncertainty estimates from classifiers and the "no free lunch" theorem.

Classification vs. Regression

• Classification predicts discrete values (e.g., categories).
• Regression predicts continuous values (e.g., numbers).

Classification

• The objective is to find a function that divides data points based on different parameters, classifying them into categories.
• Input (x) is mapped to discrete output (y).
• Examples include techniques like k-NN, SVM, logistic regression, decision trees, ANN, and Naïve Bayes.

Regression

• The objective is to find correlations between dependent and independent variables.
• Input (x) is mapped to continuous output (y).
• Examples using linear regression, polynomial regression, support vector regression, and other regression types.

Linear Regression

• Used to find the relationship between two or more variables.
• The model aims to find a best-fit line (or a hyperplane in multiple regression) that minimizes residuals.
• Expressed as ŷ = β₀ + β₁x , where ŷ is predicted values, and x and β₁ and β₀ are variables.

Simple Linear Regression

• Aims to find the relationship between one independent variable (x) and a dependent variable (y).
• The model seeks a line of best fit by minimizing the sum of squared errors. ŷ = β₀ + β₁x, where β₀ is the y-intercept and β₁ represents the slope.

Building a Simple Linear Regression Model

• Uses the sklearn library in Python.
• Splits data into training and testing sets for model evaluation.
• Calculates model coefficients to minimize residuals.

Multiple Linear Regression

• Learns the relationship between one dependent variable and multiple independent variables.
• Utilizes a multiple linear equation: y = β₀ + β₁x₁ + β₂x₂ + ... + βₖxₖ + ɛ.

Data Pre-processing

• Data cleaning and preparation steps for model training.
• Categorical variable encoding (e.g., using LabelEncoder or OneHotEncoder).

Predicting Test Set Results

• Using the trained model to predict outputs for the test data set.
• The predicted values are stored in the y_pred vector.

R-Squared

• Represents the proportion of variance that the model explains.
• Higher R-squared values indicate a better model fit.

Regression Applications

• Credit scoring
• Economic growth prediction
• Sports analytics
• Salary prediction based on experience
• House price prediction

Other Regression Techniques

• Polynomial regression: Extends linear regression to non-linear relationships using polynomial terms.
• Lasso regression (L1 Regularization): Reduces the number of predictors by penalizing large coefficients.
• Ridge regression (L2 Regularization): Penalizes large coefficients to prevent overfitting while minimizing the loss.
• Elastic Net Regression: Combines L1 and L2 regularization.
• Poisson Regression: Used for count data, where outputs represent the number of occurrences.
• Quantile Regression: Estimates various quantiles (e.g., median) for the dependent variable, not only the mean.
• Robust Regression: Provides less sensitivity to outliers in the data.

Logistic Regression

• Used for binary classification problems, predicting the probability of an output.
• Logistic function maps input to a probability value between 0 and 1.

Binary Logistic Regression

• Calculates coefficients by maximizing the likelihood function.
• The likelihood function is a measure of the probability of the observed data given the model parameters.

Meaning of Regression Coefficients

• The signs of coefficients indicate the direction of the relationship.
• P-values help determine statistical significance of the factors in the model.
• Odds ratio can interpret how a change in a predictor affects the probabilities of a class.

Logistic Regression Applications

• Credit scoring
• Medical studies
• Text editing (e.g., sentiment analysis, email sorting)

Naïve Bayes

• A classification algorithm based on Bayes' theorem.
• Assumes that the features are independent given the class, simplifying the calculations.

Recap: Bayes' Theorem

• A theorem used to calculate the probability of an event given the evidence.
• Crucial for Naïve Bayes classification.

K-Nearest Neighbor (k-NN)

• A simple, lazy learning algorithm based on the "nearest neighbor" rule.
• Classifies new data points based on the majority class of their k-nearest neighbors.
• Has parameters like k value and distance measure to find neighbors.

Choosing the Right Value for k

• Experiment with different values of k.
• Lower k values can cause instability, leading to noisy predictions.
• Higher k values may cause the model to be less accurate as it might ignore closer neighbors.

k-NN Distance Metrics

• Minkowski distance
• Manhattan distance
• Euclidean distance
• Cosine distance
• Jaccard distance
• Hamming distance

Advantages of k-NN

• Simple and easy to implement
• Versatile for both classification and regression problems
• Doesn't require model building or assumptions

Disadvantages of k-NN

• Computationally expensive with large datasets
• Sensitive to outliers and noise in data
• Choosing an optimal k value can be complex

SVM (Support Vector Machine)

• Aims to find a hyperplane that optimally separates data points into different classes.
• Maximizes the margin between data points on opposite sides of the hyperplane.

Hyperplanes & Support Vectors

• Hyperplanes are decision boundaries.
• Support vectors are closest data points to the hyperplane.
• Distance of support vectors to hyperplane represents how well classes are separated.

Cost Function & Gradient Updates

• The cost function evaluates the model's performance by balancing maximized margin and minimizing loss.
• C parameter controls the model's sensitivity to outliers while maintaining the margin.

Kernel Functions in SVM

• Transforming data into higher dimensions to allow non-linear separation.
• Examples include linear, polynomial, Gaussian (RBF), and sigmoid kernels.
• The kernel function choice affects the model's ability to classify data accurately.

When to Apply SVM

• Binary Classification
• High-dimensional data
• Non-linear decision boundaries
• Relatively small datasets

Pros & Cons of SVM

• Pros: Works well with clear separation, effective in high-dimensional data, accurate for a larger number of samples.
• Cons: Not ideal for large datasets as prediction time is higher. Sensitive to overlapping or noisy data.