Lecture (9): Classification Models And KNN PDF

University of Science and Technology Faculty of Computer Science and Information Technology Department of Computer Science …. Semester 8 Subject: Introduction to Machine Learning Lecture (9): Classificat...

University of Science and Technology Faculty of Computer Science and Information Technology Department of Computer Science …. Semester 8 Subject: Introduction to Machine Learning Lecture (9): Classification Models and KNN ___________________________________________________________________ Instructor: Prof. Noureldien Abdelrahman Noureldien Date: 20-12-2023 8.1 What is the Classification Algorithm? The classification algorithm is a supervised learning technique that is used to identify the category of new observations on the basis of training data. In classification, a program learns from the given dataset or observations and then classifies new observation into a number of classes or groups. Such as, Yes or No, 0 or 1, Spam or Not Spam, cat or dog, etc. Classes can be called as targets/labels or categories. Unlike regression, the output variable of classification is a category, not a value, such as "Green or Blue", "fruit or animal", etc. Since the classification algorithm is a supervised learning technique, hence it takes labeled input data, which means it contains input with the corresponding output. In classification algorithm, a discrete output function(y) is mapped to input variable(x). y=f(x), where y = categorical output The algorithm which implements the classification on a dataset is known as a classifier. There are two types of classification: o Binary Classifier: If the classification problem has only two possible outcomes, then it is called as Binary Classifier. o Examples: YES or NO, MALE or FEMALE, SPAM or NOT SPAM, CAT or DOG, etc. o Multi-class Classifier: If a classification problem has more than two outcomes, then it is called as Multi-class Classifier. Example: classifications of types of crops, classification of types of music. 8.2 Learners in Classification Problems In the classification problems, there are two types of learners: 1. Lazy Learners: Lazy Learner firstly stores the training dataset and wait until it receives the test dataset. In Lazy learner case, classification is done on the basis of the most related data stored in the training dataset. It takes less time in training but more time for `1  predictions. Example: K-NN algorithm 2. Eager Learners: Eager Learners develop a classification model based on a training dataset before receiving a test dataset. Opposite to Lazy learners, Eager Learner takes more time in learning, and less time in prediction. Example: Decision Trees, Naïve Bayes, ANN. 8.3 Types of ML Classification Algorithms Classification algorithms can be further divided into the mainly two category: o Linear Models o Logistic Regression o Support Vector Machines o Non-linear Models o K-Nearest Neighbors o Kernel SVM o Naïve Bayes o Decision Tree Classification o Random Forest Classification 8.4 Evaluating Classification Models The following measures can be used to measure the performance of classifiers. 1. Log Loss or Cross-Entropy Loss Log loss is used instead of MSE, the equation of log loss function works. Equation of Log loss function. y-Actual output, p-probability of the predicted output o It is used for evaluating the performance of a classifier, whose output is a probability value between the 0 and 1. o For a good binary classification model, the value of log loss should be near to 0. o The value of log loss increases if the predicted value deviates from the actual value. o The lower log loss represents the higher accuracy of the model. `2 2. Confusion Matrix The confusion matrix provides a matrix/table as output and describes the performance of the model. It is also known as the error matrix. The matrix consists of predictions result in a summarized form, which has a total number of correct predictions and incorrect predictions. The matrix looks as follows: Confusion Table Where:  TP: True Positive: Actual True values are correctly predicted as True.  FP: False Positive: Actual False values are incorrectly predicted as True  FN: False Negative: Actual True values are incorrectly predicted as False  TN: True Negative: Actual False values are correctly predicted as False Confusion Matrix is a useful machine learning method which allows to measure Recall, Precision, Accuracy, F-Score and AUC-ROC curve.  Sensitivity: Sensitivity computes the ratio of positive classes correctly detected. This metric gives how good the model is to recognize a positive class.  Precision: The precision metric shows the accuracy of the positive class. It measures how likely the prediction of the positive class is correct.  Accuracy  F Score: F1 score is a weighted average score of the true positive (recall) and precision.  Roc Curve: Roc curve shows the true positive rates against the false positive rate at various cut points. It also demonstrates a trade-off between sensitivity (recall and specificity or the true negative rate). `3  K-Nearest Neighbor (KNN) Algorithm for Machine Learning o K-Nearest Neighbor is one of the simplest Machine Learning algorithms based on Supervised Learning technique. o K-NN algorithm assumes the similarity between the new case/data and available cases and put the new case into the category that is most similar to the available categories. o K-NN algorithm stores all the available data and classifies a new data point based on the similarity. This means when new data appears then it can be easily classified into a well suite category by using K- NN algorithm. o K-NN algorithm can be used for Regression as well as for Classification but mostly it is used for the Classification problems. o K-NN is a non-parametric algorithm, which means it does not make any assumption on underlying data. o It is also called a lazy learner algorithm because it does not learn from the training set immediately instead it stores the dataset and at the time of classification, it performs an action on the dataset. o KNN algorithm at the training phase just stores the dataset and when it gets new data, then it classifies that data into a category that is much similar to the new data. o Example: Suppose, we have an image of a creature that looks similar to cat and dog, but we want to know either it is a cat or dog. So for this identification, we can use the KNN algorithm, as it works on a similarity measure. Our KNN model will find the similar features of the new data set to the cats and dogs images and based on the most similar features it will put it in either cat or dog category. `4 Why do we need a K-NN Algorithm? Suppose there are two categories, i.e., Category A and Category B, and we have a new data point x1, so this data point will lie in which of these categories. To solve this type of problem, we need a K-NN algorithm. With the help of K-NN, we can easily identify the category or class of a particular dataset. Consider the below diagram: How does K-NN work? The K-NN working can be explained on the basis of the below algorithm: o Step-1: Select the number K of the neighbors o Step-2: Calculate the Euclidean distance of K number of neighbors o Step-3: Take the K nearest neighbors as per the calculated Euclidean distance. o Step-4: Among these k neighbors, count the number of the data points in each category. o Step-5: Assign the new data points to that category for which the number of the neighbor is maximum. o Step-6: Our model is ready. Suppose we have a new data point and we need to put it in the required category. Consider the below image: `5 o Firstly, we will choose the number of neighbors, so we will choose the k=5. o Next, we will calculate the Euclidean distance between the data points. The Euclidean distance is the distance between two points, which we have already studied in geometry. It can be calculated as: o By calculating the Euclidean distance we got the nearest neighbors, as three nearest neighbors in category A and two nearest neighbors in category B. Consider the below image: `6  o As we can see the 3 nearest neighbors are from category A, hence this new data point must belong to category A. How to select the value of K in the K-NN Algorithm? Below are some points to remember while selecting the value of K in the K-NN algorithm: o There is no particular way to determine the best value for "K", so we need to try some values to find the best out of them. The most preferred value for K is 5. o A very low value for K such as K=1 or K=2, can be noisy and lead to the effects of outliers in the model. o Large values for K are good, but it may find some difficulties. Advantages of KNN Algorithm: o It is simple to implement. o It is robust to the noisy training data o It can be more effective if the training data is large. Disadvantages of KNN Algorithm: o Always needs to determine the value of K which may be complex some time. o The computation cost is high because of calculating the distance between the data points for all the training samples. `7