Machine Learning Chapter 3: Classification using KNN and Weka

K Nearest Neighbor (KNN) Classification

• KNN is a classification algorithm used in machine learning.
• Steps involved in KNN:
  • Assign a value to K (number of nearest neighbors).
  • Calculate the distance between the new data entry and all other existing data entries using Euclidean distance (or other distance measurements).
  • Arrange the distances in ascending order.
  • Find the K nearest neighbors to the new entry based on the calculated distances.
  • Assign the new data entry to the majority class in the nearest neighbors.

Using Weka Software for KNN

• Open Weka software and choose the Explorer option.
• Select the training dataset.
• From Classify, choose Lazy → IBK.
• Select the number of neighbors (e.g., 5).
• Train the model.
• Choose the testing data.
• Right-click on the trained model and choose re-evaluate on the testing data.

Example of KNN Algorithm

• Given a dataset with three columns: Weight, Color, and Sweetness, and two classes: Apple and Orange.
• Calculate the distance between a new data entry (Weight: 160 grams, Color: Red, Sweetness: Sweet) and existing data entries.
• Sort the distances in ascending order and find the 3 nearest neighbors.
• The majority class among the 3 nearest neighbors is Apple, so the new data entry is classified as Apple.

Issues with Nearest-Neighbor Classifiers

• Value of K: choosing the right value of K is important, as a small K can lead to sensitivity to noise points, and a large K can include points from other classes.
• Computational complexity: determining the nearest neighbor of a query point requires computing the distance to all training examples, which can be expensive.
• Storage requirements: all training data must be stored.
• High-dimensional data: the required amount of training data increases exponentially with dimension, and computational cost also increases dramatically, known as the "curse of dimensionality".