Machine Learning Classification vs Clustering

Data Science Tools and Software

• This is a presentation title slide, and likely part of a larger data science course.
• It is about classification and regression tools in data science.

Classification vs Clustering

• Classification involves identifying known categories, such as recognizing patterns in data.
• Unsupervised learning distinguishes between classification and clustering—clustering involves working with unknown categories.

Pattern Recognition Tasks

• The first task, classification, requires finding a model within a provided dataset to categorize data points.
• Clustering groups data points based on similarity to other data points. Data points within a cluster are similar, while points in separate clusters are dissimilar.
• Association rule discovery is another related task that finds which combined items tend to occur together.

Pattern Recognition Applications

• This section details specific applications of pattern recognition in various domains such as Document image analysis, optical character recognition, document classification, internet search, and more.

Linear Classifiers

• Linear classifiers separate data points based on a linear equation (w x + b=0, where 'w' and 'b' are learned parameters.)
• The equation determines which side of the line each point belongs to (+ / -).
• Learners need to find the best linear equation to classify data points.
• Margin is the width to grow the dividing line without hitting any data point.

Maximum Margin

• Maximizing the margin is an important concept in Support Vector Machines that maximizes the separation line by using only the points which are most difficult to separate, also referred to as support vectors.
• Other points don't affect the separating line.

How to do multi-class classification with SVM

• One-to-rest approach creates separate SVM classifiers to classify a single class against the rest of the classes, with the highest score indicating the output/prediction.
• One-to-one approach compares each pair of classification classes to form a decision boundary.

K-Nearest Neighbor (KNN)

• KNN compares new instances to existing points based on the feature vector distance, and decides based on the class label of its k-nearest neighbors with respect to the new point of interest.
• Distance can be Euclidean.

KNN Classification

• A graph example showing Loan amount vs Age and the classification of whether a person is likely to repay their loan, and whether the classification is correct or not.

KNN Classification - Standardized Distance

• Standardized variables are compared to find the nearest neighbor
• Formula to calculate Standardized Variable displayed.

Distance Weighted KNN

• A refinement to KNN that assigns weights to neighboring points based on their distance from the query point. The closer the points, the higher their weight.
• The weights decrease as the distance increases between both points.

KNN Summary

• KNN is efficient for data with fewer features and large datasets.
• It is slow with large datasets and many features, but can work well with irregular shaped target classes, which are not easily distinguished by linear methods.

How to choose K

• The optimal 'k' value depends on available data. More 'k' values improves accuracy, but more 'k' increases processing time.

Performance Metrics of Classification

• Error Rate, Accuracy, Sensitivity, Specificity, Precision, Recall, and F-Measure provide different ways to evaluate the performance of classifiers. These metrics calculate the proportion of successful predictions vs. errors based on different parameters.

Example

• Examples of how to calculate performance metrics using provided datasets and algorithms.

KNN Regression Example – (k=1)

• An example using KNN for regression that finds the house price index based on age and loan amount.

KNN Regression - Standardized Distance

• The process of standardizing variables by calculating the standardized variable's value, which is useful for comparing different data points with different scales.

Performance Metrics of Regression

• Root Mean Square Error (RMSE) and Relative Absolute Error (RAE) and Root Relative Squared Error (RRSE) are metrics to measure the error rate between predicted values and actual values during regression analysis.

K-folds cross-validation

• A technique to evaluate a machine learning model by dividing the dataset into k folds, then training and testing the model k times, each time using a different fold as the test set.

Leave-one-out Method

• A specific type of k-fold cross-validation where k = N, and each data point is used once as the test set. This commonly results in the most accurate model during testing..

Cross validation Example

• This section demonstrates calculating accuracy of a model using leave-one-out cross validation.

Rationale for Ensemble Learning

• There's no single algorithm that consistently delivers superior accuracy.
• Combining algorithms that use different attributes, parameters, or even small sample sets to improve the accuracy of models improves results.

Voting

• Voting is an ensemble method that combines the predictions of multiple base learners.

Clustering algorithms

• Clustering algorithms group similar data points. The methods are either hierarchical or partitional.

Algorithm k-means

• K-means is a partitional clustering algorithm to group similar data points into k clusters.

K-means Clustering: Step 1- Step 5

• Steps of the k-means algorithm in the context of data clustering. Illustrations of points moving towards centroid values.

Objective function

• Mathematical representation of the k-means objective function used for minimizing distance between data points from their assigned cluster's centroid.

Comments on the K-Means Method

• K-means is relatively efficient with good performance in finding clusters.
• It can have problems finding the global optimum and requires knowing k in advance. The method isn't good for handling various data types/distributions.

How can we tell the right number of clusters?

• Approximate methods are used to estimate the optimal number of clusters. This includes plotting and evaluating the objective function with respect to k to find the optimal k value.

Clustering Method Examples

• Several techniques/methods for evaluating clustering results.

Database/Python Example

• Python examples with K-means clustering, including how to perform clustering on data with datasets and functions for calculating the Davies-Bouldin Index.

Hierarchical Clustering : Agglomerative

• A hierarchical clustering approach that starts by treating each data point as a separate cluster. Clusters are merged to form larger clusters until all points are combined into a single cluster.

Intermediate State

• Intermediate state of algorithm in the process of clustering.

After Merging

• The process of updating the distance matrix

Distance between two clusters

• How to calculate distance between two clusters based on the similarity between most similar points in both clusters.

Single-link clustering: example

• An example of how to perform single link clustering on data using proximity graphs and evaluating clustering similarities.

Important Python Functions

• Python functions for hierarchical clustering using the AgglomerativeClustering class.

Quiz

• Questions about the various clustering algorithms displayed for practical application in data science.

Density-Based Clustering Methods

• Clustering methods that group data points based on density. It handles data in arbitrary shapes and is robust to noise; but has some limitations; the density needs to be predefined.

Density-Based Clustering: Basic Concepts

• Basic concepts and parameters of density-based clustering methods.

Density-Reachable and Density-Connected

• Relations between data points which are directly or indirectly related to each other based on their mutual density.

DBSCAN: The Algorithm

• A step-by-step explanation of the DBSCAN clustering algorithm.

This quiz tests your understanding of key concepts in machine learning, focusing on classification and clustering techniques. You'll explore differences between the two, the purposes of regression analysis, and performance metrics in data analysis. Assess your knowledge of advanced methods like k-folds and ensemble learning.