Lecture 8 - Model Evaluation and Selection 8d843cc1cb4541bb923d914a9fc1bfc5.pdf

Lecture 8 - Model Evaluation and Selection Knowledge Discovery Process Data Mining Tasks and Their Algorithms Descriptive Task Clustering K-means Hierarchical agglomerative clustering Predictive Task Regression Lecture 8 - Model Evaluation and Selection 1 Classification Naïve Bayes Decision Tree Ind...

Lecture 8 - Model Evaluation and Selection Knowledge Discovery Process Data Mining Tasks and Their Algorithms Descriptive Task Clustering K-means Hierarchical agglomerative clustering Predictive Task Regression Lecture 8 - Model Evaluation and Selection 1 Classification Naïve Bayes Decision Tree Induction Artificial Neural Networks (ANNs) Evaluation of Classification Models Different classification models can be used to discover patterns from a single data set Need systematic ways to evaluate and compare those different models Predict the ability of different models to accurately classify independent test data Repeated k-fold cross-validation Tenfold cross-validation Leave-one-out cross-validation for a small dataset Divide the dataset intoa training set and a test set Lecture 8 - Model Evaluation and Selection 2 Stratification: each class is properly represented in both training and test sets Test data is not used in any way in the formation of the classification model Assuming both training and test data are representative samples of the underlying problem, then the classifier’s performance on the test set will be similar to its future performance on unseen data of this problem Tenfold Cross-Validation Randomly divide the data into 10 equal parts Stratification in each fold Lecture 8 - Model Evaluation and Selection 3 One fold is held out as a training set Remaining 9 folds as a traning set Stratification in both training and test sets Use the training set to create the classifier Use the test set to calculate its predictive accuracy Repeat 10 times (10 rounds) Each fold is held out in turn as the test set Predictive accuracy = mean accuracy over 10 rounds For each classification algorithm, perform the stratified tenfold crossvalidation and obtain its predictive accuracy (mean predictive accuracy over 10 rounds) Select the classification algorithm with the highest predictive accuracy Use all the data to re-train the selected algorithm and produce the final classification model Use the predictive accuracy of the selected algorithm as the predicative accuracy of the final classification model Lecture 8 - Model Evaluation and Selection 4 Maximise the amount of data used to produce the classification model Increase the predictive performance of the final classification model Tenfold Cross-Validation Advantage Reduce the effect of uneven representation of examples in training and test sets - Each example is used exactly once for testing - Training sets between rounds are very similar, having about 90% overlapping Disadvantage Variations Computationally expensive - Each classification algorithm is trained 10 times, with 90% of the data used for training each time Approximation in stratified 10 fold division - Divide data into 10 approximately equal sts in which class values are about the right proportion Statistically more robust accuracy estimate than single training/test set partition Other folds - 5 fold cross validation - 20 fold cross validation Leave-one-out Cross Validation Used for small data sets : n fold cross-validation, where n is the number of examples in the data set Each example is a fold One fold (one example) for testing Remaining n-1 examples for training Repeat n times (n rounds), with each example held out in turn for testing Predictive accuracy of the classification algorithm is the mean predictive accuracy For example, a dataset with 30 examples Advantage Lecture 8 - Model Evaluation and Selection Disadvantage 5 Greatest possible amount of data is used for training in each round, increasing the chance to create an accurate classifier High computational cost Deterministic procedure: no random sampling is involved No stratification in the test set Lecture 8 - Model Evaluation and Selection 6