Model Evaluation in Data Science

Questions and Answers

What is the purpose of model evaluation?

To find the best model for the data

What is the problem with evaluating a model with the data used for training?

It can generate overfitted models

Which of the following is NOT a method of model evaluation?

Random Sampling

What is the purpose of using a test set when evaluating models?

To avoid overfitting Signup and view all the answers

What is k-fold cross-validation?

A method of dividing the data into k subsets of equal size Signup and view all the answers

What are training errors?

Errors committed on the training set Signup and view all the answers

What is the best method to use when a limited amount of data is available?

Cross-Validation Signup and view all the answers

What is model overfitting?

When a model is too complex Signup and view all the answers

What is model selection used for?

To ensure a model is not overly complex Signup and view all the answers

What is the mean absolute error (MAE)?

A measure of the average difference between the model's predicted values and the actual values Signup and view all the answers

Study Notes

Model evaluation is an integral part of the model development process.
It helps to find the best model that represents our data and how well the chosen model will work in the future.
Evaluating model performance with the data used for training is not acceptable in data science because it can easily generate overoptimistic and overfitted models.
There are two methods of evaluating models in data science: Hold-Out and Cross-Validation.
To avoid overfitting, both methods use a test set (not seen by the model) to evaluate model performance.
Hold-Out uses a subset of the dataset used to build predictive models. Validation set is a subset of the dataset used to assess the performance of model built in the training phase. It provides a test platform for fine tuning model's parameters and selecting the best-performing model.
Cross-Validation uses a limited amount of data to evaluate the model performance. In k-fold cross-validation, we divide the data into k subsets of equal size. We build models k times, each time leaving out one of the subsets from training and use it as the test set. If k equals the sample size, this is called "leave-one-out".
Training errors are errors committed on the training set. Test errors are errors committed on the test set. Generalization errors are expected error of a model over random selection of records from same distribution.
When only a limited amount of data is available, to achieve an unbiased estimate of the model performance we use k-fold cross-validation.
As the model becomes more and more complex, test errors can start increasing even though training error may be decreasing.
Model overfitting occurs when model is too complex, both training and test errors are large.
Model overfitting occurs when a model is built that is too complex for the data it is being trained on.
Model selection is used to ensure that a model is not overly complex.
Model selection is based on estimating generalization error.
Two types of evaluation are used to measure the performance of a classifier: classification evaluation and regression evaluation.
Relative squared error is a more accurate measure of error than the standard error.
Regression is used to predict future events.
Relative squared error is a better measure of error than the standard error when comparing models whose errors are measured in the same units.
The mean absolute error (MAE) is a measure of the average difference between the model's predicted values and the actual values.
The relative absolute error (RAE) is a measure of the average difference between the model's predicted values and the model's own predicted values, divided by the average difference between the model's predicted values and the actual values for the other model.