Model Performance Evaluation in Data Science

• Evaluating model performance in data science is important to avoid overfitting.
• There are two methods of evaluating models in data science: hold-out and cross-validation.
• In k-fold cross-validation, we divide the data into k subsets of equal size.
• We build models k times, each time leaving out one of the subsets from training and use it as the test set.
• Training errors: Errors committed on the training set
• Test errors: Errors committed on the test set
• Generalization errors: Expected error of a model over random selection of records from same distribution
• Underfitting: when model is too simple, both training and test errors are large
• Overfitting: when model is too complex, training error is small but test error is large
• Reasons for model overfitting: not enough training data, over training (train NN with large number of iterations).
• Model selection is performed during model building.
• The purpose of model selection is to ensure that the model is not overly complex and that generalization error is estimated.
• Two types of evaluation are used in data science: classification evaluation and regression evaluation.
• Classification evaluation uses a confusion matrix to measure the number of correct and incorrect predictions made by the classification model.
• Regression evaluation uses performance metrics such as accuracy and precision.
• The key factors to consider when evaluating a regression model are its sensitivity, specificity, and RMSE.
• The RMSE is a popular measure of error for regression models, but it can only be compared between models whose errors are measured in the same units.
• The RSE can be compared between models whose errors are measured in the different units, while the MAE is usually similar in magnitude to the RMSE but smaller.

This quiz covers the importance of evaluating model performance in data science, methods of evaluation such as hold-out and cross-validation, types of errors, overfitting, model selection, and evaluation in classification and regression. It also discusses key factors like sensitivity, specificity, RMSE, and MAE in regression evaluation.