Model Fit, Bias, and Variance Overview

Model Fit, Bias, and Variance

• Model Fit: A poor model performance can stem from its fit. Overfitting occurs when a model performs exceptionally well on training data but poorly on evaluation data. An example is a line that perfectly fits every point in the training dataset; this is exceptionally good on training data, but will likely not accurately predict new data points. Underfitting occurs when a model performs poorly on training data. An underfit model fails to capture the underlying trend of the data, like a horizontal line on a dataset with an upward trend. The ideal scenario is a balanced model, which fits the overall trend of the data while accounting for normal variance.

Bias

• Bias: Bias represents the error or difference between predicted and actual values. Bias results from choices made during the model-building process. A model with high bias, like a horizontal line in a dataset that shows upward growth, is characterized by significant mismatching between prediction and reality. Underfitting is a strong indication of high bias. Bias can originate, for instance, from using a linear model and non-linear data.

Variance

• Variance: Variance describes how a model's performance will shift when trained on different, similar datasets. Overfitting leads to high variance—the model heavily reacts to slight changes in the training set. High variance points scatter widely. Low variance means the model's reactions to input variations are small and predictable.

Reducing Bias and Variance

• Bias Reduction: Improve the model to better fit the data or increase relevant features (if feature engineering is inadequate)

• Variance Reduction: Use reduced features, split data into many smaller training-test sets (to get a better mean), to lessen the model's sensitivity to specific training subsets.

Overfitting, Underfitting, and Balance

• Overfitting: High variance, low bias. Model performs well on known data but poorly on new data.

• Underfitting: High bias, low variance. Model performs poorly on both training and new data.

• Balanced Model: Low variance, low bias. Model performs well on both training and new data. It captures the data trend with minimal over or under reaction to small test set changes.

Visualization and Summary

• A dart board analogy visualizes models: Low bias and low variance is the target; high bias is far from the center, with high variance implying scattered points.

• A model matrix charts bias and variance. For optimal results, you need a low bias/low variance model (that is, in the middle of the matrix).