Bias-Variance Tradeoff (PDF)

 Underfitted model ; high bias and low variance. It uses a simple linear function 80 01X to fit + the data resulting in a ,...

 Underfitted model ; high bias and low variance. It uses a simple linear function 80 01X to fit + the data resulting in a , straight line that doesn't capture the underlying patterns of data points. It has a high error. Well-fitted model ; it uses a quadratic function 00 01X + 82x + Good balance between bias and. variance. It captures the general trend of data points and fit the data well on the training set. Overfitted model ; high variance and low bias. It uses a polynomial degree 80 + 01x +... + 03 which captures the general trend and the noise in the data training. This results in a cavy curve. The gool in regression is to find a balance between bias and variance. So the bias-variance tradeoff is a fundamental concept in machine learning.  BIAS. It represents the error due to overly simplistic assumptions in the learning algorithm. High bias lack of flexibility. So for given data point difference between bias a "x" the is the , , the expected prediction of the model f () and the true value f(x). Bias = Elfra]-f(x). VARIANCE. It represents the due to the model's small error sensitivity to fluctuations in the training data. High variance highly flexible , capture details and noise, this can lead to overfitting. So for , a given data point "x" , the variance measures the varia. bility of a model's predictions around the expected prediction. Variance = El/f(x)-Etf(x)](] The tradeoff occurs because : increasing complexity of the model reduces bias but increases variance. decreasing complexity of the model reduces variance but increases bios. The Mean Squared Error /MSE) is. E [ly-fixi)] = Bias" [f(x)] + Variance [f(] + sa Regularization is a technique to prevent overfitting by adding a penalty term to the loss function. this discourages excessively complex models that may fit in the data rather than the true noise training capturing underlying patterns. 1 (1 Regularization (lasse). It adds the absolute value of the coefficients as a penalty term to the loss function. loss : - /gi - i) + 1 2 wil It drives come weights to zero. It is suitable for high dimensions sparse models. 2 Le Regularization (Ridge). It adds the squared values of the coefficients as a penalty term to the loss function. loss = /gi - i) + + Zai It reduces some weights but doesn't set them to zero. It is suitable for models where all features contribute. 3 Elastic Net (le + (2) Combines L1 and L to take advantage of both methods. loss = Elgi-gi + 11 Zwil + te Zwi ? It is useful when features are correlated. The cross-validation is a model evaluation technique used to assess how well a machine learning model generalizes to unseen data. It helps prevent overfitting : The Hold-Out Cross-Validation is one of the simplest techniques used to evaluate machine learning models. It involves splitting the dataset into two or three subsets : 1) Training Set to train the model. 2) Validation set foptional) used for hyperparameter tuning. 3) Test Set to evaluate the final models performance. The model is trained on the training set and tested on the test set.