CH 3

Problems with RSS and R² as Model Selection Measures

• RSS and R² are goodness-of-fit measures of a linear model to the training data
• These measures don't explicitly consider prediction performance
• Adding more predictors to a linear model always decreases RSS and increases R²
• Choosing between linear models with different predictor numbers will result in complex model selection

AIC and BIC as Indirect Measures of Prediction Performance

• AIC and BIC adjust a goodness-of-fit measure on the training set, like training RSS or log likelihood
• They account for model complexity and prevent overfitting

Rationale and Difference Between AIC and BIC

• Both AIC and BIC consist of goodness of fit to training data (-2l) and complexity.
• The goal is to reduce these quantities, but they often conflict each other
• AIC and BIC combine these quantities, using the second quantity to penalize an overfitted model
• AIC and BIC differ in their penalty term: BIC is more stringent than AIC
• BIC tends to yield a simpler final model when used as the model selection criterion

Properties of Residuals in a Linear Model

• Residuals should cluster around zero randomly, both on their own and when plotted against fitted values
• Homoscedasticity: Residuals should possess approximately the same variance
• Normality: Residuals should be approximately normally distributed

Advantages and Disadvantages of Polynomial Regression

• Polynomial regression handles more complex relationships between the target and predictors than linear regression
• Polynomial model regression coefficients are more difficult to interpret
• There isn't a simple rule for choosing the value of m (the order of the polynomial term), which is a hyperparameter

Handling Categorical Predictors in a Linear Model

• Categorical predictors are handled via binarization.
• Binarization converts a categorical predictor into a collection of artificial "binary" variables (dummy or indicator variables)
• Each binary variable indicates one level of the categorical predictor
• These dummy variables serve as predictors in the model equation

Meaning of Interaction

• Interaction occurs when the association between one predictor and the target depends on the value (or level) of another predictor

Problems with Collinear Variables in a Linear Model

• Collinear variables may not provide much additional information because their values can be deduced from other variables' values
• This leads to redundancy
• Exact collinearity results in a rank-deficient model, which R can detect
• Real collinearity issues arise when features are strongly related but not perfectly collinear

How Best Subset Selection Works and Its Limitations

• Best subset selection fits a separate linear model for each possible combination of available features
• It selects the "best subset" of features to form the best model
• For p features, fitting 2^p models makes it computationally difficult to implement

Stepwise Selection as an Alternative to Best Subset Selection

• Stepwise selection determines the best model from a restricted list of candidate models by sequentially adding/dropping features
• For p features, the maximum number of models to fit is 1 + p(p+1)/2 which is far less than best subset selection

Drawbacks of Adding/Dropping Multiple Features in Stepwise Selection

• Adding/dropping multiple features can significantly affect a feature's significance because of correlations with others
• A feature that is significant on its own may become insignificant in the presence of another feature

Differences Between Forward and Backward Stepwise Selections

• Forward stepwise selection starts with an intercept-only model, while backward stepwise selection starts with the full model
• Forward selection is more likely to result in a simpler and more interpretable model because it starts simpler

Regularization

• Regularization generates coefficient estimates by minimizing a modified objective function
• Using RSS as the starting point, regularization adds a penalty term reflecting model complexity
• Regularization minimizes the augmented objective function
• λ (lambda) is the regularization parameter that controls regularization extent
• fR(β) is the penalty function that captures the size of the regression coefficients
• The product of the regularization parameter and the penalty function is the regularization penalty

Modification Before Fitting a Regularized Regression Model

• Standardize predictors by dividing by their standard error
• This puts all predictors and coefficients on the same scale

How the Regularization Parameter Affects a Regularized Model

• When λ = 0, the regularization penalty vanishes and coefficient estimates are identical to OLS estimates
• As λ increases, regularization becomes more severe
• Increased pressure forces coefficient estimates closer to zero, reducing model flexibility

Hyperparameters of a Regularized Model

• λ and α are hyperparameters since they are pre-specified inputs not determined during optimization
  • A fine grid of (λ, α) values is set up in advance
  • CV error is computed for each pair of candidate values of (λ, α)
  • The pair producing the lowest CV error is chosen

Differences Between Stepwise Selection and Regularization

• Model selection: Stepwise selection drops certain predictors entirely, while regularization pushes coefficient estimates towards zero
• Parameter indexing: Stepwise selection uses the number of features as a measure of model complexity, while regularization uses λ as an indirect measure

The One-Standard-Error Rule

• The simplest regularized regression model with a CV error within "one standard error" of minimum error has similar predictive performance
• Selecting the simplest and most interpretable model among these is preferred.