Statistical Learning: Model Selection and Regularization

A statistical model where the response variable is a linear combination of predictor variables, with added noise represented by the error term 'e'. It's used to predict an outcome based on a set of input variables.

Techniques that aim to improve linear models by controlling the complexity of the model to prevent overfitting and improve interpretability.

The ability of a model to generalize well to unseen data, indicating its ability to accurately predict outcomes for new observations.

A statistical method for finding the best linear relationship between a response variable and predictor variables by minimizing the sum of squared errors.

A scenario where a model performs very well on the training data but poorly on unseen data, indicating it has learned the training data too well and is not generalizable.

Using a subset of the available features to build a simpler and more interpretable model while improving prediction accuracy.

A technique to prevent overfitting by shrinking or penalizing the estimated coefficients towards zero, making the model more robust and less complex.

A class of methods that aims to reduce the dimensionality of data by finding a set of lower-dimensional representations that capture most of the information in the original data.

A technique for selecting the optimal model size by dividing the data into training and validation sets, then calculating the error on the validation set for each model size and choosing the one with the lowest error.

A method that splits the data into k folds, trains the model on k-1 folds, and calculates the error on the remaining fold. This process is repeated k times, and the average error is the cross-validation error.

A rule used in model selection that selects the most simple (smallest) model whose estimated test error is within one standard error of the lowest point on the model size vs. error curve.

A type of shrinkage method that penalizes large coefficients in the linear model to prevent overfitting, leading to a model that emphasizes simplicity and generalizability.

A family of methods that aim to improve the prediction accuracy of models by shrinking the coefficients of the linear model towards zero. This helps prevent overfitting by reducing the influence of features that have weak or no predictive power.

Another type of shrinkage method that performs feature selection by driving the coefficients of irrelevant features towards zero, effectively removing them from the model.

A technique that aims to find the best subset of predictors for a linear regression model by considering all possible combinations of predictors.

A measure of the overall fit of a linear regression model, representing the sum of squared differences between the actual and predicted values.

A method for building regression models by adding one predictor at a time, starting with no predictors and continuing until a stopping rule is met.

This technique is computationally expensive and prone to overfitting when dealing with many predictors.

Using methods like stepwise selection is often more efficient and reliable for building models with a large number of potential predictors.

A measure of how well a model explains the variation in the data, ranging from 0 to 1, where 1 indicates a perfect fit. Higher R-squared values suggest better model performance.

A measure used in various models, including logistic regression, to evaluate model fit and predict probabilities of an event occurring. It's a generalization of RSS.

An advantage of forward stepwise selection over best subset selection is that it evaluates fewer models, making it computationally more efficient, especially when the number of predictors is large (p).

Forward stepwise selection does not guarantee finding the absolute best model among all possible combinations of predictors. This is because it considers variables one at a time, potentially missing a combination that might have been even better together.

A technique that starts with all predictors included in the model and iteratively removes the least useful one at each step, until a satisfactory model is reached.

Backward stepwise selection, similar to forward stepwise selection, evaluates fewer models than best subset selection, making it a more efficient alternative when p is large.

Backward stepwise selection, like forward stepwise selection, does not guarantee finding the best model but provides a good and computationally efficient alternative.

Backward stepwise selection requires the number of samples (n) to be larger than the number of variables (p) because it starts with a model containing all predictors. Forward stepwise selection can be used even when n is less than p, making it useful for high-dimensional data where p is much larger than n.

Forward stepwise selection can be used in situations where the number of variables (p) is very large, even when the number of samples (n) is smaller than p. In contrast, backward selection requires n > p, making forward selection a better option for high-dimensional datasets.

PCR (Principal Components Regression) is a dimension reduction technique used for predicting a response variable (Y) based on a set of predictor variables (X). It identifies directions that best represent the predictors, essentially finding linear combinations of the original features.

PCR identifies directions based solely on the predictor variables (X), without considering the response variable (Y). This means the directions found might not be the most effective for predicting the response.

Partial Least Squares (PLS) is another dimension reduction technique that, like PCR, also creates new features as linear combinations of the original ones. It identifies these new features through a supervised approach, considering both the predictor variables (X) and the response variable (Y).

PLS aims to find directions that are both good at explaining the predictor variables (X) and the response variable (Y). This ensures that the new features are relevant and informative for predicting the response.

In PLS, the first direction Z1 is calculated by assigning weights proportional to the correlation between each predictor variable (Xj) and the response variable (Y). This means variables strongly related to the response are given higher weights.

After computing the first direction Z1 in PLS, subsequent directions are found by taking the residuals (differences between actual values and predicted values) of the predictors and repeating the same process of assigning weights based on correlations with the response.

The ℓ1 norm of a coefficient vector 𝛽 is calculated by summing the absolute values of all its components. It's represented as the sum of the absolute values of each element in 𝛽, denoted as ∑ |𝛽|_i.

Lasso is a type of regression that shrinks coefficients towards zero and can force some to become exactly zero, thus performing variable selection. This leads to sparse models - models that only involve a subset of variables. Choosing the right tuning parameter (𝜆) is crucial for optimal performance.

Ridge Regression (L2) shrinks coefficients toward zero but doesn't force them to be exactly zero. Lasso (L1) uses an ℓ1 penalty that can force coefficients to zero, performing variable selection.

The tuning parameter (𝜆) in Lasso controls how much the coefficients are shrunk towards zero. A higher 𝜆 leads to more zero coefficients and a more sparse model. Choosing the right 𝜆 is crucial and is often done using cross-validation.

Lasso can yield coefficient estimates that are exactly zero because its L1 penalty pushes them towards zero. This is unlike Ridge Regression, which uses an L2 penalty and only shrinks coefficients towards zero.

Lasso can be represented through a constrained optimization problem. This involves minimizing the sum of squared errors (the loss function) subject to a constraint on the sum of absolute coefficients (L1 penalty).

While both Lasso and Ridge achieve different trade-offs between bias and variance, Lasso performs variable selection and results in a sparse model. Ridge mainly focuses on reducing variance.

Cross-validation is a technique used to select the optimal 𝜆 for Lasso models. It involves splitting the data into multiple folds, training the model on a portion and validating on the remaining portion, repeating this process for different 𝜆 values to find the one that minimizes the validation error.