Predictor Reduction and Model Selection

Multiple Linear Regression

• Linear regression models are used for prediction
• The chapter explains classical statistics vs models for prediction
• Predictive metrics are employed to evaluate model performance using a validation set
• The chapter discusses the issues of using numerous predictors and different variable selection algorithms commonly used in linear regression.

The Python Language

• Pandas is used for handling data
• Scikit-learn is used to build the models
• Statsmodels is avoided due to providing unnecessary information for predictive modeling

Introduction To Linear Regression

• The most popular predictive model is the multiple linear regression model
• This model showcases the relationship between a numerical outcome variable Y and a set of predictors X1, X2, Xp
• Y or Numerical variable is the response, target, or dependent variable
• X1, X2, Xp is a set of predictors, independent variables, input variables, regressors, or covariates
• The equation Y = β0 + β1x1 + β2x2 + ... + βpxp + ε approximates the connection between predictors and outcome.
• Data are used to estimate the coefficients and evaluate model performance in predictive modeling.

Explanatory vs. Predictive Modeling

• Regression modeling means not only estimating the coefficients but also choosing which predictors to include
• Includes predictor forms such as numerical, logarithmic [log (X)], or binned
• Multiple linear regression has many applications, like predicting customer activity, vacation expenditures, staffing needs, cross-selling, and the impact of retail discounts.
• Purposes of linear regression: Explain/quantify input effects on an outcome or predict outcome values for new entries
• Classical stats prioritizes the former
• Predictive analytics is the latter: predicting individual records.
• Explanatory modeling focuses on causal structures and actionable policies, while descriptive modeling quantifies associations
• Predictive modeling focuses on new individual records, making micro-decisions at the record level

Explanatory vs. Prediction Continued

• Explanatory models aim for a close fit to the data, predictive models prioritize new record accuracy
• Explanatory models use the entire dataset to maximize information, while predictive models split data into training and validation sets.
• Explanatory performance measures assess model fit and relationship strength; predictive models focus on predictive accuracy.
• Explanatory models emphasize the coefficients (β); predictive models emphasize the predictions (1).

Estimating the Regression Equation and Prediction

• Ordinary least squares (OLS) estimates regression formula coefficients, minimizing the sum of squared deviations between actual (Y) and predicted (Y) outcome values.
• Using new values x1, x2, ..., xp to predict outcome variables use: Ŷ = β0 + β1x1 + β2x2 + ... + βpxp (equation 6.2)
• Predictions are unbiased and have the smallest mean squared error if:
• Noise follows a normal distribution
• Predictor choices and forms are correct (linearity)
• Records are independent
• Outcome variability is consistent across predictor values (homoskedasticity)
• Even without normally distributed noise (arbitrary distribution), predictions remain strong
• The least-squares estimates still results in the smallest mean squared errors

Predicting the Price of Used Toyota Corolla Cars

• Regression coefficients are used to predict prices using age and mileage of cars
• Chapter 6 shows predicted prices for 20 cars in the validation set
• Mean error (ME) is $103 and RMSE = $1313.
• Below the predictions, the overall measures of predictive accuracy are shown
• Residuals are mostly between ° $2000, according to the histogram of the residuals
• Large positive residuals (under-predictions) are noted.
• Predictive performance in terms of mean error and error percentiles compare regression models.

Variable Selection in Linear Regression

• Data mining regression equations predict a dependent value from many variables
• High speed calculation of modern algorithms encourage taking the "kitchen-sink" approach
• Include many variables, with the consideration that previously unknown relationships will emerge
• Several reasons to exercise caution before throwing possible variables into a model:
• Expense or not being feasible to collect a full complement of predictors for future predictions.
• Ability to measure fewer predictors more accurately
• The more predictors, the higher the chance of missing values in the data.
• Parsimony is an important property of good models, leading to insight of predictor models with few parameters.
• Regression coefficient estimates are unstable due to multicollinearity with a a higher number of variables
• Uncorrelated predictors increase the variance of predictions, while correlated predictors increase the average error of predictions.
• Techniques to reduce the number of predictors include domain knowledge, summary statistics, and computational methods

Techniques To Reduce Predictors

• Domain knowledge is important to understanding and measuring variables relative to predicting an outcome
• Practical reasons for predictor elimination include the expense of information, inaccuracy, high correlation with another predictor etc.
• Summary statistics and graphs are also helpful to examine

Exhaustive search and R Squared

• Exhaustive search assesses all predictor subsets to evaluate the criteria for best models
• Challenge to picking a model: balance simplistic models missing parameters and overly complex models with noise
• Adjusted R² is a popular criterion defined in the equation R2 adj = 1 - (n-1)/(n-p-1) * (1 - R²).
• Adjusted R² looks at the proportion of explained variability in the model for a single predictor while using the squared correlation.
• Higher adjusted R² signals better fit, but unlike R², it penalizes more predictors to avoid artificial increases from simply increasing the number of predictors
• Using adjusted R² can choose a subset to minimize training RMSE
• Akaike Information Criterion (AIC) and Schwartz's Bayesian Information Criterion (BIC) are also useful measures

AIC + BIC

• AIC and BIC measure the goodness of fit, and also a Models with smaller AIC and BIC values are considered better.
• For a fixed subset size, R2, R2 adj, AIC, and BIC all select the same subset
• If comparing models with predictors, compare the models using the same number of predictors

Subset Selection Algorithms in Practice

• Iterative search aims for 'best' predictor subsets but may miss some
• Guarantees for best subset based on criteria like R2 adj
• Three popular iterations: forward, backward and stepwise selections
• Forward selection starts with no and adds one predictor, contributing the greatest value to R². Algorithm halts when additional predictors are insignificant.
• Disadvantage: the algorithm misses groups of pairs.
• Backward elimination starts with all predictors and then one-by-one removes the the least useful. Algorithm halts when the remaining predictors are significant.
• Disadvantage: computing the initial model can be time-consuming/unstable.
• Stepwise regression: forward selection considers dropping insignificant predictors like in backward elimination

Regularization

• Predictor selection of equivalent to setting some of the model coefficients to zero
• Interpretable result by knowing which predictors are retained.
• Regularization/shrinkage “shrinks” coefficients toward zero, and adjusted R² has penalty from number of predictors p.
• Shrinkage imposes a penalty on the model fit using aggregations of coefficient values instead of directly basing the penalty off of the predictors.
• High correlated predictors exhibit coefficients with high standard errors, and changes affect which predictors get emphasized
• Instability from high standard errors is reduces by constraining the combined magnitude

Ridge regression and lasso

• Ridge regression and lasso are the two popular shrinkage methods
• In ridge regression∑j=1P ∑βj2, the penalty is based on the sum of squared coefficients (called L2 penalty)
• Lasso uses ∑j=1P | βj | ∑ (called L1 penalty) for p predictors, where lasso penalty effectively shrinks some coefficients to zero, resulting in predictors.
• Linear regression coefficients estimate by minimizing the training data
• Ridge regression and lasso estimate coefficients by minimizing the training data, but subjecting it to a threshold (t)
• Methods in Python for regularized linear regression are Lasso and Ridge
• Penalty parameter α determines the threshold (α = 1 is default; α = 0 means no penalty and yields ordinary regression)
• LassoCV, RidgeCV, and BayesianRidge automatically select penalty parameter, while LassoCV and RidgeCV use cross-validation
• BayesianRidge uses iteration to derive the whole penalty.
• Normalize with normalize=True when the model for regularized regression is being created.

Explore predictor reduction techniques in statistical modeling. Learn about the importance of initial steps, practical reasons for elimination, and challenges in exhaustive searches. Understand adjusted R-squared and methods for examining potential predictors.