Simple Linear Regression Model

• Linear regression is a simple approach to supervised learning
• It assumes a linear relationship between the dependent variable Y and the independent variables X1, X2, ..., Xp.
• In reality, relationships are rarely linear
• Regardless, linear regression is still useful conceptually and practically

Questions to ask about advertising data:

• Is there a relationship between advertising budget and sales?
• What is the strength of the relationship between advertising budget and sales?
• Which media types contribute to sales?
• How accurately can future sales be predicted?
• Is the relationship linear?
• Are there synergies between advertising media?

Simple Linear Regression Model

• Model assumes: Y = β0 + β1X + ε
• β0 and β1 are unknown constants representing the intercept and slope
• Β0 and β1 are also known as coefficients or parameters
• ε is the error term.
• Predicted sales are calculated as ŷ = β̂0 + β̂1x.
• ŷ is a prediction of Y based on X = x
• ^ symbol denotes an estimated value.

Parameter Estimation using Least Squares

• Prediction for Y based on the ith value of X is ŷi = β̂0 + β̂1xi
• ei = yi − ŷi represents the ith residual
• Residual Sum of Squares (RSS) is defined as RSS = e1^2 + e2^2 + ... + en^2
• Expressed equivalently as RSS = (y1 −β̂0 −β̂1 x1)^2 + (y2 −β̂0 −β̂1 x2)^2 +...+(yn −β̂0 −β̂1 xn )^2
• The values that minimize RSS are: β̂1 = Σ(xi − x̄)(yi − ȳ) / Σ(xi − x̄)^2 and β̂0 = ȳ − β̂1 x̄
• Where ȳ and x̄ are the sample means

Assessing Coefficient Accuracy

• Standard error of an estimator measures its variability under repeated sampling
• Standard Error of β̂1: SE(β̂1)^2 = σ^2 / Σ(xi − x̄)^2
• Standard Error of β̂0: SE(β̂0)^2 = σ^2 [1/n + x̄^2 / Σ(xi − x̄)^2]
• σ^2 = Var(ε)
• Confidence intervals can be computed using standard errors
• A 95% confidence interval is a range of values that will contain the true unknown parameter value with 95% probability
• The 95% confidence interval takes the form β̂1 ± 2 · SE(β̂1).
• There is approximately a 95% chance the interval [β̂1 − 2 · SE(β̂1), β̂1 + 2 · SE(β̂1)] contains the true value of β1
• For the advertising data, the 95% confidence interval for β1 is [0.042, 0.053]

Hypothesis Testing With Standard Errors

• Standard errors are used to conduct hypothesis tests on the coefficients
• A common test is testing the null hypothesis of no relationship between X and Y (H0)
• Compare to the alternative hypothesis that there is a relationship (HA)
• Mathematically H0: β1 = 0 versus HA: β1 ≠ 0
• If β1 = 0, the model simplifies to Y = β0 + ε, indicating X is not associated with Y.
• Compute a t-statistic: t = β̂1 / SE(β̂1)
• Under the null hypothesis (β1=0), this statistic has a t-distribution with n-2 degrees of freedom
• Calculate the probability of observing a value equal to |t| or larger, called the p-value
• Advertising data shows the intercept is 7.0325 (p < 0.0001) and for TV is 0.0475 (p < 0.0001)

Assessing Overall Model Accuracy

• Compute the Residual Standard Error
• RSE = √(1/(n-2) * RSS) = √(1/(n-2) * Σ(yi - ŷi)^2)
• R-squared (R^2) measures the fraction of variance explained by the model
• Calculated as R^2 = (TSS - RSS) / TSS = 1 - RSS / TSS -TSS is the total sum of squares, TSS = Σ(yi - ȳ)^2
• In simple linear regression, R^2 = r^2 where r is the correlation between X and Y
• Correlation: r = Σ(xi - x̄)(yi - ȳ) / √Σ(xi - x̄)^2 * Σ(yi - ȳ)^2
• The advertising data RSE is 3.26, R^2 is 0.612 and F-statistic is 312.1

Multiple Linear Regression

• Model assumes the form Y = β0 + β1X1 + β2X2 + ... + βpXp + ε
• βj is interpreted as the average effect on Y for a one-unit increase in Xj, holding other predictors fixed
• For the advertising example, sales = β0 + β1 × TV + β2 × radio + β3 × newspaper + ε

Interpreting Regression Coefficients Can Be Complex

• The ideal scenario is when predictors are uncorrelated (balanced design)
• Each coefficient can be estimated and tested separately
• Interpretations like "a unit change in Xj is associated with βj change in Y" are possible
• Correlations among predictors cause problems
• Variance of coefficients tend to increase dramatically
• Interpretations become difficult, because everything changes
• Claims of causality should be avoided for observational data

Estimation and Prediction

• Given estimates β̂0, β̂1, ..., β̂p, predictions are made with the formula: ŷ = β̂0 + β̂1x1 + β̂2x2 + ... + β̂pxp
• Coefficients β0, β1, ..., βp are estimated by minimizing the sum of squared residuals
• RSS = Σ(yi - ŷi)^2 = Σ(yi - β̂0 - β̂1xi1 - β̂2xi2 - ... - β̂pxip)^2
• Statistical software minimizes RSS to obtain multiple least squares regression coefficient estimates
• Advertising data's coefficients, standard errors, t-statistics, and p-values are:
  • Intercept: 2.939, 0.3119, 9.42, < 0.0001
  • TV: 0.046, 0.0014, 32.81, < 0.0001
  • Radio: 0.189, 0.0086, 21.89, < 0.0001
  • Newspaper: -0.001, 0.0059, -0.18, 0.8599
• Correlations between the media include:
  • TV/Radio: 0.0548, TV/Newspaper: 0.0567, TV/Sales: 0.7822
  • Radio/Newspaper: 0.3541, Radio/Sales: 0.5762
  • Newspaper/Sales: 0.2283

Key Questions

• Is at least one predictor useful in predicting the response?
• Is every predictor helpful to explain Y, or just a subset?
• How well does the model fit the data?
• Is there a way to predict outcomes from sets of various input, and how reliable are these predictions?

Predictor Usefulness

• Use the F-statistic to answer whether at least one predictor is useful
• F = (TSS - RSS) / p / RSS / (n - p - 1), which follows Fp,n-p-1 distribution
• Example values: Residual Standard Error is 1.69, R^2 is 0.897, F-statistic is 570

Identifying Important Variables

• All Subsets or Best Subsets Regression assesses all possible subsets, balancing accuracy (training error) and parsimony (model size)
• For p predictors, there are 2^p possible models
• With many predictors (p = 40), assessing all models becomes computationally infeasible
• Automated approaches search though sets by:

Forward Selection

• Start with a null model that only contains an intercept
• Fit p simple linear regressions and add the variable with the lowest RSS to the null model
• Add the variable that results in the lowest RSS, resulting in all possible two-variable models.
• Continue until a stopping rule is satisfied (e.g., all remaining variables have a p-value above some threshold)

Backward Selection

• Start with a model that includes all variables
• Remove the predictor with the largest p-value (least statistically significant)
• Fit a new model with p - 1 variables and remove the predictor with the largest p-value
• Repeat until a stopping rule is reached, like remaining variables have a significant p-value

Model Selection

• Systematic criteria includes Mallow’s Cp
• akaike information criterion (AIC)
• bayesian information criterion (BIC)
• adjusted R^2
• cross-validation (CV) can helps determine an optimal choice produced through forward or backward selective processes

Qualitative Predictors in Regression

• Categorical factors take a discrete set of values
• Scatterplot matrices are useful
• The four different forms of Qualitative variables are:
  • gender
  • student
  • status
  • ethnicity

Representing Qualitative Predictors

• To investigate credit card balance differences between males and females, create a binary: xi = 1 if female, 0 if male
• Resulting model: yi = β0 + β1xi + εi, translates to β0 + β1 + εi if female and β0 + εi if male
• The credit card data results for gender model are:
• Intercept is 509.80 (p < 0.0001), is 19.73 (p = 0.6690) if female
• Create dummy variables for predictors with > 2 levels, like ethnicity
• Asians have xi1 = 1 if ith person is Asian, 0 if not Asian
• Caucasians have xi2 = 1 if ith person is Caucasian, 0 if not Caucasian
• In the model yi = β0 + β1xi1 + β2xi2 + εi:
• Asians have: β0 + β1 + εi and Caucasians have β0 + β2 + εi
• Africans Americans have β0 + εi
• Always have one fewer dummy variable than levels, African American/the baseline

Findings for Intercepts

• For the Ethnicity model the intercepts are as follows:
  • 531.00 p<0.0001
  • Ethnicity Asians -18.69 p=0.7740
  • Ethnicity Caucasians -12.50 p=0.8260

Extensions to the Linear Model

• Interactions and nonlinearity from previous additive assumptions can be fixed
• Interaction models say that, the effect on salves on the independence of the amount of advertisement
• Sales: β0 + β1 × TV + β2 × radio + β3 × newspaper
• States that the average impact on sales, will always increase by TV and not due to radio

Radio Advertisement Interactions

• A model states that increases in TV value will lead to radio advertisements
• Given a budget the radio ads are half and have of what a TV has on sales through radio
• Synergy exists and the name is known as interaction effect

TV and Radio

• Levels of TV is set to low, but the true sales are set for that
• With this TV model, there is an underestimate on sales

Advertising Radio and TV

• Sales = β0 + β1 × TV + β2 × radio + β3 × (radio × TV) +  = β0 + (β1 + β3 × radio) × TV + β2 × radio + .
• The results coefficients are, for:
• 6.7502: Intercept p<0.0001
• 0.0191: TV p<0.0001
• 0.0289: Radio p=0.0014
• 0.0011: TV x radio p<0.0001

Tv and Radio Key Findings

• A table indicates that interactions are important
• TV radio gives a strong indication as evidence
• interaction models is equal to 96%, it is said to only be around 90% from the basic TV model
• The means a percentage, which is the interaction term

Radio increases

• Increases of that type can lead to a set percentage
• A one TV increases to a good average estimate on total unit sale
• The means to 30 and for a large average estimate on total TV percentage units

Hierachy

• All means will have some p-value, the effects of that will impact the radio
• Models should include the interaction, the p-values is not signficant
• If interaction is in place, also indicate the major effect, if not those coeffcients will not remain significance

Hierarchy and Main Effect

• Rationale impacts the main effects of the interactions of the whole models
• Specializing will impact some of the main effect and the full impact of the model

Qualitative Interations

• Consider the income and the credit
• If they are students the ith will impact them more
• They are set as not students so some ith with those with lower incomes will be impacted

Interactions with Income

• Interactions that happen with an income
• Some models where ither student are also the same to others
• And will continue to increase in the long run