Linear and Bivariate Regression Overview

Linear Regression Recap

• A linear function describes a relationship where all data points fall precisely on a line: y = a + βx.

• In social sciences, a statistical linear model is used: ŷ = a + bx or y = a + bx + e

  • This model finds the best-fitting line through a scatter of data points (sample).
  • The population model is E(y)= α + βx or y = a + βx + ε
    • The difference between the actual and predicted value of y is the error term.

• Population mean and standard deviation: μ and σ (unknown constants)

• Sample mean and standard deviation: ӯ and s (variables)

Bivariate Regression Recap

• The slope (b) of the prediction equation (ŷ = a + bx or y = a + bx + e) is dependent on the units of the dependent and independent variables.
• Standardized b is equivalent to Pearson's correlation coefficient (r). This coefficient measures the relationship strength between two variables unit-less.
• To interpret the slope (b) in bivariate regression, units of measurement are needed.

Variance Decomposition in Linear Regression

• In linear regression, the variance in the dependent variable (TSS) is broken down into the Regression Sum of Squares (SSR) and the Error Sum of Squares (SSE): TSS = SSR + SSE.

Hypothesis Testing Recap Page Two

• A test statistic is used in hypothesis testing to decide whether to accept or reject a null hypothesis.
• A test statistic is calculated from the data (like from an experiment or survey), and compared to its expected or critical value.
  • If the test statistic is higher than the critical value, then the null hypothesis will be rejected.

T-test calculations and interpretations

• The t-statistic is calculated as follows: t = (b – 0) / se. 'b' represents the estimated slope and 'se' is standard error of the sample slope.

• Se (standard error) estimates the variability of values obtained if many samples were repeatedly drawn from a population..

• Standard error of the b (se) = S /√Σ(x –x)²

• S = SSE/(n − p − 1), where SSE is the sum of squared errors, n is sample size, and p is number of predictor variables.

• To test a hypothesis:

  • First, set a significance level (e.g., α = 0.05).
  • Then, calculate degrees of freedom: (n – p – 1)
  • Use a t-distribution table to ascertain the critical t value.
  • Lastly , compare the calculated t-value to the critical t-value. If the calculated t is greater than the critical t, then the null hypothesis is rejected.

Hypothesis Testing Procedures

• Option 1: Follow these steps

  • Formulate the null and alternative hypotheses (and choose the type of test).
  • Set the significance level (e.g., α = 0.05).
  • Calculate the observed test statistic.
  • Find the critical value using a t-distribution table with (n − p − 1) degrees of freedom.
  • Reject the null hypothesis if the observed test statistic is greater than the critical value.
  • Interpret the results.

• Option 2: Follow these steps

  • Formulate the null and alternative hypotheses (and choose the type of test).
  • Set the significance level (e.g., α = 0.05).
  • Calculate the observed test statistic.
  • Calculate and read the p-value associated with the observed test statistic.
  • Reject the null hypothesis if the p-value is less than the significance level.
  • Interpret the results

Interpretation and Context

• Look into correlation's strength between the variables.
• A positive correlation between time spent studying and grades means increased study time correlates with better grades.
• Important takeaway: note small samples won't likely meet the sampling assumptions.

Correlation and Causality

• Correlation does not equal causation: a correlation between variables does not prove one causes the other.

• Three criteria for establishing causality between two variables:

  • Association: There must be an observed relationship between the variables.
  • Time order: The cause (exposure) has to precede the effect (outcome).
  • Elimination of alternative explanations: Other variables are controlled and eliminated (via a well designed experimental study).

• Statistical Control-in non experimental studies-can be used in place of experimental controls in a non experimental studies. Like controlling for age in a study comparing income and education.

Three-Variable Relationships

• Spurious association: The apparent relationship between two variables disappears when controlling for a third variable. For example, age can be a suppressor if age is related to both variables and changes the initial relationship between two variables.
• Chain/indirect relationship (mediation): The relationship between two variables is mediated (influenced) by a third variable, in a way an intervening variable.
• Interaction (moderation): The relationship between two variables differs depending on the level of a third variable. For example, the relationship between education and income might look different based on gender.
• Suppressor variables: A third variable (e.g., age in education and income), can mask the relationship between the two primary variables.

Statistical Interaction

• The effect of one independent variable on the dependent variable can be impacted by another independent variable.
• For instance, financial and technological industries may offer higher returns on education than other fields like transportation and retail.