Statistics Chapter: Regression Analysis

Regression Analysis and Modeling

• Regression analysis is a statistical technique that estimates the relationship between a single dependent variable and one or more independent variables.
• The goal of regression analysis is to tell a story about the relationships between variables and the data, which helps stakeholders adjust their business strategy and decisions.

Introduction to Regression Models

• Regression models are based on a statistical foundation and rely on existing data to inform what we might think other data points will look like.
• Regression models are a family of techniques that use existing information or data points to inform what we might think other data points will look like.

PACE Framework

• PACE stands for Plan, Analyze, Construct, and Execute, and provides a foundation for conducting regression analysis.
• The plan stage involves understanding the data in the problem context, considering what data you have access to, and how the data was collected.
• The analyze stage involves examining the data more closely to choose a model or a couple of models that might be appropriate.
• The construct stage involves building the model in Python or the coding language of choice, selecting variables, transforming data as needed, and writing code.
• The execute stage involves interpreting the results, preparing formal results and visualizations, and sharing them with stakeholders.

Linear Regression

• Linear regression is a technique that estimates the linear relationship between a continuous dependent variable Y and one or more independent variables X.
• The linear in linear regression indicates the kind of relationship that can be visualized on a graph, a line.
• Linear regression allows data analytics professionals to estimate continuous dependent variables.
• The slope refers to the amount we expect Y to increase or decrease per one unit increase of X.
• The intercept is the value of Y when X equals zero.

Correlation and Causation

• Correlation describes a relationship between two variables that tend to increase or decrease together.
• There are two kinds of correlation: positive and negative.
• Positive correlation is a relationship between two variables that tend to increase or decrease together.
• Negative correlation is an inverse relationship between two variables.
• Correlation is not causation, and a data scientist must be mindful of the extent of their claims.
• Proving causation statistically requires much more rigorous methods and data collection than correlation.

Regression Analysis in Practice

• Regression analysis helps data analytics professionals tell nuanced stories without needing to prove causation.
• Regression analysis can be used to answer questions such as which factors are associated with an increase or decrease in product sales.
• Regression analysis can be used to identify relationships between variables, such as which factors make a social service provider increase resources in a given region.
• Regression analysis can be used to identify relationships between variables, such as which factors lead to more or less demand for public transportation.### Linear Regression Analysis
• Focuses on the mean of Y given a particular value of X, denoted by μ (mu)
• μ represents the value on the line in a linear regression
• Parameters: β₀ (beta-zero) and β₁ (beta-one) are used to define a linear relationship
• β₀ is the intercept and β₁ is the slope
• Parameters are properties of populations, not samples, and their true values can't be known

Estimating Parameters

• Estimates of parameters are denoted by β₀-hat and β₁-hat
• β₀-hat and β₁-hat are calculated using the sample data
• The hat symbol indicates that they are estimates of parameters

Ordinary Least Squares (OLS) Estimation

• A method that minimizes the sum of squared residuals to estimate parameters in a linear regression model
• Used to calculate β₀-hat and β₁-hat

Residuals and Sum of Squared Residuals

• Residual: ε (epsilon) = observed value - predicted value
• Sum of squared residuals: the sum of the squared differences between each observed value and the associated predicted value

Linear Regression Equation

• Y = β₀ + β₁X
• Y: continuous dependent variable
• X: independent variable
• β₀: intercept
• β₁: slope

Logistic Regression

• Models a categorical variable based on one or more independent variables
• Dependent variable: categorical (e.g., 0 or 1, yes or no)
• Independent variable: continuous (e.g., minutes spent on a webpage)
• Link function: connects the dependent variable to the independent variables

Differences between Linear and Logistic Regression

• Linear regression: continuous dependent variable, models the mean of Y
• Logistic regression: categorical dependent variable, models the probability of Y
• Linear regression: Y = β₀ + β₁X
• Logistic regression: uses a link function to connect the probability of Y with X

PACE Framework

• Planning: consider how the data was collected and what the business needs are
• Analyzing: perform EDA to determine if the data meets the model assumptions
• Constructing: build the model using the data
• Executing: communicate the results to stakeholders### Model Assumptions in Simple Linear Regression
• Model assumptions act as a bridge between the analyze and construct phases of the PACE framework
• Assumptions should be checked before and after model construction
• Data visualizations can help determine if model assumptions are met

Four Key Assumptions of Simple Linear Regression

• Linearity: The relationship between X and Y should be linear
  • Checked using scatter plots of X and Y
  • If the points appear to fall along a straight line, the assumption is met
• Normality: Residual values should be normally distributed
  • Checked using a quantile-quantile (Q-Q) plot of the residuals
  • If the points appear to form a straight diagonal line, the assumption is met
• Independent Observations: Each observation in the dataset should be independent
  • Checked using contextual information and a scatter plot of fitted values versus residuals
  • If the points appear to be randomly scattered, the assumption is met
• Homoscedasticity: The variance of the residuals should be constant across all levels of X
  • Checked using a scatter plot of fitted values versus residuals
  • If there is no pattern in the scatter plot, the assumption is met

Applying Model Assumptions to a Dataset

• Example dataset: Penguin structural measurements and body mass
• Bill length and body mass are positively correlated
• Flipper length and bill length are positively correlated
• Body mass and flipper length are correlated

Building a Simple Linear Regression Model

• Import necessary libraries (Pandas, Seaborn, Statsmodels)
• Load the dataset and create a data frame
• Use the pair plot function to visualize the relationships between variables
• Subset the data to isolate the variables of interest (bill length and body mass)
• Create a regression formula and an OLS object
• Fit the model to the data and print the results
• Use the summary method to print a table of statistics

Interpreting the Results

• Coefficients: Intercept (β0) and slope (β1)
• Linear equation: Y = β0 + β1X
• Interpretation: For every 1 millimeter increase in bill length, body mass increases by 141.19 grams on average

Checking Model Assumptions

• Calculate fitted values and residuals
• Create a scatter plot of fitted values versus residuals to check for homoscedasticity and independence
• Create a histogram of residuals to check for normality
• Use a Q-Q plot to verify normality

Model Evaluation

• Focus on the construct phase of the PACE framework
• Evaluate the performance and accuracy of the model
• Communicate uncertainty using confidence intervals and confidence bands
• Metrics: R-squared, mean squared error (MSE), mean absolute error (MAE)

R-Squared (Coefficient of Determination)

• Measures the proportion of variation in Y explained by X
• Range: 0 (X explains 0% of variance in Y) to 1 (X explains 100% of variance in Y)
• Example: Bill length explains about 77% of the variance in body mass

Model Evaluation Processes

• Use part of the dataset to build and test the model
• Calculate measures of difference between actual and predicted values (e.g. sum of squared residuals)
• Use the model to make predictions for new data

Regression Analysis and Modeling

• Regression analysis estimates the relationship between a dependent variable and one or more independent variables.
• It helps stakeholders adjust their business strategy and decisions by telling a story about the relationships between variables and data.

Introduction to Regression Models

• Regression models rely on existing data to inform what other data points might look like.
• They are a family of techniques that use existing information or data points to make predictions.

PACE Framework

• PACE stands for Plan, Analyze, Construct, and Execute, providing a foundation for conducting regression analysis.
• Plan Stage: Understand the data in the problem context, consider available data, and how it was collected.
• Analyze Stage: Examine the data to choose a model or models that might be appropriate.
• Construct Stage: Build the model in Python or coding language of choice, select variables, transform data, and write code.
• Execute Stage: Interpret the results, prepare formal results and visualizations, and share them with stakeholders.

Linear Regression

• Linear regression estimates the linear relationship between a continuous dependent variable Y and one or more independent variables X.
• Slope: The amount Y is expected to increase or decrease per one unit increase of X.
• Intercept: The value of Y when X equals zero.

Correlation and Causation

• Correlation: A relationship between two variables that tend to increase or decrease together.
• Types of Correlation: Positive (increase/decrease together) and Negative (inverse relationship).
• Correlation vs. Causation: Correlation does not imply causation; proving causation requires more rigorous methods and data collection.

Regression Analysis in Practice

• Applications: Identify relationships between variables, answer questions (e.g., factors affecting product sales), and identify factors leading to more/less demand for public transportation.
• Regression analysis helps tell nuanced stories without needing to prove causation.

Linear Regression Analysis

• Focus: The mean of Y given a particular value of X, denoted by μ (mu).
• Parameters: β₀ (beta-zero) and β₁ (beta-one) define a linear relationship.
• β₀: The intercept, and β₁: The slope, which are properties of populations, not samples.

Estimating Parameters

• Estimates: β₀-hat and β₁-hat are calculated using sample data.
• The hat symbol indicates they are estimates of parameters.

Ordinary Least Squares (OLS) Estimation

• A method that minimizes the sum of squared residuals to estimate parameters in a linear regression model.
• Used to calculate β₀-hat and β₁-hat.

Residuals and Sum of Squared Residuals

• Residuals: The difference between observed and predicted values.
• Sum of Squared Residuals: A measure of the total deviation between observed and predicted values.