Statistical Learning: Regression, Classification

• The content will be on the following topics: Statistical Learning, Linear Regression, Classification and Resampling Methods
• Statistical learning develops methods for understanding relationships between variables that enables predictive modeling, data analysis, and inference, which helps identify trends and patterns in data for improved decision-making and forms the basis for AI and machine learning applications

Types of Statistical Learning

• Supervised learning trains models with labeled data to map inputs to outputs, for example regression for predicting continuous values like house prices, and classification for predicting discrete categories like spam emails
• Unsupervised learning uncovers hidden patterns in unlabeled data, such as clustering to group similar observations and dimensionality reduction to reduce variables while preserving information

Applications of Statistical Learning

• Used in healthcare for predicting outcomes and personalizing treatments
• Used in finance for fraud detection and stock market forecasting
• Used in marketing for customer segmentation
• Used for AI in self-driving cars for object recognition/image classification

Challenges in Statistical Learning

• Overfitting, where models learn noise instead of patterns
• Underfitting, where models are too simple to capture patterns
• Bias-variance tradeoff, balancing model complexity and flexibility
• Ensuring data quality and managing computational complexity

Key Concepts in Statistical Learning

• Statistical learning uses techniques for understanding and predicting from data
• Supervised learning is for predicted outcomes from labeled data
• Unsupervised learning is for pattern extraction from unlabeled data
• Statistical learning has applications in healthcare, finance, and marketing
• Challenges include overfitting, data quality and managing computational requirements

Statistical Learning Chapter 2

• A field of study focusing on models and understanding the relationships between input(predictors) and output variables(responses). The techniques allow for both prediction and interference.

Key goals of statistical learning

• Accurately predicting the response variable based on the predictor values.
• Understanding how the predictor variables will influence, identify significant variables. This can be expressed as Y = f(X) + € (relationship between predictors and response, € = irreducible error term capturing randomness)

Prediction

• Prioritize building a model that minimizes errors when it is applied to unseen data
• An estimated f(x) allows for making forecasts.

Inference

• Aim to understand relationships rather than just making predictions
• Knowing about smoking increases risk of cancer is more important than just predicting the patient's risk

Two broad approaches to estimating are the following:

• Parametric methods use a relationship such as linear relationship (Y =B(0) + B(1)X, requiring less data for interpretation which can perform if you assume the assumption are false.
• Non-Parametric methods don't use a predefined shape, making them more flexible and they can model complex relationships, which requires large data sets and is computationally expensive

Trade-off between prediction accuracy and model interpretability

• Simple Models(Linear Regression for example) allow ease in understanding, but at the cost of capturing complex patterns
• Flexible Models (Neural Networks) are more accurate, but harder to interpret. More flexible models have low bias and higher variance, making them more prone to overfitting.

Supervised learning

• Model is trained on labelled data (outputs Y are give for inputs X). This could include predicting home prices.

Unsupervised learning

• There are no labelled out puts, the goal is to discover hidden patterns (grouping customers by their purchasing behaviour)

Regression vs Classification problems

• Regression predicts continuous variables. Classification predicts categorical variables.

Assessing Model Accuracy

• A good model must balance accuracy and generalizability. Includes the need of measures to get the quality of a fit.

Regression problems

• Includes Mean Squared Error(MSE)(average squared difference between actual and predicted values. Indicates a better-fitting model.

Training vs Test Error

• Training data, error when the model is applied on it. Test data, testing performed on unseen data. Overfitting occurs when the model performs well on training data, but not test data.

Bias-Variance Trade-Off

• Bias is the error from oversimplified assumptions, and variance is the error from sensitivity to small fluctuations in the training data. The goal is minimize both metrics to achieve low test error.

Classification Setting

• Ability to assign labels appropriately, while calculating the proportion of misclassified observation. Bayes Classifier (theoretical best classifier), KNN(non-parametric method that classifies based on vote of neighbors).

Summary of Key Concepts

• Estimating can be done using parametric methods, or non parametric ones
• Supervised learning requires labelled data, unsupervised extracts patterns from unlabelled data.
• Regression model, with MSE rate measures, vs classification models. Bias-Variance trade-off is a key concept for picking a model to new data.

Chapter 3: Linear Regression

• Statistical technique to model the relationship between a dependent variable (response) and one or more independent variables (predictors)

Simple Linear Regression

• Models the relationship between a single predictor variable and a response variable using function Y = intercept + slope * x + error
• Intercept is the expected value of Y when X = 0
• Slope is the amount Y changes due to a one-unit increase in X
• Goal: Estimate the coefficients using observed data

Estimating the Coefficients

• Least squares method minimizes the Residual Sum of Squares (RSS)
• Formulas exist to calculate slope and intercept based on the data

Assessing Model Accuracy

• Residual Standard Error (RSE) measures how much actual values deviate from the regression line
• R-squared measures the proportion of variance explained by the model, with higher values indicating a better fit

Multiple Linear Regression

• Extends the model to multiple predictor variables
• Each coefficient measures the change in Y when the corresponding X changes by one unit, holding others constant

Model Evaluation

• F-test checks if at least one predictor is useful
• T-tests determine if individual predictors are significantly different from zero

Considerations in Regression

• Categorical variables can be included using dummy variables (0/1 encoding)
• Interaction terms allow one predictor’s effect to depend on another
• Polynomial regression uses quadratic or cubic terms for non-linear relationships

Potential Issues:

• Non-linearity, correlation of errors, multicollinearity (addressed using VIF)

Linear Regression vs. K-Nearest Neighbors (KNN)

• Linear regression assumes a linear relationship
• KNN is non-parametric with no assumption, but lower interpretability and high flexibility

Chapter 3 Class PowerPoint

• Goal, is to predict or explain a variable using predictors. Intercept, Slope and error metrics are relevant.
• Indentify impactful predictors on a response and measure strength / prediction.

Simple Linear Regression

• Supervised learning model in the form Linearity, Independence, Normality,Homoscedasticity
• Example in investifating the relationship between advertisting budgets and scales

Multiple Linear Regression

• Model with multiple predictors. No multicollinearity and no high leverage points

Common Problems in Regression Analysis

• Outliers and leverage point issues, can distort estimates and have disproportionate impact; High leverage are observations that have predictor values
• Non Linearity can be solved with polynomial functions
• Heteroscedascity will have difference with variance. Use weighted least square regression
• Muliticollinearity can be solved with PCA

Steps to Improve Regression Models

• EDA to identify relationships, feature selections, Transformation and interactions and validation methods(train/test)

Key Business Analytics Vocabulary

• Regression coefficients is the effect on the predictor to the data
• Residuals are the difference between observed to predicted values
• Multi collinearity is when predictors are correlated, Homoscadisticity has constant variance, opposite if heterosdasticity present
• Adjusted R is when number effects are present. VIF has measure of mulitcollinearity, F statistic measure model. Binary regression tool.

Take Aways

• Liner regression has single predictor models, while multiple linear regression includes multiple factors.

Chapter 4: Classification

• Technique used to predict categorical outcomes to discrete categories, spam vs non-spam emails.

Why not utilize Linear Regression for classification

• Its unreliable and as such discriminant analysis and KNN is preferential

Logistic Regression

• Models the probability of an observation. Ensures values are 0 and 1, and MLE are used to find parameter values.

Multiple / Multinomial Regression

• Extends to multiple predictors, useful when there are many multiple outcome categories

Linear descriminat Analysis

• Assumed distribution of predictors with class, computes probability to determine the class boundary

Quadratic descriminat Analysis

• Allows class to have matrices, with more flexibility to work with training data, assuming predictors are conditionally independent

Analytical Comparison

• Compares methods based on assumptions and descision boundary

GLMs

• Generalizes to various response type by outcomes

Chapter 4 PowerPoint:

• Catagorize observations into predefined values/techniques. Look at models

Multiple Bussiness objectives

• Identifying predictors.

Logistic Regression

• Used for response variable

Class PowerPoint Part 2: Roc, LDA and KNN

• Data and data with high performance by low complexity

Chapter 5 Resampling Methods

• A method to estimate model performance, which will help select models

Cross Validation

• Technique is to assess model's performance

Validation Set Approach

• Radnomly split into subsets, estimating error based on classification (Regression)

K Hold Approach

• Radnomly divided in folds and the process is often repeated. Lower bias is chosen often

Bootstrapping

• Sampling to examine its variablility when working with datasets