Statistical Modeling and Machine Learning Intro

Introduction to Statistical Modeling

• Statistical modeling uses mathematical equations and probability distributions to represent complex systems.
• Its models contain parameters estimated from data, which enables predictions and inferences.
• It provides a framework for understanding uncertainty and informs data-driven decisions.

Introduction to Machine Learning

• Machine learning involves algorithms that learn patterns from data without explicit programming.
• Its primary focus is on prediction and automation.
• It often utilizes complex models and large datasets.

Overlap Between Statistical Modeling and Machine Learning

• Statistical modeling and machine learning are closely related with overlapping techniques and applications.
• Statistical modeling gives a theoretical basis for machine learning algorithms.
• Machine learning provides tools for creating and assessing statistical models.

Examples of Statistical Modeling

• Linear Regression models the relationship between dependent and independent variables, represented as $y = \beta_0 + \beta_1 x + \epsilon$.
• Logistic Regression models the probability of a binary outcome, represented as $P(y=1) = \frac{1}{1 + e^{-(\beta_0 + \beta_1 x)}}$.
• Time series analysis models data collected over time.
• Analysis of variance (ANOVA) compares means across different groups.

Examples of Machine Learning

• Classification assigns data points to categories.
• Regression predicts a continuous outcome.
• Clustering groups similar data points together.
• Dimensionality reduction minimizes the number of variables while retaining key information.

Types of Machine Learning

• Supervised learning involves training a model on labeled data for predictions on new data.
• Unsupervised learning involves discovering patterns in unlabelled data.
• Reinforcement learning involves training an agent to maximize a reward in an environment.

Model Evaluation (Supervised Learning)

• Training data trains the model.
• Validation data fine-tunes model hyperparameters.
• Test data assesses the final model performance.

Common Evaluation Metrics

• Regression metrics include Mean Squared Error (MSE), Root Mean Squared Error (RMSE), and R-squared, with formulas:
  • $MSE = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y_i})^2$
  • $RMSE = \sqrt{\frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y_i})^2}$
  • $R^2 = 1 - \frac{\sum_{i=1}^{n} (y_i - \hat{y_i})^2}{\sum_{i=1}^{n} (y_i - \bar{y})^2}$
• Classification metrics include Accuracy, Precision, Recall, F1-score, and AUC-ROC.

Bias-Variance Tradeoff

• Bias is error from inaccurate assumptions, causing underfitting by missing relations between features and targets.
• Variance is error from training set fluctuations, causing overfitting by modelling noise, not intended outputs.
• A trade-off exists between bias and variance, that is a balance between accuracy and consistency..
• Complex models usually have low bias and high variance.
• Simple models usually have high bias and low variance.

Regularization Techniques

• Regularization prevents overfitting by adding a penalty to the loss function.
• L1 regularization (LASSO) and L2 regularization (Ridge regression) are examples.

Model Selection

• Model selection involves choosing the best model from a set of candidates.
• It uses validation set performance or cross-validation.

Cross-Validation

• Cross-validation evaluates model performance by training/testing on different data fold combinations.
• K-fold cross-validation divides data into k folds, training on k-1 folds and testing on the remaining one. This is repeated k times.
• Leave-one-out cross-validation is a special case where k equals the number of data points.

Statistical Modeling vs, Machine Learning