Regression Analysis Overview

Problem Setup

• Regression analysis: Predicts a scalar t as a function of scalar x given a data set of pairs (x_i, t_i).
• Inputs: x_i are the input values.
• Targets: t_i are the target values.
• Model: y (prediction) is a linear function of x, represented as y = wx + b.
• Parameters: w (weight) and b (bias) are parameters of the model.
• Hypotheses: Different settings of the w and b values are considered hypotheses.
• Loss function: Measures how well the model fits the data, often using squared error ((y-t)^2).
• Cost function: Average loss function considering all training examples.
• Multivariable regression: Uses multiple inputs (x1, x2, ..., xD) to predict the target t.

Optimization

• Gradient descent: Used to minimize the cost function.

Polynomial regression

• Data: May have non-linear relationships.
• Polynomial regression: Uses a polynomial function to model the relationship between x and y.
• Degree of the polynomial: Determines the complexity of the model.

Linear Classification

• Classification: Predicts a discrete-valued target.
• Binary: Targets are binary (0 or 1).
• Linear: Model is a linear function of x with a threshold at zero.

Logistic Regression

• Surrogate loss function: Used to optimize classification accuracy when the actual loss function is discontinuous.
• Model output: A continuous value representing the probability of the example being positive.
• Logistic function: A sigmoid function used to map the linear model output to a probability.

Generalization

• Underfitting: The model is too simple and doesn't fit the data well.
• Overfitting: The model is too complex and fits the training data perfectly but fails to generalize to new data.
• Validation set: Used to tune hyperparameters (e.g., degree of the polynomial) and select the model that generalizes best.

L2 Regularization

• Regularization: Prevents overfitting by penalizing large weights.
• L2 penalty: Adds a term to the cost function that discourages large weights.
• Tradeoff: The regularized cost function balances fitting the data with minimizing the norm of the weights.

Model Evaluation

• Mean squared error (MSE): Calculates the average squared difference between the predicted and actual values.
• Root mean squared error (RMSE): Square root of MSE.
• Mean absolute error (MAE): Calculates the average absolute difference between the predicted and actual values.
• Cross-validation: Used to estimate the model's performance on unseen data.

Goodness-of-fit

• R-squared (R²): Measures how well the model explains the variation in the target variable.
• p-value: Indicates the statistical significance of the regression coefficient.
• Null hypothesis: Assumes the regression coefficient is zero.
• Reject the null hypothesis: When the p-value is less than 0.05, there is enough evidence to reject the null hypothesis and conclude the coefficient is statistically significant.