Linear Regression and Simple Linear Regression

Linear Regression

• Linear regression is a supervised learning technique used to model the relationship between a dependent variable (target) and one or more independent variables (features).
• The goal is to predict the value of the dependent variable based on the values of the independent variables.

Simple Linear Regression

• Simple linear regression models the relationship between two variables (1 feature and the target) by fitting a linear equation to observed data.
• Notations: m = number of training examples, x's = "input" variable / feature, y's = "output" variable / "target" variable, (x, y) = one training example, (X(i), y(i)) = ith training example.

Hypothesis

• Parameters: θ's are the variables in the model that are adjusted to minimize the cost function.
• How to choose θ's? It is a crucial part of training models.

Gradient Descent

• Objective (Cost) Function: the function that you want to minimize, typically the loss function, which measures the difference between the model's predictions and the actual values.
• Parameters: the variables in the model that are adjusted to minimize the cost function.
• Gradient: the vector of partial derivatives of the cost function with respect to each parameter.
• Learning Rate α: a hyperparameter that controls how much the parameters are adjusted with respect to the gradient during each update.
• Gradient descent works by moving downward toward the pits or valleys in the graph to find the minimum value.
• It seeks to reach the minimum of the cost function and find the best-fit values for the parameters by adjusting the parameters in the direction of the steepest descent.

Gradient Descent Algorithm

• Simultaneous update: update θ0 and θ1 simultaneously.
• Correct: θj = θj - α * (slope)

Convergence of Gradient Descent

• Gradient descent can converge to a local minimum, even with the learning rate α fixed.
• As we approach a local minimum, gradient descent will automatically take smaller steps.
• No need to decrease α over time.

How to Find the Best Learning Rates

• There is no formula to find the right learning rate.
• Try several values of learning rate and for each value plot the number of iterations versus the cost function.
• This is called hyperparameter tuning.

The Number of Iterations

• The cost function will decrease after each iteration if the gradient descent is working optimally.
• Gradient descent converges when it fails to reduce the cost function and stays at the same level.
• The number of iterations required for gradient descent to converge varies considerably.

Making Sure Gradient Descent is Working Correctly

• Example automatic convergence test: declare convergence if the cost function decreases by less than a certain value in one iteration.
• No. of iterations: 43

Gradient Descent Types

• 1-Batch Gradient Descent: each step of gradient descent uses all the training examples (m).
• 2-Stochastic Gradient Descent (SGD): calculate the gradient using just a random small part of the observations instead of all of them.

Linear Regression with Multiple Variables

• Multiple features (variables) example: housing price prediction.
• Cost function for multivariate linear regression: J(θ0, θ1, ..., θn) = 1/2m * Σ(y - (θ0 + θ1*x1 + ... + θn\*xn))^2.

Gradient Descent for Multiple Variables

• Gradient descent algorithm for multiple variables: update each parameter θj simultaneously.