Linear Regression and Gradient Descent Quiz

Linear Regression

• The main goal of linear regression is to find the best-fitting linear line that minimizes the sum of the squared errors.

Gradient Descent

• Stochastic Gradient Descent (SGD) with online learning uses a single example per iteration.
• If the learning rate is too large, it can cause oscillations and fail to converge.
• The learning rate controls how quickly the model learns from new data.
• Momentum in gradient descent helps the process by adding a fraction of the previous weight update to the current update, helping to escape local minima.

Analytical Solution

• The analytical solution to linear regression involves minimizing the cost function using normal equations, which have a closed-form solution.

Stochastic Gradient Descent

• Mini-batch in Stochastic Gradient Descent refers to a subset of the training data used to compute the gradient of the loss function.
• A potential drawback of using mini-batch is that it can still be computationally expensive.
• Adjusting the learning rate is necessary to ensure convergence and avoid oscillations.

Optimization Algorithms

• The Adam optimization algorithm automatically adjusts learning rates for different parameters.

Mini-Batch Characteristic

• Mini-batch is not associated with full-batch gradient descent, which uses the entire training dataset to compute the gradient.