Linear Algebra key concepts

Linear Algebra

• Focuses on vector spaces and linear transformations, providing a framework for data representation in machine learning.

Key Concepts in Linear Algebra

• Vectors: One-dimensional number arrays for data points or features, e.g., $\vec{v} = \begin{bmatrix} 1 \ 2 \ 3 \end{bmatrix}$.
• Matrices: Two-dimensional number arrays for datasets or linear transformations, e.g., $A = \begin{bmatrix} 1 & 2 \ 3 & 4 \end{bmatrix}$.
• Scalar: Single number to scale vectors/matrices.
• Transpose: Rows and columns of a matrix are interchanged, e.g., if $A = \begin{bmatrix} 1 & 2 \ 3 & 4 \end{bmatrix}$, $A^T = \begin{bmatrix} 1 & 3 \ 2 & 4 \end{bmatrix}$.
• Dot Product: Sum of products of corresponding components of two vectors, e.g., $\vec{u} = \begin{bmatrix} 1 \ 2 \end{bmatrix}$, $\vec{v} = \begin{bmatrix} 3 \ 4 \end{bmatrix}$, $\vec{u} \cdot \vec{v} = 11$.
• Matrix Multiplication: Resultant matrix elements are dot products of rows of first matrix and columns of second.
• Identity Matrix: Square matrix with ones on diagonal, zeros elsewhere, e.g., $I = \begin{bmatrix} 1 & 0 \ 0 & 1 \end{bmatrix}$.
• Inverse Matrix: When multiplied by original, yields identity matrix, e.g., if $A = \begin{bmatrix} a & b \ c & d \end{bmatrix}$, $A^{-1} = \frac{1}{ad - bc} \begin{bmatrix} d & -b \ -c & a \end{bmatrix}$.
• Eigenvalues/Eigenvectors: Eigenvector scaled by eigenvalue when multiplied by matrix: $Av = \lambda v$.

Calculus

• Deals with continuous change, useful for understanding and optimizing machine learning models.

Key Concepts in Calculus

• Derivatives: Measure function change with input changes; used for slope and optimization.
  • Power Rule: $\frac{d}{dx} x^n = nx^{n-1}$.
  • Chain Rule: $\frac{d}{dx} f(g(x)) = f'(g(x)) \cdot g'(x)$.
• Gradients: Vectors pointing to greatest rate of increase, optimizing multi-input functions: $\nabla f = \begin{bmatrix} \frac{\partial f}{\partial x_1} \ \vdots \ \frac{\partial f}{\partial x_n} \end{bmatrix}$.
• Chain Rule (Multivariable): $\frac{\partial f}{\partial x_i} = \sum_{j=1}^m \frac{\partial f}{\partial y_j} \frac{\partial y_j}{\partial x_i}$.
• Integrals: Measure area under curve; used for probabilities and differential equations.

Probability and Statistics

• Tackles uncertainty, providing modeling framework for developing machine learning algorithms.

Key Concepts in Probability and Statistics

• Probability: Likelihood measure of event occurrence, values from 0 to 1.
  • Conditional probability: $P(A|B) = \frac{P(A \cap B)}{P(B)}$.
  • Bayes' Theorem: $P(A|B) = \frac{P(B|A) P(A)}{P(B)}$.
• Random Variables: Variables with numerical outcomes from random phenomena.
• Distributions: Functions detailing probabilities of random variable values.
  • Normal distribution: $f(x; \mu, \sigma) = \frac{1}{\sigma \sqrt{2\pi}} e^{-\frac{1}{2} (\frac{x - \mu}{\sigma})^2}$.
• Expected Value: Average variable value accross many trials.
  • Discrete: $E[X] = \sum_{i} x_i P(x_i)$.
  • Continuous: $E[X] = \int x f(x) dx$.
• Variance: Measure of variable value spread.
  • Equation: $Var(X) = E[(X - E[X])^2] = E[X^2] - E[X]^2$.
• Standard Deviation: Square root of the variance.
• Maximum Likelihood Estimation (MLE): Method to estimate statistical model parameters.
• Maximum a Posteriori (MAP): Method estimating statistical model parameters using prior knowledge.
• Central Limit Theorem: Distribution of sum/average of independent, identically distributed random variables approximates normal distribution, regardless of original distribution.

Optimization

• Seeks the best problem solution, key for training machine learning models.

Key Concepts in Optimization

• Loss Functions: Measure difference between model predictions and actual values.
  • Mean Squared Error (MSE): $MSE = \frac{1}{n} \sum_{i=1}^n (y_i - \hat{y}_i)^2$.
  • Cross-Entropy Loss: $L = - \sum_{i=1}^C y_i \log(\hat{y}_i)$, where $C$ is the number of classes.
• Gradient Descent: Iterative algorithm minimizing function by steps in negative gradient direction.
  • Update Rule: $\theta = \theta - \alpha \nabla J(\theta)$, where $\theta$ consists of parameters, $\alpha$ is learning rate, and $J(\theta)$ is the cost function.
• Stochastic Gradient Descent (SGD): Gradient descent variant updating parameters with small training data subsets.
• Regularization: Technique adding loss function penalty to prevent overfitting.
  • L1 Regularization (Lasso): Adds sum of absolute weight values to loss function.
  • L2 Regularization (Ridge): Adds sum of squared weight values to loss function.