2.3.8. Linear Equations and Systems

Linear Equation

• Linear equations are of the form: $a_1x_1 + a_2x_2 +... + a_nx_n = b$
• $x_1, x_2,..., x_n$ are variables
• $a_1, a_2,..., a_n$ are coefficients
• $b$ is a constant term

System of Linear Equations

• Systems are a set of linear equations.
• The general form of a system is: $$ \begin{cases} a_{11}x_1 + a_{12}x_2 +... + a_{1n}x_n = b_1 \ a_{21}x_1 + a_{22}x_2 +... + a_{2n}x_n = b_2 \... \ a_{m1}x_1 + a_{m2}x_2 +... + a_{mn}x_n = b_m \end{cases} $$
• $a_{ij}$ are coefficients
• $x_j$ are variables
• $b_i$ are constant terms.

Solution of Linear Equations

• Solutions are values for variables $x_1, x_2,..., x_n$
• These values satisfy all equations of the system simultaneously.

Resolution Methods: Substitution

• Expresses one variable in terms of others from one equation.
• Substitutes this expression into other equations to reduce variables.

Resolution Methods: Elimination

• Multiplies equations by constants.
• Adds or subtracts the equations to eliminate certain variables.

Resolution Methods: Matrices

• Uses matrices to represent the system of linear equations.
• Gaussian elimination transforms the matrix into a row-echelon form to facilitate solving.

Matrices: Coefficients

• Formed by the coefficients of the variables.
• For the general system, the coefficient matrix is: $$ A = \begin{bmatrix} a_{11} & a_{12} &... & a_{1n} \ a_{21} & a_{22} &... & a_{2n} \... &... &... &... \ a_{m1} & a_{m2} &... & a_{mn} \end{bmatrix} $$

Matrices: Vectors

• The variable vector is a column vector containing the system's variables: $$ X = \begin{bmatrix} x_1 \ x_2 \... \ x_n \end{bmatrix} $$
• The constant vector is a column vector containing the constant terms: $$ B = \begin{bmatrix} b_1 \ b_2 \... \ b_m \end{bmatrix} $$

Matrices: Equations

• The linear equations can be written in matrix from as: $$AX=B$$

Geometric Interpretation

• For two variables, each linear equation represents a line in the plane.
  • A unique solution is where lines intersect.
  • No solution when lines are parallel and distinctive.
  • Infinite solutions when the lines are identical.
• For three variables, each linear equation represents a plane in space where the solution is the intersection of these planes.
  • A unique solution is where planes intersect at a dot.
  • No solution when the planes do intersect.
  • Infinite solutions is when the planes intersect at line or are identical.

Quantum Mechanics Key Concepts

• Fundamental theory at atomic and subatomic scales.
• Involves quantization, wave-particle duality, uncertainty.
• Quantization: Energy, momentum, etc., take discrete values.
• Wave-Particle Duality: Particles act as waves and vice versa.
• Uncertainty Principle: Precision limit on knowing pairs of properties.

Uncertainty Principle Formula

• Expressed as: $\Delta x \Delta p \geq \frac{\hbar}{2}$ $\Delta x$ = Uncertainty in position $\Delta p$ = Uncertainty in momentum $\hbar$ = Reduced Planck constant

Core quantum Principles

• Wave functions describe the state of the system.
• Probability of finding a particle near point $r$ at time $t$ is given by $|\Psi(r, t)|^2 dr$.
• Wave functions are normalized: $\int_{-\infty}^{\infty} |\Psi(r, t)|^2 dr = 1$.
• Linear Hermitian operators represent physical observables.
• Time evolution is from the time-dependent Schrödinger equation: $i\hbar \frac{\partial \Psi(r, t)}{\partial t} = \hat{H} \Psi(r, t)$.

Quantum Mechanics Applications

• Core to lasers, semiconductors, and MRI.
• Quantum computing uses superposition and entanglement.

Quantum Mechanics Formalism

• States reside in Hilbert space.
• Dirac's bra-ket notation is used to represent quantum states.
• Operators act on states; use eigenvalues.
• $|\Psi\rangle$ represents a quantum state (ket).
• $\langle\Psi|$ represents the dual vector (bra).
• $\langle\Phi|\Psi\rangle$ is the inner product of states $|\Phi\rangle$ and $|\Psi\rangle$.

Quantum Mechanics Key Experiments

• The double-slit experiment demonstrates wave-particle duality.
• The Stern-Gerlach experiment shows angular momentum quantization.
• The Photoelectric Effect shows quantification of light, leading to photons where $E = hf$.

Gradient Descent Algorithms

• Method to minimize a function by iteratively moving towards the steepest decrease.
• Batch Gradient Descent uses all data points, update rule: $\theta = \theta - \eta \nabla J(\theta)$.
• In Stochastic Gradient Descent only uses one data point, update rule: $\theta = \theta - \eta \nabla J_i(\theta)$.
• Mini-Batch Gradient Descent uses a subset of the training data.
• When the batch size $B = 1$ it's equivalens to Stochastic Gradient Descent.
• Wuen batchsize is eual to the size of you training dataset, then you are doing Batch Gradient Descent. $$\theta = \theta - \eta \nabla J_B(\theta)$$

Learning Rate Tuning

• Learning rates that are too large may overshoot the minimum.
• Low learning rates will take long until convergence.

Momentum Method

• Accumulates gradients of previous steps while updating parameters $$V_t = \beta V_{t-1} + \eta \nabla J(\theta)$$ $$\theta = \theta - V_t$$

AdaGrad Algorithm

• Assigns individual learning rates $$\theta_i = \theta_i - \frac{\eta}{\sqrt{G_i + \epsilon}} \nabla J(\theta_i)$$ $$G_i = \sum_{t=1}^T (\nabla J(\theta_i))^2$$

Adam Algorithm

• Combines momentum and AdaGrad concepts. $$V_t = \beta_1 V_{t-1} + (1 - \beta_1) \nabla J(\theta)$$ $$S_t = \beta_2 S_{t-1} + (1 - \beta_2) (\nabla J(\theta))^2$$ $$\hat{V_t} = \frac{V_t}{1 - \beta_1^t}$$ $$\hat{S_t} = \frac{S_t}{1 - \beta_2^t}$$ $$\theta = \theta - \frac{\eta}{\sqrt{\hat{S_t}} + \epsilon} \hat{V_t}$$

Parameter Initialization

• Bad initialization can cause:
  • Neurons to learn the same patterns
  • Gradients exploading and vanishing

Initializations

• Xavier Initialization samples from: $\mathcal{N}(0, \sqrt{\frac{2}{n_{in} + n_{out}}})$.
• Kaiming Initialization samples from: $\mathcal{N}(0, \sqrt{\frac{2}{n_{in}}}})$.

Regularization Techniques

• Process of adding information in order to solve an ill-posed problem or to prevent overfitting.
  • L1 Regularization adds $\lambda ||\theta||_1$ to the cost function.
  • L2 Regularization adds $\lambda ||\theta||_2^2$ to the cost function.
  • Dropout Randomly sets neurons to zero during training.

Data Augumentation

• Data Augmentation Creates new data by augmenting existing data.

Batch Normalization

• Batch Normalization is a technique to standardize the inputs to a network

1. Calculate Batch Mean and Variance: $$\mu_B = \frac{1}{m} \sum_{i = 1}^m x_i$$ $$\sigma_B^2 = \frac{1}{m} \sum_{i = 1}^m (x_i - \mu_B)^2$$
2. Normalize the Batch $$\hat{x_i} = \frac{x_i - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}}$$
3. Scale and shift $$y_i = \gamma \hat{x_i} + \beta$$

Advantages of Batch Normalization

• Allows you to use a higher learning rate.
• Makes the model more robust.
• Reduces the need for other regularization techniques.
• Batch normalization makes the decision surface smoother.

Learn about linear equations and systems of equations, including their general form, coefficients, variables, and constant terms. Discover solution methods such as substitution and elimination to solve these systems effectively.