Understanding Complex Numbers

Complex Numbers - Definition

• A complex number is expressed as $a + bi$.
• $a$ and $b$ represent real numbers.
• $i$ is the imaginary unit, where $i = \sqrt{-1}$.
• The real part: $a$.
• The imaginary part: $b$.
• $4i$ is purely imaginary.
• $5$ is a real number.

Complex Number - Operations

• Addition: $(a + bi) + (c + di) = (a + c) + (b + d)i$.
  • $(2 + 3i) + (1 - i) = 3 + 2i$.
• Subtraction: $(a + bi) - (c + di) = (a - c) + (b - d)i$.
  • $(5 - 2i) - (3 + i) = 2 - 3i$.
• Multiplication: $(a + bi)(c + di) = (ac - bd) + (ad + bc)i$, noting $i^2 = -1$.
  • $(1 + i)(2 - i) = 3 + i$.
• Division: $\frac{a + bi}{c + di} = \frac{(ac + bd) + (bc - ad)i}{c^2 + d^2}$, achieved by multiplying the numerator and denominator by the conjugate of the denominator.
  • $\frac{1 + 2i}{3 - i} = \frac{1}{10} + \frac{7}{10}i$.

Complex Number - Complex Conjugate

• For $a + bi$, the complex conjugate is $a - bi$.
• Properties:
  • $(a + bi)(a - bi) = a^2 + b^2$ yields a real number.
  • $(a + bi) + (a - bi) = 2a$ yields a real number.

Complex Number - Absolute Value (Modulus)

• Defined as $|a + bi| = \sqrt{a^2 + b^2}$.
  • $|3 - 4i| = 5$.

Complex Number - Argument

• Represented as $\theta = \arctan(\frac{b}{a})$.
• Adjust the angle appropriately depending on the complex number's quadrant.

Complex Number - Polar Form

• Represented as $r(\cos \theta + i \sin \theta)$.
  • $r = |a + bi|$ as the modulus.
  • $\theta$ as the argument.
• Euler's formula: $a + bi = re^{i\theta}$.

Complex Number - De Moivre's Theorem

• States that $[r(\cos \theta + i \sin \theta)]^n = r^n(\cos(n\theta) + i \sin(n\theta))$ or $(re^{i\theta})^n = r^ne^{in\theta}$.
  • $(1 + i)^4 = -4$.

M-estimation - Introduction

• M-estimators are a broad class of estimators that include maximum likelihood estimators as a special case.
• An M-estimator $\hat{\theta}$ is defined as the value that minimizes $\sum_{i=1}^{n} \rho(y_i, f(x_i, \theta))$, where $\rho$ is a loss function.

M-estimation - Examples

• Least squares: $\rho(y_i, f(x_i, \theta)) = (y_i - f(x_i, \theta))^2$
• Maximum likelihood estimators: $\rho(y_i, f(x_i, \theta)) = -\log p(y_i | x_i, \theta)$

M-estimation- Properties

• M-estimators are generally consistent and asymptotically normal under certain regularity conditions.

M-estimation - Consistency

• Under certain regularity conditions, M-estimators are consistent, meaning that $\hat{\theta} \rightarrow \theta_0$ in probability as $n \rightarrow \infty$, where $\theta_0$ is the true value of the parameter.

M-estimation- Asymptotic Normality

• Under certain regularity conditions, M-estimators are asymptotically normal, meaning that $\sqrt{n}(\hat{\theta} - \theta_0) \xrightarrow{d} N(0, V)$ where $V$ is the asymptotic variance. The asymptotic variance can be estimated using the sandwich estimator: $\hat{V} = H^{-1} B H^{-1}$.
• $H = \frac{1}{n} \sum_{i=1}^{n} \nabla^2 \rho(y_i, f(x_i, \hat{\theta}))$
• $B = \frac{1}{n} \sum_{i=1}^{n} [\nabla \rho(y_i, f(x_i, \hat{\theta}))]^2$

M-estimation - Robustness

• M-estimators can be made robust to outliers by choosing a loss function $\rho$ that is less sensitive to large errors.

M-estimation - Examples of Robust Loss Functions

• Huber loss: $\rho(u) = \begin{cases} \frac{1}{2}u^2 & \text{for } |u| \le k \ k|u| - \frac{1}{2}k^2 & \text{for } |u| > k \end{cases}$
• Tukey's biweight loss: $\rho(u) = \begin{cases} \frac{k^2}{6}[1 - (1 - (\frac{u}{k})^2)^3] & \text{for } |u| \le k \ \frac{k^2}{6} & \text{for } |u| > k \end{cases}$

M-estimation - Advantages

• Generality: M-estimators include many common estimators as special cases.
• Flexibility: The choice of loss function allows for tailoring the estimator to the specific problem at hand.
• Robustness: M-estimators can be made robust to outliers.

M-estimation - Disadvantages

• Computation: M-estimators can be computationally expensive to compute.
• Regularity conditions: M-estimators require certain regularity conditions to hold in order to be consistent and asymptotically normal.
• Choice of loss function: The choice of loss function can be difficult.

M-estimation - Conclusion

• M-estimators are a powerful and flexible class of estimators that can be used in a wide variety of applications. They offer a good balance between generality, efficiency, and robustness.

Linear Transformation - Definition

• For vector spaces $V$ and $W$ over $F$, a linear transformation $T: V \rightarrow W$ satisfies:
  • $T(\mathbf{v}{1} + \mathbf{v}{2}) = T(\mathbf{v}{1}) + T(\mathbf{v}{2})$ for all $\mathbf{v}{1}, \mathbf{v}{2} \in V$
  • $T(c \mathbf{v}) = c T(\mathbf{v})$ for all $\mathbf{v} \in V, c \in F$

Linear Transformation - Examples

• $T: \mathbb{R}^{2} \rightarrow \mathbb{R}^{3}$ where $T(x, y) = (x + y, x - y, 3x)$
  • Demonstrates the properties of a linear transformation.
• $T: P(\mathbb{R}) \rightarrow \mathbb{R}$ where $T(f) = f(2)$
  • Is a linear transformation example.
• $T: C^{1}(\mathbb{R}) \rightarrow C(\mathbb{R})$ where $T(f) = f'$
  • A linear transformation example.
• $T: M_{n \times n}(F) \rightarrow M_{n \times n}(F)$ where $T(A) = A^{t}$
  • Is a linear transformation example.

Linear Transformation - Theorem

• If ${\mathbf{v}{1}, \dots, \mathbf{v}{n}}$ is a basis for $V$ and $\mathbf{w}{1}, \dots, \mathbf{w}{n} \in W$, there exists a unique linear transformation $T: V \rightarrow W$ such that $T(\mathbf{v}{i}) = \mathbf{w}{i}$.
• For any $\mathbf{v} \in V$ expressed as $\mathbf{v} = \sum_{i=1}^{n} a_{i}\mathbf{v}{i}$, the transformation is defined as $T(\mathbf{v}) = \sum{i=1}^{n} a_{i}\mathbf{w}_{i}$.

Linear Transformation - Definition: Null Space and Range

• Null Space (Kernel): $N(T) = {\mathbf{v} \in V: T(\mathbf{v}) = \mathbf{0}}$
• Range (Image): $R(T) = {\mathbf{w} \in W: T(\mathbf{v}) = \mathbf{w} \text{ for some } \mathbf{v} \in V}$

Linear Transformation - Theorem and Proof: Null Space and Range

• For vector spaces $V$ and $W$ over $F$ and a linear transformation $T: V \rightarrow W$:
  • $N(T)$ is a subspace of $V$
  • $R(T)$ is a subspace of $W$