Least Squares Problems

Last Time

• Orthogonal matrices satisfy the equation $Q^T Q = I$.
• The concepts of orthogonal bases and vectors were discussed.
• The Gram-Schmidt process can be used to find the $QR$ decomposition of a matrix ($A = QR$).

This Time

• Focus is on Least Squares (LS) problems.

Problem Definition

• The equation $Ax = b$ has no solution.
• This happens when there are more equations than unknowns.
• The vector $b$ is not in the column space of $A$ ($b \notin C(A)$).

Example Scenario

• The goal is to find the best-fit line through three points: (1, 1), (2, 2), and (3, 4).
• The equation for the line is $y = C + Dx$.
• A system of equations is set up基于 the points:
  • $C + D = 1$
  • $C + 2D = 2$
  • $C + 3D = 4$
• This system is represented in matrix form as: $\begin{bmatrix} 1 & 1 \ 1 & 2 \ 1 & 3 \end{bmatrix} \begin{bmatrix} C \ D \end{bmatrix} = \begin{bmatrix} 1 \ 2 \ 4 \end{bmatrix}$
• $Ax = b$, where $A$ is a $3 \times 2$ matrix.
• No solution exists because lines do not intersect at a single point and $b$ is not in the column space of $A$.

Finding the Closest Vector

• The closest vector to $b$ in $C(A)$ is the projection vector $p$.
• $p = A\hat{x}$, and $\hat{x} = \begin{bmatrix} \hat{C} \ \hat{D} \end{bmatrix}$ such that $A\hat{x} = p$.
• The error vector is $e = b - p$.
• Minimize $||e|| = ||b - Ax||$ or $||e||^2 = ||b - Ax||^2$ using "Least squares".

Normal Equations

• $A^T e = 0$
• $A^T(b - Ax) = 0$
• $A^TAx = A^Tb$ represents the "Normal equations".
• If $A$ has independent columns, then $A^TA$ is invertible.
• The solution is $\hat{x} = (A^TA)^{-1}A^Tb$.
• The projection $p = A\hat{x} = A(A^TA)^{-1}A^Tb$.
• The projection matrix onto $C(A)$ is $P = A(A^TA)^{-1}A^T$.

Example Solution

• Given $A = \begin{bmatrix} 1 & 1 \ 1 & 2 \ 1 & 3 \end{bmatrix}$ and $b = \begin{bmatrix} 1 \ 2 \ 4 \end{bmatrix}$
• $A^TA = \begin{bmatrix} 1 & 1 & 1 \ 1 & 2 & 3 \end{bmatrix} \begin{bmatrix} 1 & 1 \ 1 & 2 \ 1 & 3 \end{bmatrix} = \begin{bmatrix} 3 & 6 \ 6 & 14 \end{bmatrix}$
• $A^Tb = \begin{bmatrix} 1 & 1 & 1 \ 1 & 2 & 3 \end{bmatrix} \begin{bmatrix} 1 \ 2 \ 4 \end{bmatrix} = \begin{bmatrix} 7 \ 17 \end{bmatrix}$
• Solving $\begin{bmatrix} 3 & 6 \ 6 & 14 \end{bmatrix} \begin{bmatrix} \hat{C} \ \hat{D} \end{bmatrix} = \begin{bmatrix} 7 \ 17 \end{bmatrix}$ gives $\hat{C} = -\frac{2}{3}$ and $\hat{D} = \frac{3}{2}$.
• The best-fit line is $y = -\frac{2}{3} + \frac{3}{2}x$, passing through $(\bar{x}, \bar{y}) = (2, \frac{7}{3})$.

Alternative Solution

• Using QR decomposition: $A = QR$.
• Substitute into the normal equations: $A^TAx = A^Tb$ becomes $(QR)^T(QR)x = (QR)^Tb$.
• $R^TQ^TQRx = R^TQ^Tb$, simplifying to $R^T R x = R^T Q^T b$.
• Further simplifies to $Rx = Q^Tb$ because $Q^TQ = I$.
• Solve $Rx = Q^Tb$, which is triangular.

Weighted Least Squares

• Minimize $||\omega(b - Ax)||$, where $\omega$ is a diagonal matrix.
• This is equivalent to minimizing $||\omega b - \omega A x||$.
• This becomes minimize $||b' - A'x||$, where $b' = \omega b$ and $A' = \omega A$.

Non-Independent Columns

• If $A$ does not have independent columns, $A^T A$ is not invertible.
• In this case, $\hat{x} = (A^T A)^+ A^T b$, where $(A^T A)^+$ is the pseudoinverse.

Next time

• Starting chapter 5.
• Orthogonal bases leading to functions.
• Topics include Fourier series.
• Exam 2 is scheduled for November 17.

Modes of Heat Transfer

• Conduction: Heat transfer through a material due to a temperature gradient.
• Convection: Heat transfer due to fluid movement.
  • Natural Convection: Caused by buoyancy.
  • Forced Convection: Caused by external forces.
• Radiation: Heat transfer via electromagnetic waves.

Thermal Resistance

• Conduction Resistance: $R_{cond} = \frac{L}{kA}$ where $L$ is thickness, $k$ is thermal conductivity, $A$ is area.
• Convection Resistance: $R_{conv} = \frac{1}{hA}$ where $h$ is the heat transfer coefficient and $A$ is area.
• Radiation Resistance: $R_{rad} = \frac{1}{h_{rad}A}$ where $h_{rad}$ is the radiation heat transfer coefficient and $A$ is area.

Heat Transfer Rate

• $\qquad Q = \frac{\Delta T}{R_{total}}$

Overall Heat Transfer Coefficient

• $U = \frac{1}{A R_{total}}$
• $Q = UA\Delta T$

Fin Efficiency

• $\eta_{fin} = \frac{Q_{actual}}{Q_{max}}$

Heat Exchangers

• Parallel Flow: Fluids move in the same direction.
• Counter Flow: Fluids move in opposite directions.
• Cross Flow: Fluids move perpendicular to each other.

Log Mean Temperature Difference (LMTD)

• $\qquad \Delta T_{lm} = \frac{\Delta T_1 - \Delta T_2}{ln(\frac{\Delta T_1}{\Delta T_2})}$

Number of Transfer Units (NTU)

• $\qquad NTU = \frac{UA}{C_{min}}$

Vector Space Defined

• Given set E with addition ($E \times E \rightarrow E$, $(u, v) \mapsto u + v$) and scalar multiplication ($\mathbb{R} \times E \rightarrow E$, $(\lambda, u) \mapsto \lambda u$), it is a vector space if:
  • Addition is associative and commutative.
  • There exists additive identity ($0 \in E$) and inverse ($-u \in E$).
  • Scalar multiplication is distributive and compatible.
  • $1u = u$.

Vector Subspace Defined

• Given a subset F of vector space E, F is a subspace if:
  • F is non-empty.
  • F is closed under addition $\forall u, v \in F, u + v \in F$
  • F is closed under scalar multiplication $\forall \lambda \in \mathbb{R}, \forall u \in F, \lambda u \in F$

Linear Combination

• A linear combination with $u_1, ... u_n$ is $\lambda_1 u_1 + ... + \lambda_n u_n$ with scalars $\lambda$.

Linear Independence

• Vectors $u_1,...,u_n$ are independent ONLY if the ONLY solution to $\lambda_1 u_1 + ... + \lambda_n u_n = 0$ is $\lambda_1 = ... = \lambda_n = 0$.

Basis

• The vector space E has a Basis - a set of vectors which are linearly independent and span or "engender" E.

Dimension

• Has Dimension, or the number of vectors in its basis

Linear Mapping Defined

• A linear mapping of E -> F will be $f: E \rightarrow F$ if:
  • $\forall u, v \in E, f(u + v) = f(u) + f(v)$
  • $\forall \lambda \in \mathbb{R}, \forall u \in E, f(\lambda u) = \lambda f(u)$

Kernel

• The kernel of f is $ker(f) = {u \in E \mid f(u) = 0}$, or where all vectors == 0

Image

• The image of an application $f: E \rightarrow F$ is $im(f) = {v \in F \mid \exists u \in E, f(u) = v}$

Physical Situation

• Considering a taut string/rope with length denoted by '$L$'.
• The displacement of the string at position '$x$' and time '$t$' is denoted by $u(x, t)$.
• The displacement is small.

Equation of Motion

• The wave equation is given as:

$$\frac{\partial^2 u}{\partial t^2} = c^2 \frac{\partial^2 u}{\partial x^2}, \quad 0 < x < L, \quad t > 0$$

• where $c = \sqrt{T/\rho}$, T is the string tension, and $\rho$ is the mass density.

Boundary Conditions

• String is fixed, therefore: $$\qquad u(0, t) = 0, \quad u(L, t) = 0, \quad t > 0$$

Initial Conditions

• To find the solution, specify:
  • initial displacement: $u(x, 0) = f(x)$
  • initial velocity: $\frac{\partial u}{\partial t}(x, 0) = g(x), \quad 0 \le x \le L$

Method of Separation of Variables

• Assume general solution in the form of: $\qquad u(x, t) = X(x) T(t)$
• Substitute into the wave equation: $\qquad X(x) T''(t) = c^2 X''(x) T(t)$
• Divide by$c^2 X(x) T(t)$: $\qquad \frac{T''(t)}{c^2 T(t)} = \frac{X''(x)}{X(x)} = -\sigma$
  • Two ordinary differential equations obtained following these steps: $\qquad X''(x) + \sigma X(x) = 0$ $\qquad T''(t) + c^2 \sigma T(t) = 0$

Solving the Spatial Equation

• Spatial equation with corresponding values: $\qquad X''(x) + \sigma X(x) = 0, \quad X(0) = 0, \quad X(L) = 0$

  • The Sturm-Liouville problem with eigenvalues and eigenvectors obtained is:

  $\qquad \sigma_n = \left( \frac{n \pi}{L} \right)^2, \quad X_n(x) = \sin \left( \frac{n \pi x}{L} \right), \quad n = 1, 2, 3, \dots$

Solving the Time Equation

• The time equation with corresponding values: $\qquad T''(t) + c^2 \sigma_n T(t) = 0$ $\qquad T''(t) + \left( \frac{cn\pi}{L} \right)^2 T(t) = 0$
  • The general solution is obtained: $\qquad T_n(t) = A_n \cos \left( \frac{cn\pi t}{L} \right) + B_n \sin \left( \frac{cn\pi t}{L} \right)$

General Solution

• The general solution of the wave equation: $\qquad u(x, t) = \sum_{n=1}^{\infty} \left( A_n \cos \left( \frac{cn\pi t}{L} \right) + B_n \sin \left( \frac{cn\pi t}{L} \right) \right) \sin \left( \frac{n \pi x}{L} \right)$

Determining the Coefficients

• Coefficients determined through the following initial conditions: $\qquad u(x, 0) = f(x) = \sum_{n=1}^{\infty} A_n \sin \left( \frac{n \pi x}{L} \right)$ $\qquad \frac{\partial u}{\partial t}(x, 0) = g(x) = \sum_{n=1}^{\infty} B_n \frac{cn\pi}{L} \sin \left( \frac{n \pi x}{L} \right)$ The orthogonality of the eigenfunctions used to solve the following: $\qquad A_n = \frac{2}{L} \int_0^L f(x) \sin \left( \frac{n \pi x}{L} \right) dx$ $\qquad B_n = \frac{2}{cn\pi} \int_0^L g(x) \sin \left( \frac{n \pi x}{L} \right) dx$

Electromagnetic Spectrum Overview

• Includes all types of electromagnetic radiation.
• Examples of radiation: visible light and radio waves.
• Includes: microwaves, infrared radiation, ultraviolet radiation, x-rays, and gamma rays.

Electromagnetic Spectrum Defined

• Ranges from long wavelengths (radio waves) to short wavelengths (gamma rays).

EM Spectrum Table

• Radiowave:
  • Wavelength: Over 10^-1 meter
  • Frequency: Up to 10^9 Hertz
• Microwave:
  • Wavelength: 10^-3 to 10^-1 meter
  • Frequency: 10^9 to 10^12 Hertz
• Infrared:
  • Wavelength: 7x10^-7 to 10^-3 meter
  • Frequency: 10^12 to 4x10^14 Hz
• Visible:
  • Wavelength: 4x10^-7 to 7x10^-7 meter
  • Frequency: 4x10^14 to 8x10^14 Hz
• Ultraviolet:
  • Wavelength: 10^-8 to 4x10^-7 meter
  • Frequency: 8x10^14 to 3x10^16 Hz
• X-ray:
  • Wavelength: 10^-10 to 10^-8 meter
  • Frequency: 10^16 to 10^18 Hz
• Gamma rays:
  • Wavelength: Less than 10^-10 meter
  • Frequency: Over 10^18 Hertz

Wavelength and Frequency

• Meters are used to measure wavelength, and frequency is measured in Hertz (Hz).
• One Hertz equals one wave per second.

Inverse Proportionality

• Wavelength and frequency are inversely proportional.
• Higher frequency corresponds to a shorter wavelength.

$\qquad c = \lambda v$

• where:
  • c is the speed of light (3.0 x 10^8 m/s)
  • λ is the wavelength
  • v is the frequency

Photon Energy

• Electromagnetic radiation can be described as particles called photons.
• The energy of a photon is proportional to its frequency.

$\qquad E = h v$

• where:
  • E is the energy
  • h is Planck's constant (6.626 x 10^-34 Js)
  • v is the frequency

Energy and Frequency

• Higher frequency means higher energy.
• Gamma rays have the highest energy, and radio waves have the lowest energy.

Uses

• Radiowaves: Radios and televisions transmit information through radiowaves.
• Microwaves: Heats food, used in communications and radar technology.
• Infrared: Heat generation as well as night vision with too little light.
• Visible light: Allows humans to see stuff with their eyes.
• Ultraviolet: Can kill bacteria, sterilize equipment and produce Vitamin D.
• X-Rays: See bones for medical diagnosis.
• Gamma rays: Kill cancer cells in radiation therapy.