Linear Algebra Chapter 6 Part 2

Least Squares Concepts

• Least squares solution: A vector ( x ) exists such that ( ||Ax-b|| \approx 0 ), indicating the best fit for the equation.
• Least squares error: Represented as ( ||Ax-b|| ), this quantity is what is minimized in least squares problems.

Theorems and Projections

• Best approximation theorem: For a vector ( b ) in a vector space ( V ), the projection ( \text{proj}_W b ) is the best approximation from a finite-dimensional subspace ( W ), satisfying ( ||b - \text{proj}_W b|| < ||b - w|| ) for all other ( w ) in ( W ).
• Orthogonal projection: The least squares solution relates to the equation ( Ax = \text{proj}_W b ), indicating how ( b ) is approximated within the subspace ( W ).

Normal Equations

• Normal equation: Linked to the equation ( Ax = b ), it is represented as ( A^T A x = A^T b ). These equations are essential for solving ( Ax = b ) in least squares.
• Normal system theorem: The consistent nature of ( A^T A x = A^T b ) guarantees that every solution to ( A^T b ) is a least squares solution.

Independence and Invertibility

• Invertible ( A^T A ): The conditions that determine this property include the linear independence of ( A )’s column vectors. This ensures ( A^T A ) is square and invertible.
• Linear independence implications: When column vectors of ( A ) are linearly independent, the system ( Ax = b ) yields a unique least squares solution, expressed as ( x = (A^T A)^{-1} A^T b ). The orthogonal projection onto the column space ( W ) is ( \text{proj}_W b = Ax = A(A^T A)^{-1} A^T b ).

Summary of Normal System Properties

• Consistency of normal system: The equation ( A^T A = A^T b ) is consistent, confirming that each solution to ( A^T b ) represents a feasible least squares solution.