Lecture 24-27 Numerical Linear Algebra PDF

Chapter 3: Matrix Algebra: Lecture 24: Numerical Linear Algebra (UMA021): Matrix Algebra Dr. Meenu Rani Department of Mathematics TIET, Patiala...

Chapter 3: Matrix Algebra: Lecture 24: Numerical Linear Algebra (UMA021): Matrix Algebra Dr. Meenu Rani Department of Mathematics TIET, Patiala Punjab-India Chapter 3: Matrix Algebra: System of linear equations Chapter 3: Matrix Algebra: System of linear equations Direct Methods 1 Gauss Elimination 2 LU Decomposition(Factorization) Chapter 3: Matrix Algebra: System of linear equations Gauss Elimination Chapter 3: Matrix Algebra: System of linear equations Example Use Gaussian elimination with backward substitution and two-digit rounding arithmetic to solve the following linear system. (The exact solution to each system is x1 = 1, x2 = 1, x3 = 3.) x1 + 4x2 + x3 = 8 5 2 2 x1 + x2 + x3 = 1 3 3 3 2x1 + x2 + 4x3 = 11. Solution: Chapter 3: Matrix Algebra: System of linear equations Chapter 3: Matrix Algebra: System of linear equations Example Apply Gaussian elimination to the system: 0.003000x1 + 59.14x2 = 59.17 5.291x1 6.130x2 = 46.78 using four-digit arithmetic with rounding, and compare the results to the exact solution x1 = 10.00 and x2 = 1.000. Solution: Chapter 3: Matrix Algebra: System of linear equations Chapter 3: Matrix Algebra: Lecture 25: Numerical Linear Algebra (UMA021): Matrix Algebra Dr. Meenu Rani Department of Mathematics TIET, Patiala Punjab-India Chapter 3: Matrix Algebra: System of linear equations Example Apply Gaussian elimination to the system: 0.003000x1 + 59.14x2 = 59.17 5.291x1 6.130x2 = 46.78 using four-digit arithmetic with rounding, and compare the results to the exact solution x1 = 10.00 and x2 = 1.000. Chapter 3: Matrix Algebra: System of linear equations Pivot element In the elimination process, we divide with diagonal element aii at each stage and assume that aii 6= 0. These elements are known as pivot element. Pivot Strategies Partial Pivoting Chapter 3: Matrix Algebra: System of linear equations Partial Pivoting If at any stage of elimination, one of the pivot becomes small (or zero) then we bring other element as pivot by interchanging the rows. This process is called Gauss elimination with partial pivoting. Chapter 3: Matrix Algebra: System of linear equations Example Apply Gaussian elimination to the system: 0.003000x1 + 59.14x2 = 59.17 5.291x1 6.130x2 = 46.78 using partial pivoting and four-digit arithmetic with rounding, and compare the results to the exact solution x1 = 10.00 and x2 = 1.000. Solution: Chapter 3: Matrix Algebra: System of linear equations Chapter 3: Matrix Algebra: System of linear equations Example: Using four-digit arithmetic operations, solve the following system of equations by Gaussian elimination with partial pivoting 0.729x1 + 0.81x2 + 0.9x3 = 0.6867 x1 + x2 + x3 = 0.8338 1.331x1 + 1.21x2 + 1.1x3 = 1.000. This system has exact solution, rounded to four places x1 = 0.2245, x2 = 0.2814, x3 = 0.3279. Compare your answers! Chapter 3: Matrix Algebra: System of linear equations Chapter 3: Matrix Algebra: System of linear equations: Exercise: 1 Use Gaussian elimination with partial pivoting and three-digit chopping arithmetics to solve the following linear system, and compare the approximations with the actual solution [0, 10, 1/7]T. 3.03x1 12.1x2 + 14x3 = 119 3.03x1 + 12.1x2 7x3 = 120 6.11x1 14.2x2 + 21x3 = 139. Chapter 3: Matrix Algebra: System of linear equations LU Factorization: Chapter 3: Matrix Algebra: System of linear equations Example: Determine the LU factorization 2 for matrix 3 A in the 2 linear 3 system 1 1 0 3 1 62 1 1 17 6 7 Ax = b, where A = 6 7 , B = 6 1 7. Then 43 1 1 2 5 4 35 1 2 3 1 4 use the factorization to solve the system. Chapter 3: Matrix Algebra: System of linear equations Solution: Chapter 3: Matrix Algebra: System of linear equations: Exercise: 1 Modify the LU Factorization Algorithm so that it can be used to solve a linear system, and then solve the following linear system: 2x1 x2 + x3 = 1 3x1 + 3x2 + 9x3 = 0 3x1 + 3x2 + 5x3 = 4. Chapter 3: Matrix Algebra: Iterative methods to solve System of linear equations: Distance between n dimensional vectors To discuss iterative methods for solving linear systems, we first need to determine a way to measure the distance between n dimensional column vectors. Chapter 3: Matrix Algebra: Norms Vector norms Let Rn denote the set of all n dimensional column vectors with real-number components. To define a distance in Rn we use the notion of a norm, which is the generalization of the absolute value on R, the set of real numbers. Chapter 3: Matrix Algebra: Norms Vector norms We will need only two specific norms on Rn , Chapter 3: Matrix Algebra: Norms Example: Determine the l2 norm and the l1 norm of the vector x = ( 1, 1, 2)t. Chapter 3: Matrix Algebra: Norms Distance between Vectors in Rn : If x = (x1 , x2 , · · · , xn )t and y = (y1 , y2 , · · · , yn )t are vectors in Rn , then l2 and l1 distances between x and y are defined by ( n )1/2 X 2 kx y k2 = (xi yi ) and kx y k1 = max |xi yi |. 1in i=1 Chapter 3: Matrix Algebra: Norms Convergence of a sequence in Rn : The sequence of vectors {x (k ) }1 n k =1 converges to x in R with (k ) respect to the l1 norm if and only if lim xi = xi , for each k !1 i = 1, 2, · · · , n. Chapter 3: Matrix Algebra: Convergence of a sequence in Rn Example: Show that t x (k ) = (x1 , x2 , x3 , x4 )t = 1, 2 + k1 , k32 , e k sin(k ) (k ) (k ) (k ) (k ) converges to x = (1, 2, 0, 0)t with respect to l1 norm. Chapter 3: Matrix Algebra: System of linear equations: Exercise: 1 Find l1 and l2 norms of the vectors. a) x = (3, 4, 0, 32 )t. b) x = (sin k , cos k , 2k )t for a fixed positive integer k. 2 Find the limit of the sequence ⇣ ⌘t 4 , k22 , k 2 e (k ) (k ) (k ) x (k ) = (x1 , x2 , x3 )t = k +1 k with respect to l1 norm. Chapter 3: Matrix Algebra: Lecture 26: Numerical Linear Algebra (UMA021): Matrix Algebra Dr. Meenu Rani Department of Mathematics TIET, Patiala Punjab-India Chapter 3: Matrix Algebra: System of linear equations LU Factorization: Chapter 3: Matrix Algebra: System of linear equations Example: Determine the LU factorization 2 for matrix 3 A in the 2 linear 3 system 1 1 0 3 1 62 1 1 17 6 7 Ax = b, where A = 6 7 , B = 6 1 7. Then 43 1 1 2 5 4 35 1 2 3 1 4 use the factorization to solve the system. Chapter 3: Matrix Algebra: System of linear equations Solution: Chapter 3: Matrix Algebra: System of linear equations: Exercise: 1 Modify the LU Factorization Algorithm so that it can be used to solve a linear system, and then solve the following linear system: 2x1 x2 + x3 = 1 3x1 + 3x2 + 9x3 = 0 3x1 + 3x2 + 5x3 = 4. Chapter 3: Matrix Algebra: Iterative methods to solve System of linear equations: Distance between n dimensional vectors To discuss iterative methods for solving linear systems, we first need to determine a way to measure the distance between n dimensional column vectors. Chapter 3: Matrix Algebra: Norms Vector norms Let Rn denote the set of all n dimensional column vectors with real-number components. To define a distance in Rn we use the notion of a norm, which is the generalization of the absolute value on R, the set of real numbers. Chapter 3: Matrix Algebra: Norms Vector norms We will need only two specific norms on Rn , Chapter 3: Matrix Algebra: Lecture 27: Numerical Linear Algebra (UMA021): Matrix Algebra Dr. Meenu Rani Department of Mathematics TIET, Patiala Punjab-India Chapter 3: Matrix Algebra: Norms Distance between Vectors in Rn : If x = (x1 , x2 , · · · , xn )t and y = (y1 , y2 , · · · , yn )t are vectors in Rn , then l2 and l1 distances between x and y are defined by ( n )1/2 X 2 kx y k2 = (xi yi ) and kx y k1 = max |xi yi |. 1in i=1 Chapter 3: Matrix Algebra: Norms Convergence of a sequence in Rn : The sequence of vectors {x (k ) }1 n k =1 converges to x in R with (k ) respect to the l1 norm if and only if lim xi = xi , for each k !1 i = 1, 2, · · · , n. Chapter 3: Matrix Algebra: Convergence of a sequence in Rn Example: Show that t x (k ) = (x1 , x2 , x3 , x4 )t = 1, 2 + k1 , k32 , e k sin(k ) (k ) (k ) (k ) (k ) converges to x = (1, 2, 0, 0)t with respect to l1 norm. Chapter 3: Matrix Algebra: System of linear equations: Exercise: 1 Find l1 and l2 norms of the vectors. a) x = (3, 4, 0, 32 )t. b) x = (sin k , cos k , 2k )t for a fixed positive integer k. 2 Find the limit of the sequence ⇣ ⌘t 4 , k22 , k 2 e (k ) (k ) (k ) x (k ) = (x1 , x2 , x3 )t = k +1 k with respect to l1 norm. Chapter 3: Matrix Algebra: Iterative methods to solve System of linear equations Jacobi Method Consider the system of linear equations: a11 x1 + a12 x2 + · · · + a1n xn = b1 a21 x1 + a22 x2 + · · · + a2n xn = b2.......... =.. an1 x1 + an2 x2 + · · · + ann xn = bn Chapter 3: Matrix Algebra: Iterative methods to solve System of linear equations Gauss Seidel Method Consider the system of linear equations: a11 x1 + a12 x2 + · · · + a1n xn = b1 a21 x1 + a22 x2 + · · · + a2n xn = b2.......... =.. an1 x1 + an2 x2 + · · · + ann xn = bn Chapter 3: Matrix Algebra: Iterative methods to solve System of linear equations Strictly diagonally dominant matrix: A square matrix A is called diagonally dominant if X |Aii | |Aij | 8 i. j6=i A is called strictly diagonally dominant if X |Aii | > |Aij | 8 i. j6=i Chapter 3: Matrix Algebra: Iterative methods to solve System of linear equations Example: Check whether the following 2 matrices 3 are strictly 2 diagonal 3 4 2 1 2 2 1 dominant or not: A = 4 1 6 35 and B = 4 1 3 25 2 2 5 1 2 0 Chapter 3: Matrix Algebra: Iterative methods to solve System of linear equations Result: If A is strictly diagonally dominant, then for any choice of x (0) , both the Jacobi and Gauss-Seidel methods give sequences k =0 that converge to the unique solution of Ax = b. {x (k ) }1 Chapter 3: Matrix Algebra: Iterative methods to solve System of linear equations Example: Use Jacobi and Gauss-Seidel iterative technique to find approximate solutions to 10x1 x2 + 2x3 = 6 x1 + 11x2 x3 + 3x4 = 25 2x1 x2 + 10x3 x4 = 11 3x2 x3 + 8x4 = 15 starting with x = (0, 0, 0, 0)t and iterating until kx (k ) x (k 1) k1 < 10 3. Chapter 3: Matrix Algebra: System of linear equations: Exercise: 1 The linear system x1 x3 = 0.2 1 1 x1 + x2 x3 = 1.425 2 4 1 x1 x2 + x3 = 2 2 has the solution (0.9, 0.8, 0.7)T. a Is the coefficient matrix strictly diagonally dominant? b Perform four iterations of the Jacobi and Gauss-Seidel iterative method to approximate the solution. Take x (0) = 0. Chapter 3: Matrix Algebra: Lecture 28: Numerical Linear Algebra (UMA021): Matrix Algebra Dr. Meenu Rani Department of Mathematics TIET, Patiala Punjab-India Chapter 3: Matrix Algebra: System of linear equations: Matrix representation of iterative methods Matrix norm: A matrix norm on a set of all n ⇥ n matrices is a real-valued function, k.k, defined on this set, satisfying for all n ⇥ n matrices A and B and all real numbers ↵: (i) kAk 0; (ii) kAk = 0, if and only if A is O, the matrix with all 0 entries; (iii) k↵Ak = |↵|kAk; (iv) kA + Bk  kAk + kBk; (v) kABk  kAk.kBk The distance between n ⇥ n matrices A and B with respect to this matrix norm is kA Bk. Chapter 3: Matrix Algebra: Matrix norm Matrix norm in l1 -space If A = (aij ) is n ⇥ n matrix, then the l1 norm is given by n X kAk1 = max |aij |. 1in j=1 Chapter 3: Matrix Algebra: Matrix norm Example: 2 3 1 2 1 Determine kAk1 4 norm for the matrix A = 0 3 15. 5 1 1 Solution: Chapter 3: Matrix Algebra: Matrix norm Eigenvalues and Eigenvectors: If A is a square matrix, the characteristic polynomial of A is given by |A I| = p( )(say). The zeros of p( ) are the eigenvalues for the matrix A. If is an eigenvalue of A and X 6= 0 satisfies (A I)X = 0, the X is an eigenvector corresponding to eigenvalue. Chapter 3: Matrix Algebra: Matrix norm Example: Determine 2 the eigenvalues 3 and eigenvectors of the matrix 2 0 0 A = 4 1 1 25. 1 1 4 Solution: Chapter 3: Matrix Algebra: Matrix norm Spectral radius: The spectral radius ⇢(A) of a matrix A is defined by ⇢(A) = max | |, where is an eigenvalue of A. Chapter 3: Matrix Algebra: Matrix norm Matrix norm in l2 -space If A = (aij ) is n ⇥ n matrix, then the l2 norm is given by q kAk2 = ⇢(At A) , where At is the transpose of A. Chapter 3: Matrix Algebra: Matrix norm Example: 2 3 1 1 0 Determine kAk2 norm for the matrix A = 4 1 2 15. 1 1 2 Solution: Chapter 3: Matrix Algebra: System of linear equations: Matrix representation of iterative methods Jacobi method: 0 1 n X (k ) 1 @ (k 1) A The Jacobi method is given by xi = bi aij xj aii j=1 Chapter 3: Matrix Algebra: System of linear equations: Matrix representation of iterative methods Gauss-Seidel method: The Gauss-Seidel 0 method is given by 1 iX1 n X (k ) 1 @ (k ) (k 1) A xi = bi aij xj aij xj aii j=1 j=i+1 Chapter 3: Matrix Algebra: Lecture 29: Numerical Linear Algebra (UMA021): Matrix Algebra Dr. Meenu Rani Department of Mathematics TIET, Patiala Punjab-India Chapter 3: Matrix Algebra: System of linear equations: Matrix representation of iterative methods Jacobi method: 0 1 n X (k ) 1 @ (k 1) A The Jacobi method is given by xi = bi aij xj aii j=1 Chapter 3: Matrix Algebra: System of linear equations: Matrix representation of iterative methods Gauss-Seidel method: The Gauss-Seidel 0 method is given by 1 iX1 n X (k ) 1 @ (k ) (k 1) A xi = bi aij xj aij xj aii j=1 j=i+1 Chapter 3: Matrix Algebra: System of linear equations: Matrix representation of iterative methods Result:(Stronger condition for the convergence of iterative methods): For any X (0) 2 Rn , the sequence {X (k ) }1 k =0 defined by X = TX (k ) (k 1) + C, for each k 1 converges to unique solution X = TX + C iff ⇢(T ) < 1. Chapter 3: Matrix Algebra: System of linear equations: Example: Check whether you can apply Gauss-Seidel iterative techniques to solve the following linear system of equations. 2x1 x2 + x3 = 1 2x1 + 2x2 + 2x3 = 4 x1 x2 + 2x3 = 5. Chapter 3: Matrix Algebra: System of linear equations: Exercise: Check whether you can apply Gauss-Seidel iterative techniques to solve the following linear systems. 1 2x1 + 3x2 + x3 = 1 3x1 + 2x2 + 2x3 = 1 x1 + 2x2 + 2x3 = 1 2 x1 + 2x2 2x3 = 7 x1 + x2 + x3 = 2 2x1 + 2x2 + x3 = 5 Chapter 3: Matrix Algebra: Iterative methods to find the solution of linear system: Question: Use the Gauss-Seidel method to approximate the solution of the following system: 4x1 + x2 x3 = 3 2x1 + 7x2 + x3 = 19 x1 3x2 + 12x3 = 31 Continue the iterations until two successive approximations are identical when rounded to three significant digits. Chapter 3: Matrix Algebra: Iterative methods to find the solution of linear system: Question: The linear system 2x1 x2 + x3 = 1 2x1 + 2x2 + 2x3 = 4 x1 x2 + 2x3 = 5 has the solution (1, 2, 1)t. Then show that ⇢(Tg ) = 12. Chapter 3: Matrix Algebra: Iterative methods to find the solution of linear system: Condition Number: The condition number of the non-singular matrix A relative to maximum norm k.k is 1 K (A) = kAk kA k. Chapter 3: Matrix Algebra: Iterative methods to find the solution of linear system: Condition Number: For any non-singular matrix A and infinity norm kAk 1 1 1 = k1k1 = kA.A k  kAk kA k = K (A). Note: A matrix A is well conditioned if K (A) is close to 1, and is ill conditioned when K (A) is significantly greater than 1. Chapter 3: Matrix Algebra: Iterative methods to find the solution of linear system: Example: Determine the condition number for the matrix:  1 2 1.0001 2 Chapter 3: Matrix Algebra: Iterative methods to find the solution of linear system: Exercise: Determine the condition number for the following matrices and check whether these matrices are ill-conditioned: 1.) 2 3 5 1 1 4 1 4 25 1 2 4 2.)  1 2. 0.9999 2 Chapter 3: Matrix Algebra: Least Square Approximation Method: Least Square Approximation Method: Suppose that the data points are (x0 , y0 ), (x1 , y1 ), (x2 , y2 ), · · · , (xn , yn ), where xi are the independent variable and yi are the dependent variable. Let ei = yi f (xi ) be the error at each data points. According to the method of least squares, the best fitting curve n X Xn 2 has the property that ei = (yi f (xi ))2 is minimum. i=0 i=0 Chapter 3: Matrix Algebra: Least Square Approximation Method: Least Square fit of a straight line: Suppose that the data points are (x0 , y0 ), (x1 , y1 ), (x2 , y2 ), · · · , (xn , yn ). Let f (x) = a + bx, where a, b are the constants to be determined to the given data. Now residuals is given by Xn Xn ei = yi f (xi ) = yi (a+bxi ) ) E = ei2 = (yi (a+bxi ))2. i=0 i=0 We need to find a and b such that error E is minimum. @E @E The necessary condition for minimum is = 0 and =0 @a @b Chapter 3: Matrix Algebra: Least Square Approximation Method: Example: Obtain the least square straight line fit to the following data: x 5 10 15 20 f (x) 16 19 23 26 Solution: Chapter 3: Matrix Algebra: Least Square Approximation Method: Exercise: 1 Use the method of least squares to fit the linear polynomial to the following data: x 2 1 0 1 2 f (x) 15 1 1 3 19