Math for Machine Learning Linear Algebra - Week 1 PDF

Copyright Notice These slides are distributed under the Creative Commons License. DeepLearning.ai makes these slides available for educational purposes. You may not use or distribute these slides for commercial purposes. You may make copies of these slides and use or distribute them for educationa...

Copyright Notice These slides are distributed under the Creative Commons License. DeepLearning.ai makes these slides available for educational purposes. You may not use or distribute these slides for commercial purposes. You may make copies of these slides and use or distribute them for educational purposes as long as you cite DeepLearning.AI as the source of the slides. For the rest of the details of the license, see https://creativecommons.org/licenses/by-sa/2.0/legalcode Math for Machine Learning Linear algebra - Week 1 System of Linear Equations Linear Algebra Applied I Machine Learning Machine Learning Don’t worry about the math! Don’t worry about the machine learning! Linear Algebra and Machine Learning Linear Regression Supervised Machine Learning Input Input Input Output Linear Algebra and Machine Learning Input Output wind speed power output Linear Algebra and Machine Learning power output (kW) 3500 Input Output 3000 2500 2000 1500 1000 500 wind speed power output 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 wind speed (m/s) Linear Algebra and Machine Learning power output (kW) 3500 Input Output 3000 2500 m× wind speed +b = power output y =wmx + b 2000 5m/s 1500kW 1500 1000 500 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 wind speed (m/s) Linear Algebra and Machine Learning Input Input Output wind speed temperature power output Linear Algebra and Machine Learning Input Input Output y = w1x1 + w2 x2 + b wind speed temperature power output Linear Algebra and Machine Learning temperature power output (kW) Input Input Output power output (kW) wind speed (m/s) wind speed temperature power output wind speed (m/s) Linear Algebra and Machine Learning feature 1 wind speed feature 2 temperature feature 3 pressure feature 4 humidity Other features Output  Linear Algebra and Machine Learning w1 feature 1 + w2 feature 2 + w3 feature 3 + w4 feature 4 + b = Output TARGET  Linear Algebra and Machine Learning w1 feature 1 + w2 featurex1 w1 2 + +w3 w2 xfeature 2 +3 …+ w+4 xn + b wn feature 4 + = y = Output TARGET Linear Algebra and Machine Learning w1 (1) x1 + w2 (1) x2 + … + wn (1) xn + b = y (1) w1 x1 (2) + w2 x2 (2) + … + wn xn (2) + b = y (2) w1 (3) x1 + w2 (3) x2 + … + wn (3) xn + b = y (3) System of Linear Equations... w1 x1 (m) + w2 x2 (m) + … + wn xn + b (m) = y (m) System of Linear Equations Linear Algebra Applied II Linear Algebra and Machine Learning power output (kW) 3500 Input Output 3000 2500 2000 1500 1000 500 wind speed electrical power output 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 wind speed (m/s) Linear Algebra and Machine Learning wind speed temperatur pressure humidity … feature n target x1 x2 x3 x4 xn Linear Algebra and Machine Learning wind speed temperatur pressure humidity … feature n target (1) (1) (1) w1 x1 + (1) w2 x2 + w3 x3 + (1) w4 x4 +... + wn xn + b = y (1) (2) (2) (2) (2) (2) w1 x1 + w2 x2 + w3 x3 + w4 x4 +... + wn xn + b = y (2)... (m) w1 x1 + w2 x (m) w3 x (m) + w4 x4 (m) +... + (m) wn xn + b y (m) 2 + 3 = Linear Algebra and Machine Learning wind speed temperatur pressure humidity … feature n target (1) (1) (1) w1 x1 + (1) w2 x2 + w3 x3 + (1) w4 x4 +... + wn xn + b = y (1) (2) (2) (2) (2) (2) w1 x1 + w2 x2 + w3 x3 + w4 x4 +... + wn xn + b = y (2)... (m) w1 x1 + w2 x (m) w3 x (m) + w4 x4 (m) +... + (m) wn xn + b y (m) 2 + 3 = Linear Algebra and Machine Learning w w1 w2 w3 w4 … wn ⋅ (1) x1 (1) x2 X (1) x3 (1) … x4 (1) xn + b y = (1) y (2) Y … y (m) (2) (2) (2) (2) … vector x1 x2 x3 x4 (2) xn vector... (m) (m) (m) (m) … x1 x2 x3 x4 (m) xn matrix Linear Algebra and Machine Learning wind speed temperatur pressure humidity … feature n target (1) (1) (1) (1) (1) w1 x1 + w2 x2 + w3 x3 + w4 x4 +... + wn xn + b = y (1) (2) (2) (2) (2) (2) w1 x1 + w2 x2 + w3 x3 + w4 x4 +... + wn xn + b = y (2)... (m) w1 x1 + w2 x (m) w3 x (m) + w4 x4 (m) +... + (m) wn xn + b y (m) 2 + 3 = System of Linear Equations Plan for the Week Common vector and matrix operations Questions Q. Q. Q. END OF WEEK Q. Plan for the Week Systems of Linear Equations Representing systems as vectors and matrices Computing the determinant of matrices Check your Knowledge Linear Your algebra score added to your calculus Algebra score minus your probability score was 6 Your algebra score minus your calculus Calculus score plus double your probability score was 4. Four times your algebra score minus double Probability & your calculus score added to your probability Statistics score was 10 Represent these statements as a system of linear equations. Check your Knowledge a Linear Algebra Your algebra score added to your calculus score minus your probability score was 6 a+c-p=6 Your algebra score minus your calculus c Calculus score plus double your probability score a - c + 2p = 4 was 4. Four times your algebra score minus double p Probability & Statistics your calculus score added to your probability 4a - 2c + p = 10 score was 10 Represent these statements as a system of linear equations. Check your Knowledge 1a a ++ c1c- -p1p = 6= 6 What are the weights, w? a, c, p What are the features, x? 1a a --c1c + +2p2p= =4 4 The targets, y? 6, 4, 10 4a 4a - 2c + + 1p p ==10 10 Check your Knowledge Is this system singular or non-singular? a+c-p=6 Can you solve this system of equations? a - c + 2p = 4 Can you represent this system as a matrix and a vector? Can you calculate the determinant of that 4a - 2c + p = 10 matrix? What to expect You’re here 1 System of sentences 2 System of linear equations 3 Visualising systems of linear equations 4 Singular and Non-Singular matrices 5 Determinant of matrices System of Linear Equations System of sentences Systems of sentences System 1 System 2 System 3 The dog is black The dog is black The dog is black The cat is orange The dog is black The dog is white Complete Redundant Contradictory Non-singular Singular Singular Systems of sentences System 1 System 2 System 3 System 4 The dog is black The dog is black The dog is black The dog is black The cat is orange The dog is black The dog is black The dog is white The bird is red The bird is red The dog is black The bird is red Complete Redundant Redundant Contradictory Non-singular Singular Singular Singular Quiz: Systems of sentences Given this system: Between the dog, the cat, and the bird, one is red. Between the dog and the cat, one is orange. The dog is black. Problem 1: What color is the bird? Problem 2: Is this system singular or non-singular? Solution: Systems of information Given this system: Between the dog, the cat, and the bird, one is red. Between the dog and the cat, one is orange. The dog is black. Solution 1: The bird is red. Solution 2: It is non-singular. System of Linear Equations System of equations Sentences → Equations Sentences Sentences with numbers Equations Between the dog and The price of an apple a + b = 10 the cat, one is black. and a banana is $10. Quiz: Systems of equations 1 You go two days in a row and collect this information: Day 1: You bought an apple and a banana and they cost $10. Day 2: You bought an apple and two bananas and they cost $12. Question: How much does each fruit cost? Solution: Systems of equations 1 Day 1: You bought an apple and a banana and they cost $10. + = $10 $8 $2 Day 2: You bought an apple and two bananas and they cost $12. + + = $12 $2 Solution: An apple costs $8, a banana costs $2. Quiz: Systems of equations 2 Problem 1: You’re trying to figure out the price of apples, bananas, and cherries at the store. You go three days in a row, and bring this information. Day 1: You bought an apple, a banana, and a cherry, and paid $10. Day 2: You bought an apple, two bananas, and a cherry, and paid $15. Day 3: You bought an apple, a banana, and two cherries, and paid $12. How much does each fruit cost? Solution: Systems of equations 2 System of equations 1 $5 $2 a + b + c = 10 $10 $3 a + 2b + c = 15 a + b + 2c = 12 $15 $5 Solution $12 $2 a=3 b=5 c=2 Quiz: Systems of equations 3 You go two days in a row and collect this information: Day 1: You bought an apple and a banana and they cost $10. Day 2: You bought two apples and two bananas and they cost $20. Question: How much does each fruit cost? Solution: Systems of equations 3 Day 1: You bought an apple and a banana and they cost $10. + = $10 Day 2: You bought two apples and two bananas and they cost $20. = $20 Same thing!!! + 8 2 5 5 Infinitely many solutions! 8.3 1.7 0 10 Quiz: Systems of equations 4 You go two days in a row and collect this information: Day 1: You bought an apple and a banana and they cost $10. Day 2: You bought two apples and two bananas and they cost $24. Question: How much does each fruit cost? Solution: Systems of equations 4 Day 1: You bought an apple and a banana and they cost $10. + = $10 + = $20 Day 2: You bought two apples and two bananas and they cost $24. + = $24 Contradiction! No solutions! Systems of equations System 1 System 2 System 3 a + b = 10 a + b = 10 a + b = 10 a + 2b = 12 2a + 2b = 20 2a + 2b = 24 Unique solution: Infinite solutions No solution a=8 a=8 , 7 , 6 … b=2 b=2 3 4 Complete Redundant Contradictory Non-singular Singular Singular Quiz: More systems of equations System 1 System 2 System 3 a + b + c = 10 a + b + c = 10 a + b + c = 10 a + b + 2c = 15 a + b + 2c = 15 2a + 2b + 2c = 20 a + b + 3c = 20 a + b + 3c = 18 3a + 3b + 3c = 30 Solutions: More systems of equations System 2 System 3 System 4 a + b + c = 10 a + b + c = 10 a + b + c = 10 a + b + 2c = 15 a + b + 2c = 15 2a + 2b + 2c = 20 a + b + 3c = 20 a + b + 3c = 18 3a + 3b + 3c = 30 Infinitely many sols. No solutions Infinitely many solutions c=5 From 1st and 2nd: Any 3 numbers that add c=5 to 10 work. a+b=5 From 2nd and 3rd: (0,0,10), (2,7,1), … c=3 (0,5,5), (1,4,5), (2,3,5), … What is a linear equation? Linear Non-linear a + b = 10 a 2 + b 2 = 10 2a + 3b = 15 sin(a) + b 5 = 15 3.4a − 48.99b + 2c = 122.5 2a − 3b = 0 b 3 ab 2 + − − log(c) = 4a a b Numbers  System of Linear Equations System of equations as lines and planes Linear equation → line b (-4,14) b (0,10) slope = -1 y-intercept = 10 (-4,8) slope = -0.5 (0,6) (4,6) a + 2b = 12 a + b = 10 (8,2) (8,2) y-intercept = 6 (12,0) (10,0) a a (12,-2) Linear equation → line b b a + 2b = 12 a + b = 10 a a Linear equation → line b Unique solution! a + b = 10 a + 2b = 12 (8,2) a Linear equation → line b b (0,10) (0,10) 2a + 2b = 20 a + b = 10 (10,0) a (10,0) a Linear equation → line b b (0,10) (0,10) 2a + 2b = 20 a + b = 10 (10,0) a (10,0) a Linear equation → line b Every point in the line Is a solution! a + b = 10 2a + 2b = 20 a Linear equation → line b b (0,12) (0,10) 2a + 2b = 24 a + b = 10 (10,0) a a (12,0) Linear equation → line b b (0,12) (0,10) 2a + 2b = 24 a + b = 10 (10,0) a a (12,0) Linear equation → line b No solutions a + b = 10 2a + 2b = 24 a Systems of equations as lines System 1 System 2 System 3 a + b = 10 a + b = 10 a + b = 10 a + 2b = 12 2a + 2b = 20 2a + 2b = 24 b Unique b Infinite b solution No solutions solutions (8,2) a a a Complete Redundant Contradictory Non-singular Singular Singular Quiz Problem 1 Which of the following plots corresponds to the system of equations: 3a + 2b = 8 2a - b = 3 a) b) c) d) Problem 2 Is this system singular or non-singular? Solution Problem 1 b Problem 2 a) Since the lines cross at a (0,4) 2a - b = 3 unique point, the system is non-singular. (2,1) a (3/2,0) (8/3,0) (0,-3) 3a + 2b = 8 Linear equation in 3 variables asb a plane a+b+c=1 c 1+0+0=1 (0,1,0) 0+1+0=1 (1,0,0) a 0+0+1=1 (0,0,1) Linear equation in 3 variables asb a plane 3a - 5b + 2c = 0 c 3(0) + 5(0) + 2(0) = 0 (0,0,0) a  System 1 b System 1 a+b+c=0 c a + 2b + c = 0 a + b + 2c = 0 (0,0,0) a  System 2 b System 2 a+b+c=0 c a + b + 2c = 0 a + b + 3c = 0 a  System 3 b System 3 a+b+c=0 c 2a + 2b + 2c = 0 3a + 3b + 3c = 0 a b System of Linear Equations A geometric notion of singularity Systems of equations as lines System 1 System 2 System 3 a + b = 010 a + b = 010 a+ b= 0 10 a + 2b = 012 2a + 2b = 020 2a + 2b = 0 24 b b b Unique solution Infinite No solutions solutions (8,2) a a a Complete Redundant Contradictory Non-singular Singular Singular Systems of equations as lines System 1 System 2 System 3 a+ b= 0 a+ b= 0 a+ b= 0 a + 2b = 0 2a + 2b = 0 2a + 2b = 0 b b b Infinite Infinite Unique solutions solution solutions a a a Complete Redundant Redundant Non-singular Singular Singular System of Linear Equations Singular vs non-singular matrices Systems of equations as matrices System 1 System 2 a + b = 10 0 a + b = 10 0 1 1 1 1 a + 2b = 12 0 2a + 2b = 20 0 1 2 2 2 Non-singular Non-singular Singular Singular system matrix system matrix (Unique solution) (Infinitely many solutions) Constants don’t matter for singularity System 1 System 2 System 3 System 4 a + b + c = 10 a + b + c = 10 a + b + c = 10 a + b + c = 10 a + 2b + c = 15 a + b + 2c = 15 a + b + 2c = 15 2a + 2b + 2c = 15 a + b + 2c = 12 a + b + 3c = 20 a + b + 3c = 18 3a + 3b + 3c = 20 Unique solution Infinite solutions No solutions Infinite solutions Complete Redundant Contradictory Redundant Non-singular Singular Singular Singular Constants don’t matter for singularity System 1 System 2 System 3 System 4 a + b + c = 10 a + b + c = 10 a + b + c = 10 a + b + c = 10 a + 2b + c = 15 a + b + 2c = 15 a + b + 2c = 15 2a + 2b + 2c = 20 a + b + 2c = 12 a + b + 3c = 20 a + b + 3c = 18 3a + 3b + 3c = 30 a+b+c=0 a+b+c=0 a+b+c=0 a+b+c=0 a + 2b + c = 0 a + b + 2c = 0 a + b + 2c = 0 2a + 2b + 2c = 0 a + b + 2c = 0 a + b + 3c = 0 a + b + 3c = 0 3a + 3b + 3c = 0 Constants don’t matter for singularity System 1 System 2 System 3 System 4 a+b+c=0 a+b+c=0 a+b+c=0 a+b+c=0 a + 2b + c = 0 a + b + 2c = 0 a + b + 2c = 0 2a + 2b + 2c = 0 a + b + 2c = 0 a + b + 3c = 0 a + b + 3c = 0 3a + 3b + 3c = 0 Constants don’t matter for singularity System 1 System 2 System 3 System 4 a+b+c=0 a+b+c=0 a+b+c=0 a+b+c=0 a + 2b + c = 0 a + b + 2c = 0 a + b + 2c = 0 2a + 2b + 2c = 0 a + b + 2c = 0 a + b + 3c = 0 a + b + 3c = 0 3a + 3b + 3c = 0 1 1 1 1 1 1 1 1 1 1 2 1 1 1 2 2 2 2 1 1 2 1 1 3 3 3 3 Non-singular Singular Singular System of Linear Equations Linear dependence and independence Linear dependence between rows Non-singular Singular system a + b = 10 0 a + b = 10 0 1 1 1 1 a + 2b = 12 0 2a + 2b = 20 0 1 2 2 2 No equation is a No row is a multiple Second equation is Second row is a multiple of the other of the other one a multiple of the multiple of the first one first one row Rows are Rows are linearly independent linearly dependent Linear dependence and independence a + 0b + 0c = 1 a=1 b=2 + 0a + b + 0c = 2 a+b=3 a + b + 0c = 3 1 0 0 Row 1 + Row 2 = Row 3 0 1 0 Row 3 depends on rows 1 and 2 1 1 0 Rows are linearly dependent Linear dependence and independence a+b+c=0 a+b+c=0 2a + 2b + 2c = 0 + 2a + 2b + 2c = 0 3a + 3b + 3c = 0 3a + 3b + 3c = 0 1 1 1 Row 1 + Row 2 = Row 3 2 2 2 Row 3 depends on rows 1 and 2 3 3 3 Rows are linearly dependent Linear dependence and independence a+b+c=0 a+b+c=0 a + b + 2c = 0 + a + b + 3c = 0 a + b + 3c = 0 2a + 2b + 4c = 0 ÷2 1 1 1 a + b + 2c = 0 1 1 2 Average of Row 1 and Row 3 is Row 2 1 1 3 Row 2 depends on rows 1 and 3 Rows are linearly dependent Linear dependence and independence a+b+c=0 a + 2b + c = 0 No relations between equations a + b + 2c = 0 1 1 1 No relations between rows 1 2 1 Rows are linearly independent 1 1 2  Quiz: Linear dependence and independence Problem: Determine if the following matrices have linearly dependent or independent rows 1 0 1 1 1 1 1 1 1 1 2 5 0 1 0 1 1 2 0 2 2 0 3 -2 3 2 3 0 0 -1 0 0 3 2 4 10  Solution: Linear dependence and independence Problem: Determine if the following matrices have linear dependent or independent rows 1 0 1 1 1 1 1 1 1 1 2 5 0 1 0 1 1 2 0 2 2 0 3 -2 3 2 3 0 0 -1 0 0 3 2 4 10 3Row1 + 2Row2 = Row3 Row1 - Row2 = Row3 No relations 2Row1 = Row3 Dependent (singular) Dependent (singular) Independent Dependent (singular) (Non-singular) System of Linear Equations The determinant Linear dependence between rows Non-singular matrix Singular matrix 1 1 1 1 1 2 2 2 1 1 x? = 1 2 1 1 x2 = 2 2 Rows linearly independent Rows linearly dependent Determinant ak = c bk = d a b Determinant = ad − bc c d = =k c d a - b a b d c Matrix is singular if ad = bc a b *k = c d Determinant ad − bc = 0 Determinant Non-singular matrix Singular matrix 1 1 1 1 1 2 2 2 Determinant Determinant 1 - 1 1 - 1 2 1 2 2 1⋅2−1⋅1=1 1⋅2−2⋅1=0 Determinant and singularity a b ad − bc c d Matrix is singular Determinant is zero Quiz: Determinant Problem 1: Find the determinant of the following matrices Matrix 1 5 1 -1 3 Matrix 2 2 -1 -6 3 Problem 2: Are these matrices singular or non-singular? Solutions: Determinant Matrix 1: det = 5 ⋅ 3 − 1 ⋅ (−1) = 15 + 1 = 16 5 1 Non-singular -1 3 Matrix 2: det = 2 ⋅ 3 − (−1) ⋅ (−6) = 6 − 6 = 0 2 -1 Singular -6 3 Diagonals in a 3x3 matrix Determinant Add Subtract The determinant 1 1 1 1 2 1 1 1 2 The determinant 1 1 1 1 1 2 1 2 1 1 2 2 + 1⋅2⋅2 The determinant 1 1 1 1 1 1 2 1 2 1 1 1 2 2 1 + 1⋅2⋅2 + 1⋅1⋅1 The determinant 1 1 1 1 1 1 1 2 1 2 1 1 1 1 2 2 1 1 + 1⋅2⋅2 + 1⋅1⋅1 + 1⋅1⋅1 The determinant 1 1 1 1 1 1 1 2 1 2 1 1 1 1 2 2 1 1 + 1⋅2⋅2 + 1⋅1⋅1 + 1⋅1⋅1 1 2 1 − 1⋅2⋅1 The determinant 1 1 1 1 1 1 1 2 1 2 1 1 1 1 2 2 1 1 + 1⋅2⋅2 + 1⋅1⋅1 + 1⋅1⋅1 1 1 2 1 1 1 − 1⋅2⋅1 − 1⋅1⋅1 The determinant 1 1 1 1 1 1 1 2 1 2 1 1 2 1 1 1 1 2 4 1 1 + 1⋅2⋅2 + 1⋅1⋅1 + 1⋅1⋅1 Det = 4+1+1 -2-1-2 1 1 1 =1 2 1 1 1 2 1 1 2 2 − 1⋅2⋅1 − 1⋅1⋅1 − 1⋅1⋅2  Quiz: Determinants Problem: Find the determinant of the following matrices (from the previous quiz). Verify that those with determinant 0 are precisely the singular matrices. 1 0 1 1 1 1 1 1 1 1 2 5 0 1 0 1 1 2 0 2 2 0 3 -2 3 3 3 0 0 -1 0 0 3 2 4 10  Solution: Determinants Problem: Find the determinant of the following matrices (from the previous quiz). Verify that those with determinant 0 are precisely the singular matrices. 1 0 1 1 1 1 1 1 1 1 2 5 0 1 0 1 1 2 0 2 2 0 3 -2 3 3 3 0 0 -1 0 0 3 2 4 10 Determinant = 0 Determinant = 0 Determinant = 6 Determinant = 0 Singular Singular Non-singular Singular The determinant 1 1 1 1 1 1 0 2 2 2 2 0 0 0 3 3 0 0 + 1⋅2⋅3 + 1⋅2⋅0 + 1⋅0⋅0 Det = 6+0+0-0-0-0 =6 1 1 1 2 2 0 0 0 3 − 1⋅2⋅0 − 1⋅2⋅0 − 1⋅0⋅3 The determinant 1 1 1 1 1 1 0 2 2 2 2 0 0 0 0 0 0 0 + 1⋅2⋅0 + 1⋅2⋅0 + 1⋅0⋅0 Det = 0+0+0-0-0-0 =0 1 1 1 2 2 0 0 0 0 − 1⋅2⋅0 − 1⋅2⋅0 − 1⋅0⋅0 System of Linear Equations Conclusion