Mathematics for Data Science PDF Lecture Notes

Mathematics for Data Science Dr. Indu Joshi Assistant Professor at Indian Institute of Technology Mandi 29 July 2024  Motivation Vectors and matrices are fundamental in machine learning due to their ability of: Data Representatio...

Mathematics for Data Science Dr. Indu Joshi Assistant Professor at Indian Institute of Technology Mandi 29 July 2024  Motivation Vectors and matrices are fundamental in machine learning due to their ability of: Data Representation: Vectors and matrices efficiently represent and organize large datasets, enabling the handling of complex and high-dimensional data structures. Mathematical Operations: They facilitate various mathematical operations, such as dot products and matrix multiplications, which are essential for algorithms like linear regression and neural networks. Feature Transformation: Vectors and matrices are used to transform features, enabling dimensionality reduction techniques like Principal Component Analysis (PCA) and feature scaling.  Motivation (contd) Model Parameters: They store model parameters and weights, allowing for efficient computation and optimization in training processes, particularly in gradient descent and backpropagation. Linear Algebra Applications: Many machine learning algorithms rely on linear algebra concepts, which are naturally expressed using vectors and matrices, enhancing computational efficiency and algorithmic implementation.  Vector A vector is a mathematical object that encodes a length and direction. More formally these are elements of a vector space: a collection of objects that is closed under an addition rule and a rule for multiplication by scalars. A vector is often represented as a 1-dimensional array of numbers, referred to as components, and is displayed either in column form or row form. Represented geometrically, vectors typically represent coordinates within an n-dimensional space, where n is the number of dimensions. A simplistic representation of a vector might be an arrow in a vector space, with an origin, direction, and magnitude (length).  1.15 Example Consider the statement that 0 is an additive Vectors in RxnC 0 D x for all x 2 Fn : Is the 0 above the number 0 or the list 0? Solution Here 0 is a list, because we have not defined the sum A vector in Rn is an n (namely, of Fn-tuple v =x)(vand , v the ,..., v ), where number 0. vi ∈ R 1 2 n A picture can aid ou x1 , x 2  will draw pictures in R x can sketch this space on surfaces such as paper a A typical element of R2.x1 ; x2 /. Sometimes we Elements of R2 can be as a point but as an arrow thought of as points origin and ending at.x1 or as vectors. here. When we think o we refer to it as a vector When we think of v x arrows, we can move an x to itself (not changing  Vector Algebra Vector Addition: Two vectors, u and v of same dimension can be added as: u = [u1 , u2 ,...., un ], v = [v1 , v2 ,...., vn ] u + v = [u1 + v1 , u2 + v2 ,...., un + vn ] Example- In R3 , u = [1, 1, −1] and v = [2, 3, 1] u + v = [3, 4, 0] Vector Subtraction: Two vectors, u and v of same dimension can be subtracted as: u = [u1 , u2 ,...., un ], v = [v1 , v2 ,...., vn ] u − v = [u1 − v1 , u2 − v2 ,...., un − vn ] Example- In R3 , u = [1, 1, −1] and v = [2, 3, 1] u − v = [−1, −2, −2]  Vector Algebra Dot Product: Dot product of two vectors, u and v of same dimension can be computed as: u = [u1 , u2 ,...., un ], v = [v1 , v2 ,...., vn ] u · v = u1 v1 + u2 v2 +.... + un vn Example- In R3 , u = [1, 1, −1] and v = [2, 3, 1] u·v =2+3−1=4 Length/Magnitude of a Vector: Length of a vector v = [v1 , v2 ,...., vq n ] is defined as: √ |v | = v · v = v12 + v22 +.... + vn2 Example- √ √ √ v = [2, 3, 1], |v | = 22 + 32 + 12 = 4 + 9 + 1 = 14  Vector Algebra Angle between two vectors: The angle between two vectors u and v (θ) is defined as: θ = cos −1 ( |u||v u·v |)  Linear Combination of Vectors Consider a set S = {v1 , v2 ,..., vk }, then a vector v = α1 v1 + α2 v2 +... + αk vk is called a linear combination of v1 , v2 ,..., vk , where α1 , α2 ,...αk are scalar. Example- v1 = [1, 2, −1], v2 = [1, 1, 0], v3 = [0, 1, −1] α1 v1 + α2 v2 + α3 v3 = α1 [1, 2, −1] + α2 [1, 1, 0] + α3 [0, 1, −1] [(α1 + α2 ), (2α1 + α2 ), (−α1 − α3 )]  Linearly Independent and Dependent Vectors A set of vectors S = {v1 , v2 ,...., vn } is linearly independent if the equation- α1 v1 + α2 v2 +... + αn vn = 0 holds only when α1 = α2 =...αn = 0, otherwise the set S is linearly dependent. Example- S = {[1, 0], [1, 1]} in R2 α1 [1, 0] + α2 [1, 1] = [0, 0] =⇒ α1 + α2 = 0, α2 = 0 =⇒ α1 = 0. Therefore, S contains linearly independent vectors. Ŝ = {[1, 1], [3, 3]} (−3)[1, 1] + (1)[3, 3] = 0 =⇒ α1 = −3, α2 = 1. Therefore, Ŝ contains linearly dependent vectors.  Linearly Independent and Dependent Vectors R3 : S = {[1, −1, 0], [1, 0, 1], [0, 1, 1]} α1 = 1, α2 = −1, α3 = 1 α1 [1, −1, 0] + α2 [1, 0, 1] + α3 [0, 1, 1] = (1)[1, −1, 0] + (−1)[1, 0, 1] + (1)[0, 1, 1] = [0, 0, 0] v2 = v1 + v3 = [1, −1, 0] + [0, 1, 1] = [1, 0, 1] = v2 Example of linearly independent vectors in R3 : {[1, 0, 0], [0, 1, 0], [0, 1, 1]}  Linearly Independent and Dependent Vectors In Rn , a set of more than n vectors is linearly dependent. Any set containing zero vector is linearly dependent.  Orthogonal Vectors A set of vectors {v1 , v2 ,...., vn } are mutually (pairwise) orthogonal, if vi · vj = 0, for i ̸= j √ √ Example- R3 : {[1, 0, −1], [1, 2, 1], [1, − 2, 1]} √ [1, 0, −1] · [1, √ 2, 1] = 0 [1, 0, √ −1] · [1, − √2, 1] = 0 [1, 2, 1] · [1, − 2, 1] = 0 A set of orthogonal vectors is linearly independent.  Orthonormal Vectors A set of orthogonal vectors is orthonormal if each vector has length 1. −1 Example- {[ √1 , √1 ], [ √1 , √ 2 2 2 2 ]}  Matrix A matrix is a two-dimensional array of scalars with one or more columns and oneormore rows.  1   1 2 3  1 2 3 2 4 5 6 3 Usually we denote the elements of a matrix by {aij }m×n , where i is the index representing rows and varies from 1, 2,..., m, while j is the index representing columns and varies from 1, 2,..., n.  Diagonal and Triangular Matrix A square matrix whose all off-diagonal elements are zero is   1 0 0 called a diagonal matrix. Example- 0 2 0 0 0 3 A square matrix whose all elements below the main diagonal are  zero iscalled an upper triangular matrix. Example- 1 2 3 0 4 5 0 0 6 A square matrix whose all elements above the main diagonal are  zero iscalled a lower triangular matrix. Example- 1 0 0 2 3 0 4 0 5  Identity Matrix A diagonal matrix with all diagonal  entries as 1 is called as an 1 0 0 identity matrix. Example- 0 1 0 0 0 1  Matrix Algebra Two matrices are called equal if their dimensions  as well  as all the 1 0 0 corresponding elements are equal. Example- 0 1 0 0 0 1   1 0 0 0 1 0 The last element (a33 ) is different in the two matrices. 0 0 2 Therefore, these are not equal matrices.  Matrix Algebra The addition or subtraction of two Matrices X and Y of the same size returns a matrix Z of the same size zij = xij ± yij ∀i, j Matrices of different sizes cannot be added or subtracted. Commutativity: X ± Y = Y ± X Associativity: X ± (Y ± Z ) = (X ± Y ) ± Z = X ± Y ± Z  Matrix Algebra     1 0 3 1 0 0 X = 0 1 0 Y = 0 1 0 5 0 1 0 3 2     2 0 3 0 0 3 X + Y = 0 2 0 X − Y = 0 0 0 5 3 3 5 −3 −1  Scalar Multiplication   x11 x12... x1n  x21 x22... x2n  X = ... ......  xm1 xm2... xmn γ ∈ R or C   γx11 γx12... γx1n  γx21 γx22... γx2n  γX =  ... ......  γxm1 γxm2... γxmn     0 0 3 0 0 15 X = 0 0 0 , γ = 5, γX =  0 0 0 5 −3 1 25 −15 5  Matrix Multiplication The product of two matrices is a matrix. The necessary condition for multiplication of two matrices X and Y is that the number of columns in X must be equal to the number of rows in Y. Xm×n × Yn×p = Zm×p Yn×p × Xm×n – Multiplication Not Defined  Matrix Multiplication       x11 x12... x1n y11 y12... y1p z11 z12... z1p  x21 x22... x2n  y21 y22... y2p   z21 z22... z2p     =  .........  .........  .........  xm1 xm2... xmn yn1 yn2... ynp zm1 zm2... zmp z11 = x11 y11 + x12 y21 +... + x1n yn1 zij = [ith row of X ] · [jth column of Y ]  Matrix Multiplication     1 −1 1 −1 0 X = Y = 2 1 0 1 1  ×Y Z =X  1 −2 −1 Z= 2 −1 1 z11 = [1 − 1] · = 1 − 0 = 1 z12 = [1 − 1] · [−11] = −1 − 1 = −2  Transpose of a Matrix Xmn = {xij } i= 1, 2,...m, j=1, 2,...n T Xnm = {xji }     1 0 1 −1 0 T X23 = X32 = −1 1 0 1 1 0 1  Transpose of a Matrix: Properties 1. (A + B)T = AT + B T 2. (AB)T = B T AT 3. (kA)T = kAT 4. (AT )T = A  Determinant of a Matrix Every square matrix has a determinant. Determinant of a matrix is a number.   1 −1 X = 2 3 |X | = det(X ) = 3 − 2 = 1  Inverse of a Matrix Division of matrices is not defined since there may be AB = AC while B and C may not be equal. Instead matrix inversion is used. Inverse of a square matrix, X , if it exists, is the unique matrix X −1 such that: XX −1 = X −1 X = I  Inverse of a Matrix: Properties 1. (AB)−1 = B −1 A−1 2. (A−1 )−1 = A 3. (AT )−1 = (A−1 )T 4. (kA)−1 = k1 A−1 A square matrix that has an inverse is called a nonsingular matrix. A matrix that doesn’t have an inverse is called a singular matrix. Square matrices have inverses except when the determinant is zero. When the determinant of a matrix is zero, that matrix is singular.  Thank You Contact: [email protected]