Math Foundations for Deep Learning: Sets and Functions

Preliminary Concepts: Sets

• A set is a well-defined collection of objects, and elements or members belong to a set.
• Sets are denoted by capital letters, and elements are denoted by lowercase letters.
• Sets can be specified by stating the property that determines whether an object belongs to the set or by listing its elements inside a pair of braces.

Preliminary Concepts: Relations

• Set A is a subset of set B if every element of A is also an element of B.
• Set A is equal to set B if every element of B is in A.
• Empty sets have no elements and are denoted by ∅.

Vectors

• In mathematics and physics, a vector is a quantity with both direction and magnitude.
• In computer science, a vector is a one-dimensional ordered array of real-value numbers (scalars).
• Vectors are denoted by bold-font lowercase letters and can be written in column or row form.

Vector Operations: Addition

• The length of an array defines the dimension of a vector, which represents the number of axes required to represent the vector in a graph.

Calculus: Derivatives

• The derivative of a function at a point is the slope of the tangent drawn to that curve at that point.
• The derivative is represented by 𝑓 ′ 𝑥 and is defined as the limit of the ratio of the change in the function to the change in x.

Calculus: Multivariate Functions

• Multivariate functions are functions of several variables, and partial derivatives are used to find the derivative with respect to one of those variables, holding the others constant.

Calculus: Derivative of a Vector/Matrix

• Partial derivatives are used in vector calculus and differential geometry.
• The derivative of a vector/matrix is an extension of ordinary scalar derivative to higher-dimensional settings.

Calculus: Gradient

• The gradient is a vector that points in the direction of the maximum rate of change of a function.
• The dot product is used to calculate the gradient.

Vector Calculus: Angle between Two Vectors

• The angle between two vectors can be calculated using the law of cosines, which is generalized to higher-dimensional spaces.
• The cosine of the angle between two vectors is equal to the dot product of the vectors divided by the product of their magnitudes.

Vector Orthogonality

• Two vectors are orthogonal if their inner product is zero.
• The notation for a pair of orthogonal vectors is 𝒖 ⊥ 𝒗.

Matrices

• A matrix is a rectangular array of numbers.
• Matrices are used to represent systems of equations and linear transformations.