Untitled Quiz

Support Vector Machines (SVM)

• SVMs are a supervised learning algorithm, originally designed for classification, but can be extended to regression.
• Developed by Vladimir Vapnik and Alexey Chervonenkis in 1963.
• Further developed by Dr. Vapnik at Bell Labs in the 1990s.

SVM Types

• Hard Margin SVM: Used for perfectly separable data. Tries to maximize the margin (distance between the hyperplane and the closest data points of both classes).
• Soft Margin SVM: Extends hard margin to non-perfectly separable data. Includes a penalty term (C) to allow some data points to be on the wrong side of the hyperplane.

Preliminaries

• Space: A set with a defined structure, like the 2-dimensional plane.
• Subspace: A subset of a space, for example, the positive quadrant.
• Hyperplane: A subspace of one dimension less than the containing space. In 2D, it's a line; in 3D, it's a plane.

Logistic Function

• Predicts the probability of an outcome (0 or 1) using the equation: ŷ1 = 1 / (1 + e^-(βTx)).
• Assumes a larger value of βTx yields higher confidence in a prediction.
• βTx ≥ 0 predicts y₁ = 1 with high confidence; βTx < 0 predicts y₁ = 0 with high confidence.

Optimal Separating Hyperplane

• The goal of SVM is to find the optimal separating hyperplane.
• The goal is to maximize the margin, which is the distance from the hyperplane to the nearest data point from either class.

Primal Problem

• The primal problem formulates SVM as a quadratic programming problem that seeks to minimize ||w||² subject to constraints that ensure correct classification and maximize the margin, where ||w|| is the Euclidean norm.
• The constraint ensures that the correct class (y) is on the correct side of a linear transformation w ⋅ x + b ≥ 1.

Kernel Trick

• Kernels are used to map data into higher dimensions where non-linear separation may become possible.
• Used in non-linear SVMs.
• Kernels compute similarities between observations.
• Popular kernels include:
  • Linear: K(xᵢ, xⱼ) = xᵢ ⋅ xⱼ + c
  • Polynomial: K(xᵢ, xⱼ) = (xᵢ ⋅ xⱼ + c)^d.
  • Radial Basis Function (RBF): K(xᵢ, xⱼ) = e^{-||xᵢ - xⱼ||2/γ}
  • Sigmoid: K(xᵢ, xⱼ) = tanh(γxᵢ ⋅ xⱼ + c)

Dual Problem

• The dual problem is an equivalent reformulation of the primal problem and often is more efficient to solve.
• It simplifies computation by only focusing on support vectors, those observations that lie on the margin (boundary).

Hyperparameters

• Kernel: Crucial for determining the separating hyperplane's shape.
• C (penalty): Controls the penalty for misclassifying data points.