Kernel Methods and Support Vector Machines Quiz

Supervised Learning

• Supervised learning uses labelled data
• The goal is to find a mapping between inputs and outputs
• The oracle function maps inputs to outputs
• A loss function measures the approximation closeness
• Risk minimization finds the best predictive model
• The challenge is generalisation for unseen data
• The process involves defining the hypothesis space, optimisation, and generalisation.

Classification Example

• The goal is to map images to labels
• Training process is used to find the model
• The model maps inputs to outputs.

Linear Models and Concepts

• The lecture touches on topics such as linear models, distance, norms, linear regression, basis functions, matrix solution, and residual.
• The authors referenced include Ronald Aarts, Bojana Rosić, and Qianxiao Li.
• Supervised learning covers classification and regression, where prediction is based on labeled data, and generalisation is key for accurate results.

Linear Regression

• Least squares fit method minimizes squared error.
• Hypothesis space is the space of linear functions
• Euclidean norm used for the loss function.
• Basis functions can be used to expand linear regression for more complex models.
• The solution determines the parameter estimate involving derivatives of loss function.
• Linear regression includes fitting a linear function to data (underfitting) and fitting high-order polynomials (overfitting).
• Residual plots can evaluate fit quality. Random/small residuals indicate good fit; structured residuals indicate a need for a more complex model.

Linearity in Parameters and 2-Norm

• Allows straightforward mathematical analysis using linear algebra.
• Provides a single analytical solution.
• No explicit analytical solution may exist if relationships aren't linear or alternative norms are used.

Linear Regression with Basis Functions and Regularization

• Linear regression involves a hypothesis space where parameters are linearly related to basis functions or feature maps
• The goal is to minimize Euclidean or 2-norm loss
• Solution includes Moore-Penrose pseudoinverse calculation
• Regularization addresses multiple solutions by adding a regularization term to the cost function.
• Example of regularization is k^2 regularization (ridge regression).

Nonlinear Regression and Optimization

• Matrix formalism doesn't apply, needing iterative solutions (e.g., gradient descent).
• Gradient descent updates based on the cost function's local gradient.
• Nonlinear optimization is common, with stochastic gradient descent methods like Adam being used.

Applying Machine Learning with TensorFlow

• The process involves selecting a hypothesis space, optimization, checking generalisation, and splitting data into training, validation, and test sets.
• TensorFlow facilitates implementation, focusing on data import, preprocessing, data splitting, scaling, model definition, compiling, training, evaluation, and prediction.

Classification

• Outputs are discrete labels
• Binary classification uses a 'hard' or 'smooth' transition activation function (e.g., tanh).
• Multi-class classification uses one-hot encoding to represent labels.
• Hypothesis space is multi-dimensional; oracle function maps input to a hypercube vertex.
• Activation function, such as softmax, could select the maximum output.

Kernel Ridge Regression

• Minimises empirical risk (including regularization)
• The solution involves the inverse (Moore-Penrose pseudoinverse) of a matrix calculation.
• Kernel function defines predictions without an explicit feature map.
• Examples of kernel types include linear, polynomial, and Gaussian.

Gaussian/RBF Kernel and Implementation

• The kernel function is defined as an exponential function.
• Kernel ridge regression is a method which uses a hypothesis space.
• The solution can be found without using explicit feature maps.
• Kernel function implementation involves specifying kernel types and hyperparameters.

Support Vector Machines

• Used for linear and nonlinear classification, regression, and outlier detection.
• Goal is to identify the optimal hyperplane separating data points.
• Maximises separation margin (the distance between hyperplane and closest data points).
• Uses support vectors (the data points closest to the hyperplane).
• Soft margins allow for misclassifications, introduces slack variables (penalisation for violating the margins).
• Kernel trick allows for nonlinearly separable data.

Hinge Loss Function

• Used for soft margins.
• Value depends on the correct classification and margin.

Introduction to Support Vector Machines using Kernels

• Kernels are non-linear functions that transform data into higher dimensional spaces.
• Kernel trick computes similarities between points in higher dimensions without explicitly calculating coordinates.
• This allows for handling nonlinear data.

Types of Kernels

• Linear Kernel (no transformation required for linearly separable data)
• Polynomial Kernel (for more complex boundaries but more computationally expensive)
• Radial Basis Function (RBF) Kernel (highly flexible but computation intensive).

Feature Scaling

• Feature scaling is crucial for effective distance/similarity calculations.
• Methods include normalization (values into 0-1 range) and standardization (centering and scaling to unit variance).

SVM Classification and Classification Regression

• SVM classification predicts the class of a new data point.
• Can be used for binary or multi-class.
• SVM Regression predicts a function to represent the data points with maximum deviation..
• Soft classification/regression allows for misclassified data by using slack variables.

Decision Trees

• A flowchart-like tree-structure approach for classification and regression.
• Nodes represent decisions (splits), while leaf nodes hold predictions, branches connecting nodes.
• Techniques include pre-pruning and post-pruning to avoid overfitting
• Measures of impurity (e.g., entropy, gini index) are employed to guide splitting.
• Decision trees have advantages like simplicity and interpretability and can handle non-linear relationships but may be unstable.

Ensemble Learning

• Combines multiple models for prediction.
• Improves prediction stability and reduction in error.
• Methods include bagging (with replacement), pasting (without replacement).
• Random forest is a popular ensemble method with random subsets of features at each node.

Boosting

• Adjusts weights for misclassified instances to sequentially improve model performance.
• AdaBoost is a popular but non-parallelizable boosting method.
• Gradient boosting iteratively builds models, addressing misclassifications by previous models.

Artificial Neural Networks

• Simulates biological neurons, with nodes (neurons), connections (synapses).
• A perceptron is a single-layered neural network, a linear model.
• MLPs (multilayer perceptrons) have multiple hidden layers and represent complex relationships.
• Backpropagation uses gradient descent to adjust weights.
• Activation functions introduce non-linearity.

Learning Rate and Optimizers

• Learning rate controls the step size of parameter updates in training.
• Gradient descent, stochastic gradient descent, Adam, optimize the loss function.

Feature Selection and PCA

• Reducing dataset dimensionality with appropriate feature selection.
• Principal Component Analysis (PCA) identifies principal directions that account for the maximum variance
• Linear transformation projects data into a lower dimensional space.
• PCA related to Singular Value Decomposition(SVD) and Moore-Penrose inverse.

Unsupervised Learning

• Techniques used for finding hidden patterns and data groupings without prior knowledge of the data.
• Clustering, k-means, DBSCAN are examples used to cluster unlabeled data.
• GMM(Gaussian Mixture Model) estimates probability of each instance belonging to a cluster, rather than hard assigning to single class.
• Association rules describe patterns in binary data, using support and confidence measures.

Test your understanding of Kernel Ridge Regression and Support Vector Machines with this quiz. Explore concepts like the Gram matrix, hyperparameters, cross-validation, and feature scaling. Challenge yourself with questions designed to deepen your knowledge of these essential machine learning techniques.