Machine Learning Concepts Quiz

Machine Learning for Smart Industry

• This course covers unsupervised learning, supervised learning, neural nets and deep learning, and reinforcement learning.

Unsupervised Learning

• Unsupervised learning uses machine learning algorithms to analyze and cluster unlabeled data sets
• These algorithms discover hidden patterns or data groupings without human intervention
• Unsupervised learning differs from supervised learning in that it does not use labeled data
• Common algorithms include:
  • Clustering (exclusive and overlapping clustering algorithms)
  • K-means clustering
  • DBSCAN
  • Gaussian Mixture Model
  • Association rule
  • Dimensionality reduction

Clustering

• Clustering involves grouping unlabeled data into clusters based on similarities
• The goal is to identify patterns and relationships in data without the need for prior knowledge about the data's meaning.
• Key algorithms types include Exclusive and Overlapping clustering methods
  • Exclusive clustering includes k-means clustering
  • Overlapping clustering includes fuzzy k-means clustering

K-Means Clustering

• A centroid-based clustering approach based on a partitioning method
• The goal is to group data points based on their closeness
• Similarity measures options are Euclidean distance, Manhattan distance, or Minkowski distance
• Datasets are separated into a predetermined number of clusters
• Recalculate centroids for observations assigned to each cluster.
• Drawback of needing to select the number of clusters (k).
• Common technique to select k is using the elbow method and the inertia metric
• Inertia metric measures the mean squared distance between each data point and its closest centroid
• General overview of k-means performs best if data is well separated
• Not suitable for non-spherical clusters nor for overlapping data points

DBSCAN

• Density-Based Spatial Clustering of Applications with Noise
• Finds groups/categories based on the density of data points
• It determines the number of clusters automatically
• Less sensitive to initial position; used for irregular/overlapping clusters
• Manages dense and sparse data regions, very flexible for diverse morphologies

DBSCAN Algorithm

• Identifies core points within a radius
• For each core point, creates a new cluster
• Finds recursively all density-connected points and assigns them to the same cluster
• Data points that do not belong to any cluster are noise
• Parameters include Radius, Minimum number of points

Gaussian Mixture Model

• A probabilistic model assuming instances were generated from a mixture of k Gaussian distributions
• For each instance, a cluster is randomly selected from the k clusters with a probability defined by the cluster's weight (mixture weight)
• The location x(i) is sampled from a Gaussian distribution with mean μ(j) and covariance matrix ∑(j)
• The algorithm can estimate the weights Φ and the distribution parameters μ and ∑.

Association Rule

• A rule-based machine learning technique to identify relationships/associations between parameters in a large dataset
• Commonly used in market analysis to understand the relationships between different products
• Common algorithms:
  • Apriori
  • FP-Growth
  • Eclat

Dimensionality Reduction

• Looks to reduce the number of features in a dataset while preserving as much of the original information possible
• Used in preprocessing stage
• Common algorithms:
  • Principal component analysis (PCA)
  • Singular value decomposition (SVD)
  • Autoencoders

Supervised Learning

• Supervised learning algorithms are given labeled data and learn a function that maps from input to output
• A common use case is classification, where the output is a discrete label, and regression, where the output is a continuous value.

Classification vs Regression

• Classification predicts a discrete label, while regression predicts a continuous value.

Linear Models

• Well-suited to illustrate introductory concepts, and act as a baseline for more complex problems.
• How measure closeness in loss function/risk minimization?
• Measure distance with norms.

Distance and Norm

• Euclidean norm (2-norm) √(x₁² + x₂² +...+ xn²)
• Manhattan norm (1-norm) |x₁| + |x₂| + ... + |xn|
• Maximum norm (∞-norm): max(|x₁|, |x₂|, ..., |xn|)

Linear Regression - Least Squares Fit (LSQ)

• Hypothesis space H is the space of linear functions f(x) = w₀ + w₁x.
• The Euclidian 2-norm is the loss function.
• Finding the solution for the parameters w₀ and w₁.

Linear Regression - Basis Functions

• Hypothesis space where parameters w appear in a linear relation, H = {f: f(x) = φw}
• collect data into matrix Φ, input vector x, and output vector y
• Minimize the Euclidian or 2-norm. Remp(w) = ||y - Φw||₂

Linear Regression – Regularisation

• Solution for parameter estimate, ŵ = Φ⁺y (pseudoinverse)
• Handles possible infinite number of solutions, e.g. smallest norm
  • Adding regularisation using the loss function with regularisation C(w):
    • min Remp(w) = min (||y – Φw||² + λC(w)) where parameter λ controls the strength

Linear Regression - 1² Regularisation

• 1² regularisation / ridge regression: C(w) = ||w||₂
• Solution: ŵ = (ΦΦ⁺ + λΙ_M)^⁻¹y_T
• M = identity matrix

Nonlinear Regression

• Not linear-in-the-parameters, non-Euclidian norms
• Matrix formalism does not apply
• No explicit solution: Iterative Methods e.g Gradient descent

Connection to Machine Learning

• Nonlinear optimisation is common
• Numerical implementation available in TensorFlow
• Using hyperparameters to select hypothesis space
• Using optimisation procedures e.g Adam optimizer

TensorFlow

• Import and process data, using pandas
• Checking data format using TensorFlow
• Considering splitting into training, validation, and testing sets.
• Scaling numeric data
• Define model using tf.keras.Sequential
• Compile the model with the optimizer, using e.g.,tf.keras.optimizers.Adam().
• Training the model with model.fit
• Evaluating the model with model.evaluate
• Computing predictions model.predict

Neural Networks

• A series of layers where nodes in each layer are fully connected to the next layer.
• Use activation function to add nonlinearities, e.g., sigmoid, tanh, ReLU.
• In simple terms: input → feature exctraction → learning → outputs

Classification

• Outputs are discrete labels (e.g., {0,1}, or categories)
• Hypothesis space is K-dimensional.
• Use activation functions to produce output compairable to one-hot vectors, e.g., softmax
• Examples of activation function are sigmoid/tanh

Kernel Functions

• A kernel function k, maps input data into a higher dimensional feature space
• Alternative solution computes similarity between data points
• Used without explicit feature maps.
• Examples include linear, polynomial, gaussian

Kernel Ridge Regression

• Solution for regularised empirical risk minimization.
• Define the kernel function (RBF/polynomial)
• Consider hyperparameters e.g., gamma in RBF

SVM Classification

• The main objective is to identify the optimal hyperplane that effectively separates data points into classes, maximizing the margin between the support vectors.

Hyperplane

• A boundary/partition in a d-dimensional space.
• Equation of the linear hyperplane w•x+b=0.
• The normal vector w and the offset b are important properties.

SVM Formulation

• The core concept of SVM is finding the hyperplane that maximises the margin between the closest data points (support vectors)
• Minimise ||w||² and yi(w•xi + b) ≥ 1

Support Vector Machines (SVMs)

• SVMs are powerful machine learning algorithms used in both linear and non-linear classification
• It is also good for outlier detection tasks

Soft Margin SVM

• Modification to the hard margin SVM to handle instances that are not linearly separable.
• A penalty term (hinge loss) is introduced to account for misclassified instances.
• The optimization problem now includes an additional term that penalizes violations of the margin or misclassifications.

Soft Margin Slack Variable

• Allows some data points to violate the margin, improving the model's ability to generalize to unseen data
• ξ : Slack variable
  • ξ = 0 → no misclassification
  • 0 < ξ <1 → some data points violate the margin
  • ξ ≥ 1 → some data points violate the hyperplane

Hinge Loss

• A loss function for maximizing the margin in soft-margin SVM
• It rewards when data points are correctly classified, and penalizes when they are misclassified, or violate the margin

SVM Using Kernel

• Kernel function is a mathematical function used to transform data into a higher-dimensional feature space
• The kernel implicitly handles the feature transformation, simplifying calculations and allowing non-linearly separable data to be transformed into linearly separable data in higher dimensions

Feature Scaling

• Feature scaling transforms feature values into a common range, which is crucial to prevent differences in feature values from skewing the learning behavior of the model
• Normalization: maps values into [0,1]
• Standardization: maps values to mean 0 & std dev 1.

SVM Classification

• Multiclass SVM: not always binary classification

SVM Regression

• Used for predicting continuous values rather than discrete labels or categories
• Predict a function that describes the data set that has the maximum absolute deviation from all the training data.

MLP for Classification

• Output neurons use softmax for activation to ensure estimated probabilities are between 0 and 1 & add up to 1.

Cross Entropy

• Measures the difference between predicted and actual probability distributions. Used as a loss function to penalize model predictions that deviate significantly from the true distribution
• In multiclass classification, it penalizes the model when it estimates a low probability for the target class.

Learning Rate

• Controls the step size that the model uses in updating parameters (weights).
• Higher learning rate can lead to faster convergence but might miss optimal values
• Lower learning rate leads to a better chance of reaching the minimum loss function but requires more epochs

Optimizer

• Gradient descent, stochastic gradient descent, SGD with momentum, AdaGrad, RMSprop, and Adam are some available options for updating network weights/parameters

Adam Optimizer

• Combines Adagrad and RMSprop in a straightforward implementation, low memory usage, faster running time, and requires less tuning.

Summary of Neural Networks Learning

• Overview of ANN, MLP (many layers), backpropagation, and activation function.
• Focus on different activation functions such as step, sigmoid, tanh and ReLU.
• Training MLPs using backpropagation involving forward and backward pass.
• Use of activation functions to add nonlinearities.

Feature Selection and PCA

• PCA is a method for dimension reduction.
• It seeks to maximise the variance and minimize errors using eigenvalues, eigenvectors, Singular Value Decomposition (SVD)

Dimension Reduction

• It reduces the number of variables while retaining the data variance.
• Used for both supervised and unsupervised learning