Support Vector Machines (SVM)

• Support Vector Machines (SVM) can perform both classification (SVC) and regression (SVR), although classification is more common.
• Think of an SVM classifier as fitting the widest possible "street" between different classes, known as large margin classification.
• A hyperplane is a flat affine subspace with a dimension of p-1 in p-dimensional space.
• In 2D, it's a line.
• In 3D, it's a plane.

Definitions

• Margin refers to the distance between the solid line and either dashed line.
• Support vectors correlate to the blue and purple points on the dashed lines.
• The distance between these points and the hyperplane is indicated by arrows.
• In a maximal margin hyperplane, if the coefficients are represented by β0,β1,...,βp, then the maximal margin classifier classifies an x^ test observation based on the sign of f(x) = β0 + β1x1 + β2x2 +...+ βp.

Hard Margins

• Hard margin classification strictly assigns each observation to a class based on which side of the hyperplane it is, without allowing misclassification.
• Hard margin classification only works if the data is linearly separable.
• Hard margin classification is sensitive to outliers.
• Hard margin classification is impractical in real-world applications due to misclassifications and errors.

Hyperplane and Support Vectors

• A single observation can significantly change the hyperplane because the plane is very sensitive to support vectors

Challenges to Hyperplane Calculation

• If the optimization problem has no solution when M (margin) > 0, then a "soft" margin hyperplane is developed that almost separates the classes.
• Using a classifier based on a hyperplane that doesn't isolate the two classes could lead to increased robustness to individual observations.
• The benefits may also include better classification of training observations.
• The maximal margin classifier can't be used if the classes aren't separable by a hyperplane.

Soft Margins

• Soft Margins are used when observations are allowed to be on the incorrect side of the margin, or even the incorrect side of the hyperplane.
• An observation might be on the wrong side of the hyperplane, in addition to being the wrong side of the margin.
• Support vector classifiers may misclassify observations on the wrong side of the hyperplane.

C

• Slack variables ε1,...,εn allow observations to be on the wrong side of the margin or the hyperplane.
• The tuning parameter C constraints the sum of the epsilon i's, which controls how many violations to the margin and hyperplane will be tolerated.
• If εi = 0, the ith observation is on the correct side of the margin.
• If εi > 0, the ith observation violates the margin and lies on the wrong side of the margin.
• If εi > 1, the ith observation is on the wrong side of the hyperplane.
• The tuning parameter C is a hyperparameter.
• Large C means the margin is wide, many observations violate it, and there are many support vectors, which leads to low variance but potentially high bias.
• Small C results in seeking narrow margins, meaning it has a low bias, and high variance.
• Observations are called support vectors if they lie directly on the margin, or the wrong side of it as they affect the support vector classifier.

Kernel Options

• Use Polynomial kernel to solve polynomial relationships between features.
• The Radial Basis Function (Gaussian) kernel is a good default if you are unsure of the data distribution.
• Use Sigmoid kernels when data acts like a neural network.
• polynomial kernel of degree d is used to improve maps to a higher dimensional vector space and creates more flexible support vector
• γ is a positive constant that minimizes as e^(negative numbers) is gotten
• If two observations are far away from each other, the Euclidean distance will be larger. The values decreases and provides low impact on computing decisions.
• The dimensions are implicit or infinite, meaning computation is feasible.

Unsupervised Methods

• Unsupervised learning trains models using no dependent variable.
• The aim is to understand data and create groupings rather than predict a value or class probability.
• Principal Component Analysis (PCA) is an unsupervised technique that can prepare data for supervised learning.
• Clustering is the process of finding and assessing data groupings.
• The 3 main areas of unsupervised learning include:
  • Clustering (e.g., K-means, hierarchical clustering)
  • Dimensionality Reduction (e.g., Principal Component Analysis (PCA), t-SNE)
  • Association rule learning (e.g., Apriori algorithm)
• Common supervised learning tasks lack labeled data or ground truth.
  • For example, fraud detection, medical imaging, cybersecurity, Natural Language Processing, and recommender systems have this issue.
• Analysis becomes subjective, which makes evaluating models difficult without ground truth or predefined metrics.
• It is difficult to assess the results or performance, like RMSE (regression), Accuracy and Precision(classification).
• Scalability may be problematic.
  • Some algorithms are computationally intensive with large, high-dimensional data sets.
  • These algorithms can be slow and require a lot of memory.
• Overfitting sensitive methods to the parameters or model complexity is a risk.
• Unsupervised learning can be sensitive to noise and outliers.
• A constraint lies in algorithms making assumptions about the data (e.g. cluster shape or distribution)
• Increased Computational Complexity, sparse data, overfitting, distance metric issues, visualization challenges, and feature redundancy are a number of effects from the curse of dimensionality.
• Techniques to reducing dimensionality include PCA, t-SNE, and LDA.
• Feature selection can occur using the filter and wrapper methods.
• Sampling techniques to reduce dimensionality that leverage feature engineering or use random projection.

PCA

• Principle Component Analysis finds a low-dimensional data representation while retaining variance.
• It works when limited knowledge or methods do not make election of other approaches feasible.
• PCA projects observations with the largest variance via a vector or loadings
• PCA may not always be helpful to prediction as it only provides a direction that maintains the most variance in the data, due to it's unsupervised nature.
• The rotation results in resulting retains max(var).

Clustering

• Clustering groups similar data points into clusters.
• The goal is to organize objects in such a way that those in the cluster are more similar than those in other groups.
• It helps discover patterns, structures, and relationships in data.

K-Means Clustering

• K-means clustering partitions data into K distinct, non-overlapping clusters.
• The algorithm attempts to minimize intra-cluster variance
• The number of clusters K must be specified and observations assigned to the K clusters
• The steps include:
  • Initialization
  • Assignment
  • Update
  • Iterate -The point is one for which variation in the with-in cluster is as small as possible.
• Many ways to define this concept, but is typically handled by squared Euclidean distance.
• Within-cluster variation of the kth cluster is the sum of Euclidean distances squared between kth cluster observations, and then divided amongst the number of kth clusters

Algorithm Steps

• A number is randomly assigned, from 1 to K, to each observation that acts as initial cluster assignments.
• Iterate until the cluster assignments stop changing.
  • Compute each of the K clusters cluster centroid that is a vector of the p feature means for the kth cluster observations.
  • Assign each observation to the cluster with the closest centroid, where closest is found leveraging Evaluation distance. -The result is a local optimum because the K-means algorithm locates a local instead of global optimum.
• A hyperparameter is the number of clusters, with methods to optimization such as the “elbow” and “silhouette”.
• The elbow method plots the within-cluster sum of squares (WCSS) against k values, where the point of the "elbow" is where the improvement slows down
• The silhouette Score measures how similar a datapoint is to its own cluster instead of other ones, where it typically ranges from negative 1 to 1 and has higher values indicate better clustering.

Other Hyperparameters

• K-Means is sensitive to the initial placement of centroids through the Initialization Method (Centroid Initialization):
• Some common strategies:
  • Random initialization: The default method that can lead to suboptimal results.
  • K-Means++ initialization: Improves clustering by spreading out initial centroids and reduces the risk of poor convergence.
• Distance Measurements are typically handled using the Evaluation Method.

DBSCAN

• Density-based spatial clustering of applications with noise (DBSCAN) identifies clusters where points are closely packed using a density-based algorithm.
• Arbitrary-shaped clusters, with outliers identified as noise, can be found using the algorithim.

Density

• How much mass is packed into a given volume of substance can be found using density.
• In DBSCAN, the region density is much greater compared to other locations with more observations.
• There exists code points that leverage density to cluster datasets.

Key Concepts:

• Core points have a minimum number of neighboring points are within a specified distance (minPts and epsilon).
• Border points are within the ε distance of the code point, but not enough neighbours to be a border points.
• Noise points belong to no cluster.

Pros and Cons

• Does not require predetermining the number of clusters and can detect noises and outliers.
• It works wll with arbitrary shapes and has only two parameters to fine tune.
• There are limitations of being able to struggle with varying density clusters, and as it is sensitive to the parameters ε and minPts.

Applications

• Geospatial data
• Anomaly detection
• Image segmentation

DBSCAN Algorithm Steps

• Initialize Parameters with epsilon and minPts.
• For each point in the dataset, Skip it.if it has already been visited
• Determine the points that surround the current the points that are in its ε-neighborhood distance.
• Check if the current point is a core point with the following criterias:
• If the number of points(which includings itself), is greaterthan minPts, the point takes that form and starts a new cluster.
• If the number of points does not reach minPts, Then its market as noise.

Expansion ofCluster

• For any code, recursively visit its neighborhood in order of neighborhood
• If you visit a nonvisited one, mark to be part of existing cluster
• Cluster can grow by visting neighbours.
• border points is included by not expanded

Hyperparameter Tuning

• Set minPts to at least D+1, where D is the number of dimensions in the dataset, and increase to void affecting result and to use higher number for noisy data to make dense regions, and if need of small groups use lower number for small data.
• To compute the Nearest Distance:
• Choose k as mintPts - 1.
• Compute the distance of each point with its Knn.
• And to plot the sorted distances:
• Sort and plot them to look for elbow location to look for point to start point distance for good ε.

Hirarchical Clustering

• Organizes cluster hierarchy
• Has two types:
  • Agglomerative (Bottom-Up) : Starts with individual points and merges them. -Divisive: Top-Down starts with all point and splits them.

Hierarchical Clustering Advantages

• Not need specific number in K-means.
• It's usefull to visualize data It takes advantages with a genomics and market segmentation for image analysis

Interpreting the Dendrogram

• Each leaf on a dendrogram is an observation
• Moving up the tree, leaves start to fuse,
• Internal Nodes are clusters.
• Hierarchichal (2) has height number of nine in clusters
• Height at five results in 3 clusters and the role is controling the numbers in the K-means
• All are nested groups

###Agglomerative Approach

• Initializes and starts each pont with its own cluster
• Computes a matrix distance that has to be every pairwise distance.
• -It needs a chosendistance metric.

Linkage Criterias

A Matrix needs to be updated.

-Single Linkage- which distance bewteen the two closet points . -Complete distance which is two closets points Average points are in a a middle ground with spherical cluseters. .Wards Method minimzes varince.

Divise Approach

The Approach Is to begin in a smaller cluster this is less common to its higher computational complexity It initialize starts with a data point.

###Splits Has to indentifty which cluster and it goes though kmeans Repeat the process with a criteria

Considerartions . Euclidian Menhattan has to be handled elongated and can result in broken clusters. Aveage can balanced with spherucal cluseters.

Directed Acyclic Graphds

• Nodes is NN and also DAG
• DirectEdiges in ands direct acycling and meaning in edges has direction,it signifies a one -way dependency

Acyclic

It doent contait cycles or closed loops, Cant to traverses or directed in Edges.

Can have a coputational grapth can extent DAG.

###The Perceton

The Perceptron is a mathmetical Function used and can be use into a value. Its visualized by a single lay network Its the form and has an binary class tasks

###Perceptron It leverages by by the Step Backporation Eta controlls the weights and the leariningrate.

NonLiniairty it enables the model it works on the traninf OutPut helpconstrant Values Gradient Proragation they used dur the trang.

LossFuctionsand and that measures difference

###Perpectrons It test the operation it doesnt haved a handle it has network that

• a complex solt and ha a neural network

###MulityalPer The part that takes the input and activations it makes

BackPropations to update weights g. descent.

Deep Neural networks.

Hidden lays a e that network which has deeped. Incresd Capcity which is layesd with data. Fets Abstactions each has that learn abstractly and it needs to combine this from complex

• Backportation

###The need Power and Speed Parallel Processing contains the build for vector calc It can help to processed in largeamounts It supports a wide of learning Its can used for quick data

• Custom architectures and designs to designed learning.

• Tensor Operation

• high ###Batch Epochs Baches and Pass through data And with help. The modelprocess from desent. Numeric with the MSE, and MAE. The

• Binary test are used for just 1 an the are and it comparas that are daptive

They gradients the