Machine Learning Foundations - Unit 1

Measures the proportion of matching attributes in binary data. It focuses on the number of shared features between two data points.

Used to calculate the linear correlation between two attributes. It measures the strength and direction of their linear relationship.

Measures the angle between two vectors. It indicates how similar the directions of two objects are, regardless of their magnitude.

Calculates the straight-line distance between two points in a multi-dimensional space. It's a common way to measure dissimilarity between data points.

Divides data into branches based on feature values to create an easily interpretable model for making predictions.

A common algorithm used to create decision trees by iteratively splitting the data based on the feature that offers the most information gain.

Makes decisions based on a set of predefined rules that have been learned from the data. Each rule has a condition and a corresponding action.

A generalized form of distance calculation that encompasses Euclidean and Manhattan distances. It calculates a power-based distance between points.

The k-Means algorithm starts by placing the initial cluster centroids. There are different methods, but the standard one is randomly selecting k data points as centroids.

'k' in k-Means represents the number of clusters you want to create. It's the predetermined number of groups the algorithm will form.

During each iteration, the k-Means algorithm updates the positions of the centroids. It calculates the mean of all data points assigned to each cluster and moves the centroid to that mean.

A big issue with k-Means is its sensitivity to the initial placement of centroids. Bad starting positions can lead to suboptimal and inaccurate clusters.

The k-Means algorithm converges when the centroids no longer shift significantly between iterations. This means the data points have stabilized in their assigned clusters.

The k-Means algorithm usually employs the Euclidean distance as a metric to calculate the distance between data points and centroids.

The k-Means algorithm has a computational complexity of O(n*k). This means the runtime increases linearly with the number of data points (n) and clusters (k).

One of the key strengths of k-Means is its computational efficiency. It's relatively fast compared to other clustering techniques, especially for large datasets.

A technique used in SVM to transform data into a higher-dimensional space. This allows for the separation of data that cannot be linearly separated in the original space.

Different kernel functions used in SVM to achieve different levels of complexity and flexibility in the decision boundary. Linear, Polynomial, and Gaussian are common examples.

A kernel function NOT commonly used in SVMs because it's not designed for classifying data.

Represents the marginal probability of event B occurring. It's the probability of observing B without considering any relationship to A.

A classification algorithm that uses Bayes' Theorem to predict the probability of a class based on features. It assumes features are independent of each other, making it simple and fast, but potentially less accurate.

A data point that directly impacts the decision boundary in an SVM. It's a point that is closest to the boundary line.

A function used in neural networks to introduce non-linearity into the model. It transforms the input data to allow for learning complex relationships.

An activation function commonly used in the output layer of a multi-class classification neural network. It outputs a probability distribution over all classes, indicating the likelihood of each class being the correct prediction.

A clustering algorithm that groups data points based on density. It identifies clusters as areas of high density separated by areas of low density.

An algorithm that partitions data into k clusters by iteratively assigning data points to the closest cluster center and updating the cluster center based on the assigned data points.

A hierarchical clustering method that starts with each data point as a separate cluster and then merges clusters iteratively based on their similarity until a desired number of clusters is reached.

A clustering algorithm that iteratively moves data points towards the denser regions of the data, eventually converging to cluster centers.

A probabilistic classifier based on Bayes' Theorem that assumes independence of features given the class label. It calculates the probability of a data point belonging to each class and predicts the class with the highest probability.

A mathematical formula that calculates the conditional probability of an event A given that event B has occurred.

A supervised learning algorithm that finds a hyperplane that best separates data points of different classes. It maximizes the margin, or distance, between the hyperplane and the closest data points (support vectors).

A kernel function used in SVMs to handle non-linearly separable data by mapping data points into a higher-dimensional space. It calculates the similarity based on the distance between data points using a radial basis function.

A technique used in nonparametric density estimation, like kernel density estimation, to create a smooth and continuous approximation of the data distribution. This involves using a kernel function to weigh nearby data points, creating a smoother curve.

The size of the intervals used in histograms to group data points. A larger bin width combines more data points into each bin, leading to a smoother histogram, while a smaller bin width provides a more detailed but potentially jagged histogram.

Similar to bin width, bandwidth in kernel density estimation controls the smoothness of the density estimate. A larger bandwidth creates a smoother curve with a larger influence of distant data points, while a smaller bandwidth leads to a more detailed, potentially noisy curve.

Kernel Density Estimation (KDE) uses kernel functions to create a smooth, continuous density estimate, while histograms use bins to group data points, leading to a stepped, discontinuous representation.

A central concept in machine learning. In density estimation, increasing bandwidth reduces variance (noise) but introduces bias (distortion). Conversely, decreasing bandwidth increases variance but reduces bias.

Statistical methods that do not make assumptions about the underlying distribution of the data. This allows them to work with more complex, real-world data distributions.

The process of creating a smoother, more continuous representation of the data distribution by adjusting the bandwidth or bin width in kernel density or histogram estimation.

Increasing the number of bins in a histogram generally results in a more jagged, detailed representation of the data distribution. This can reveal more subtle features within the data but potentially introduce noise.

The bin width determines the number of bins in a histogram. A smaller bin width leads to more detailed visualization, but might cause gaps and instability with limited data. A larger bin width smooths out the data but might hide important details.

Histograms require a sufficient amount of data to provide a reliable representation of the underlying distribution. Small sample sizes can lead to uneven and misleading bin counts.

In Kernel density estimation, the kernel function defines the shape of the smoothing function used to estimate the probability density. It influences the smoothness and detail level of the estimated density curve.

Kernel density estimation aims to create a smooth estimate of the probability density function from a given dataset. Unlike histograms, it avoids abrupt jumps and provides a more continuous representation of the distribution.

In k-nearest neighbor density estimation, 'k' determines the number of neighboring data points considered when estimating the density at a specific point. A higher 'k' value results in smoother density estimates.

A key advantage of k-nearest neighbor density estimation is its non-parametric nature. It doesn't require assuming a specific distribution shape. It's also simple to understand and implement.

Kernel density estimation is a preferred method for handling large datasets where the underlying distribution is unknown. It offers a smooth and flexible approach to density estimation.

Histograms use bins to group data, creating a step-like visualization. Kernel density estimation uses smoothing functions to create a smooth curve representing the distribution. This makes kernel estimators more flexible and less prone to abrupt jumps.