Machine Learning Concepts Quiz

When n is very large, this dual algorithm is more likely to overfit than the primal algorithm with degree-p polynomial features.

Both primal and dual ridge regression give the same solution, no matter the size of n.

The kernel matrix is XM > MX > , where XM is the matrix multiplication of the design matrix X and the matrix M.

If the primal perceptron algorithm terminates, then the kernel perceptron algorithm terminates.

The decision tree algorithm chooses binary splits to maximize information gain at each node.

Decision tree splits are binary, meaning they split the data into two branches.

Decision trees can handle both categorical and numerical features.

Decision trees can be prone to overfitting, especially when the tree is too deep.

For every possible class label (y) in the set {0, 1, 2}, there exists a classifier (r(x)) that assigns the most likely class label to an input (x).

The classifier function r(x) is the function that maximizes the posterior probability P(Y = y|X = x), meaning it assigns the class label (y) that is most likely given the input (x).

The risk of a classifier is undefined without knowledge of the underlying data distributions P(X) and P(Y|X). These distributions are crucial for calculating the expected error of the classifier.

The classifier r(X) still minimizes the risk even if the distribution of inputs P(X) changes, as long as the conditional probability of class labels given inputs (P(Y = y|X = x)) remains the same for all inputs and classes.

If the means (µ̂C, µ̂D) and prior probabilities (π̂C, π̂D) of two classes (C, D) are equal in Gaussian Discriminant Analysis (GDA), then the Linear Discriminant Analysis (LDA) and Quadratic Discriminant Analysis (QDA) classifiers are identical.

If the covariance matrices of two classes (C, D) in Gaussian Discriminant Analysis (GDA) are equal Σ̂C = Σ̂D, and the prior probabilities are π̂C = 1/6 and π̂D = 5/6, the LDA and QDA classifiers are again identical.

If the covariance matrix of class C is the identity matrix (I) and the covariance matrix of class D is 5 times the identity matrix (Σ̂D = 5I), the LDA and QDA classifiers are identical.

If LDA and QDA classifiers give identical results, the posterior probability P(Y = C|X = x) is a logistic function of the input (x), not a linear function.

The inner product of row i of Φ(X) and column j of Φ(X).

A vector that maximizes the Rayleigh quotient, which is a measure of how much the data is spread along the direction of the vector.

A way to express the problem of maximizing the Rayleigh quotient as a maximization problem involving matrices.

The matrix that holds the inner products of all pairs of data points in the original dataset, calculated using the kernel function.

A matrix that represents the connections between nodes in a graph.

A vector that assigns each node of the graph to one of two clusters.

The problem of finding the best way to divide a graph into two clusters with equal sizes.

The minimization of a function that measures the error of a classifier based on the data distribution.

Every direction that maximizes the variance of projected data is an eigenvector of the covariance matrix, which is calculated as X'X.

Kernel PCA can only be applied to optimization problems where the solution can be expressed as a linear combination of the data points.

The kernel matrix K is an n x n matrix where each element K(i, j) represents the dot product of the feature maps of the ith and jth data points: K(i, j) = Φ(Xi)'Φ(Xj).

The kernel matrix K can also be expressed as the matrix multiplication of the feature matrix Φ(X) with its transpose: K = Φ(X)'Φ(X).

The kernel function k(x, z) = Φ(x)'Φ(z) allows calculating similarities between data points in the feature space without explicitly computing the feature maps Φ(x) and Φ(z).

The kernel matrix K is a symmetric matrix because the kernel function k(x, z) is symmetric: k(x, z) = k(z, x).

A hard-margin SVM aims to find a decision boundary that perfectly separates the data points without any errors, while a soft-margin SVM allows for some misclassifications to accommodate noisy data.

The margin of an SVM is the region around the decision boundary where data points can be classified correctly. SVMs aim to maximize the margin to improve generalization.

The smallest eigenvalue of the Laplacian matrix corresponds to the constant vector. The second smallest eigenvalue (the Fiedler value) corresponds to the Fiedler vector, which captures the best possible way to divide the graph into two clusters.

The Laplacian matrix is a matrix that captures connections between nodes in a graph. It is always singular, meaning it has an eigenvalue of 0.

Complete linkage uses the maximum distance between any two elements from the respective clusters as the inter-cluster distance. This leads to tighter, more compact clusters.

Agglomerative clustering is a bottom-up approach, starting with individual data points and merging clusters until a desired number of clusters is reached.

Single linkage can be highly sensitive to outliers as it uses the minimum distance between points in different clusters. Outliers can easily create long, skinny clusters that might not reflect the true underlying structure of the data.

AdaBoost is a powerful ensemble learning algorithm that combines multiple weak learners (e.g., decision trees) to build a strong predictor. It's used for binary classification.

AdaBoost assigns weights to each training sample, putting more emphasis on misclassified data points. This helps the algorithm focus on challenging examples and improve its accuracy.

AdaBoost can be used with decision trees, where each tree is a weak learner. The combined decision from these trees then determines the final classification.

The gradient of the log-likelihood of the mixture parameter θ (the probability of belonging to the first cluster) is expressed as a simple function of θ. This function is independent of the cluster means (µ1, µ2) and only depends on the posterior probabilities (τi) assigned to each data point.

The gradient of the log-likelihood of the first cluster mean (µ1) is a weighted sum of differences between data points and the mean, where the weights are the posterior probabilities (τi) of belonging to the first cluster.

The gradient of the log-likelihood of the second cluster mean (µ2) is similar to the gradient of µ1, but it uses the complementary probabilities (1 - τi) for each data point.

The algorithm iteratively updates the cluster parameters (means and mixing probability) and the posterior probabilities using the gradients derived from the log-likelihood function. This repetition continues until convergence, resulting in optimal cluster assignments and parameters.

The algorithm starts by arbitrarily assigning initial values for the posterior probabilities (τi) associated with each data point.

For fixed posterior probabilities (τi), the algorithm updates the cluster parameters (µ1, µ2, and θ) by maximizing the log-likelihood. The update rules are based on the gradients derived for µ1, µ2, and θ.

Using the updated cluster parameters (µ1, µ2, and θ), the algorithm re-estimates the posterior probabilities (τi) for each data point. This step essentially revisits the cluster assignment for each data point based on the new cluster definitions.

The k-means-like algorithm uses the gradients of the log-likelihood to refine the cluster parameters (µ1, µ2, and θ) and then re-evaluate the posterior probabilities (τi). This iterative process continues until convergence, resulting in the optimal cluster assignments and the corresponding parameters.