Machine Learning CS7641

Questions and Answers

What are the key elements required in breaking down a classification problem?

Instances, concept, target concept, hypotheses, input samples, candidate, testing set

What is the main purpose of decision trees in classification learning?

To provide a linear classification approach

To randomly assign outcomes

To map various choices to diverging paths (correct)

To create a decision for every possible outcome

According to the ID3 algorithm, the 'information gain' from a particular attribute A can be calculated as the difference between the overall entropy and the weighted sum of entropies of each subset. The formula is: ______(S, A) = Entropy(S) - Σ (|Sv| / |S|) * Entropy(Sv).

Gain

Decision trees for regression problems can rely on information gain to sort attributes.

False Signup and view all the answers

What is the general approach to ensemble learning algorithms?

Learn rules over smaller subsets of the training data, then combine all of the rules into a collective, smarter decision-maker. Signup and view all the answers

What is the reason behind taking the average of weak learners trained on subsets of the data to outperform a single learner trained on the entire dataset?

Avoiding overfitting Signup and view all the answers

Ensemble learning algorithms aim to prevent overfitting by learning rules over subsets of data.

True Signup and view all the answers

In boosting, what is the process of finding the weak learner that achieves the lowest error in each boosting round?

find a weak learner that achieves the lowest error Signup and view all the answers

What is supervised learning?

Supervised learning relies on human input to train a model and involves labeled data. Signup and view all the answers

What is the main difference between classification and regression in supervised learning?

Classification involves mapping complex inputs to labels, while regression maps inputs to numerical values. Signup and view all the answers

What is one of the risks in machine learning related to errors in data?

One risk is errors that can come from sources such as hardware, human mistakes, malicious intent, or unmodeled influences. Signup and view all the answers

Cross-validation is a method used to reduce the risk of overfitting in machine learning.

True Signup and view all the answers

What is the 'Goldilocks zone' in training models?

It is the ideal balance between underfitting and overfitting, where the error in training and cross-validation data is similar. Signup and view all the answers

What is the outcome when the weak learner classified incorrectly?

f(ε) = exp(−αt yi Ht(xi)) = (1−ε)/ε Signup and view all the answers

What is the outcome when the weak learner classified correctly?

f(ε) = exp(−αt yi Ht(xi)) = ε/(1−ε) Signup and view all the answers

What is the purpose of boosting in machine learning?

All of the above Signup and view all the answers

Boosting is a method that tends to overfit.

False Signup and view all the answers

What does a support vector machine try to find in the data?

The boundary that will maximize the margin from the nearest data points. Signup and view all the answers

What is the purpose of the classification function f(x) in Support Vector Machines?

To sign the dot product between support vectors and the new point x for classification purposes. Signup and view all the answers

What is the significance of the Kernel Trick in SVMs?

It allows for linear separation of non-linearly separable data. Signup and view all the answers

The kernel trick involves finding the explicit mapping function Φ in order to apply different kernels in SVMs.

False Signup and view all the answers

In the example provided, the kernel function K(xi, xj) = (1 + xTi xj)2 is a _________ product in a higher-dimensional space.

dot Signup and view all the answers

What does the choice of K(·) represent in SVMs?

Domain knowledge about the data that can help classify it better. Signup and view all the answers

What is the formula for finding the least squares approximation of the solution in linear regression?

(A^T A)^{-1} A^T y Signup and view all the answers

What is the basic model of an artificial neuron powered by?

Inputs and weights Signup and view all the answers

The perceptron is the foundational building block of neural networks.

True Signup and view all the answers

In the binary AND operation, if x1 = 1 and x2 = 1 with w1 = 1 and w2 = -2, the output is __.

0 Signup and view all the answers

Match the following terms with their definitions:

Perceptron = The simplest activation function that produces a binary output based on weighted inputs. Bias = An extra input value with a fixed weight that influences computations. Learning Rate = Controls the size of weight adjustments during training. Gradient Descent = An approach that operates on unthresholded summations for adjusting weights. Signup and view all the answers

What is the error metric defined based on the difference between the expected output (y) and the actual output (a)?

E(w) = (y - a)^2 Signup and view all the answers

What is the derivative of the error (E) with respect to a weight (wi)?

-(y - a)xi Signup and view all the answers

What function is introduced as the activation function that allows using gradient descent?

Sigmoid function Signup and view all the answers

Neural networks prefer simpler explanations over complex ones.

True Signup and view all the answers

Which optimization method allows gradient descent to 'gain speed' when descending down steep areas?

Momentum Signup and view all the answers

What is the main downside of kNN according to the text?

kNN gets slow and unwieldy very quickly due to using the training data in the querying process. Signup and view all the answers

What is the contrasting characteristic of linear regression compared to kNN?

Linear regression calculates a model upfront and makes querying cheap (constant time), while kNN is considered a lazy learner. Signup and view all the answers

What are the two important biases discussed regarding kNN representation of data?

Preference bias and Restriction bias Signup and view all the answers

Which of the following factors affect inductions in a learning problem? (Select all that apply)

Complexity of the hypothesis class Signup and view all the answers

A good question in a binary yes-or-no scenario ideally reduces the number of possibilities by half.

True Signup and view all the answers

What is the hypothesis space denoted by 'H' in machine learning?

The hypothesis space 'H' is where the candidate hypothesis 'h' is explored and considered by the learner. Signup and view all the answers

What is the version space in machine learning?

The set of all possible hypotheses consistent with the data Signup and view all the answers

A consistent learner is one that produces the correct result for all of the training samples: _______.

c(x) = h(x) Signup and view all the answers

Training error is the fraction of training examples correctly classified by the candidate hypothesis 'h'.

False Signup and view all the answers

Study Notes

Supervised Learning

Supervised learning relies on human input (or "supervision") to train a model
Examples of supervised learning include label-based learning, and it occurs more often than unsupervised learning
Supervised learning can be reduced down to function approximation
An elementary example of supervised learning is a model that "learns" a dataset represents a function, such as x²

Techniques

Supervised learning is broken up into two main schools of algorithms: classification and regression
Classification involves mapping between complex inputs and labels (discrete values)
Regression involves mapping complex inputs to an arbitrary, often-continuous, often-numeric value
Data is everything in machine learning, but it isn't perfect, and errors can come from hardware, human element, and malicious intent

Classification

Classification problems require instances, concept, target concept, hypotheses, input samples, and a testing set
Decision trees are a form of classification learning, which maps various choices to diverging paths that end with a decision
To create a decision tree, we need to identify pertinent features that describe the concept well
The "Goldilocks zone" of training is between underfitting and overfitting, where the error across both training data and cross-validation data are relatively similar

Decision Trees

Decision trees are a representation of our features, and we need to be careful about how accurately we "fit" the training data
The order in which we apply each feature to our decision tree should be correlated with its ability to reduce our space
A "best" question is one that divides our data roughly in half
The ID3 algorithm is a top-down approach to creating a decision tree, which greedily chooses the attribute with the most information gain

Asking Questions: The ID3 Algorithm

The ID3 algorithm uses information gain to qualify attributes, and the "best attribute" is the one that gives us the maximum information gain
The algorithm repeats until the labels are correctly classified, and prefers correct decision trees to incorrect ones
Attributes that give a lot of information are more valuable, and should thus be higher on the decision tree

Considerations

The ID3 algorithm has a preference bias towards trees with good splits at the top, and prefers shallower or "shorter" trees
Asking continuous questions requires binning or discretization to make them Boolean questions### Decision Trees
In decision trees, repeating attributes can be acceptable, depending on the attribute
It makes sense to ask about the same attribute twice down a branch, especially with bucketed continuous values like cost, age, etc.
Refining buckets as we go further down a branch can be beneficial

Stopping Point

The ID3 algorithm stops creating the decision tree when all training examples are classified correctly
This approach may lead to overfitting the training set and creating an infinite loop
Adopting a termination approach that is a little more general and robust can help avoid overfitting
Pruning branches that do not incur a large penalty for incorrect classification can be an effective approach

Ensemble Learning

Ensemble learning combines multiple learners to create a more accurate and robust predictor
It is powerful when features are weakly indicative of a result on their own but strongly indicative in combination
The general approach is to learn rules over smaller subsets of the training data and combine them into a collective decision-maker

Bagging

Bagging (Bootstrap Aggregation) involves creating subsets of the training data by uniformly randomly selecting examples with replacement
Combining the results with an average also works well
Bagging helps to avoid overfitting by smoothing out the specifics of each individual learner

Boosting

Boosting involves creating a sequence of weak learners that adapt to the errors of the previous learners
Each weak learner is trained on a weighted version of the training data, with more weight given to examples that were incorrectly classified by the previous learner
The final classifier is a weighted average of the individual weak learners
The weights are chosen to minimize the total error of the final classifier
Boosting can be broken down into a simple loop: construct a distribution, find a weak classifier that minimizes the error, and combine the weak classifiers into a stronger one

AdaBoost

AdaBoost is a specific boosting algorithm that follows a human approach to learning: focus on individual mistakes and adjust the weights accordingly
The algorithm starts with a uniform distribution and adjusts the weights based on the correctness of the weak learners
The final classifier is a weighted average of the individual weak learners, with more weight given to learners that did well
AdaBoost is a robust method that tries hard to avoid overfitting and achieve high confidence in its predictions### Boosting
Boosting can overfit when the underlying weak learner uses a complex artificial neural network.
Boosting can't prevent overfitting if all underlying learners overfit and can't stop overfitting (like complex neural nets).
Boosting also suffers under pink noise (uniform noise) and tends to overfit.

Support Vector Machines

SVMs operate on the notion of finding the boundary that maximizes the margin from the nearest data points.
SVMs focus on examples near the boundaries rather than the entire data set to reduce computational complexity.
The optimal margin lines will always have some special points that intersect the dashed lines, called support vectors.
SVMs try to maximize the margin while also classifying all data points correctly.
The classification function depends only on the dot product between a new point x and the support vectors xi.

Extending SVMs: The Kernel Trick

SVMs can be extended to find separation boundaries between data points in higher dimensions than the features provide.
The kernel trick is a method of finding a mapping function Φ that can represent the kernel function K as a dot product in a higher-dimensional space.
The kernel trick allows us to apply almost any kernel function and it will still find a boundary that is linear in a higher-dimensional space.

Kernel Functions

A kernel function is a function that represents the dot product in a higher-dimensional space.
The choice of kernel function encodes domain knowledge about the data and can help classify it better.
Common kernel functions include polynomial kernels and radial basis kernels.

Regression

Regression is a machine learning technique that can approximate real-valued and continuous functions.
Linear regression is the process of finding the "line of best fit" that minimizes the sum of least squared-error between the points and the chosen line.
The line of best fit can be rigorously defined by solving a linear system.
The lack of an exact solution to the system means that the vector of y-values isn't in the column space of A, and we need to find the projection of y into the column space plane.
The projection is the closest possible vector in the column space to y, which is exactly the distance we were trying to minimize.

Studying That Suits You

Use AI to generate personalized quizzes and flashcards to suit your learning preferences.

Description

An unofficial guide to Georgia Institute of Technology's CS7641 Machine Learning course, covering various concepts and topics in machine learning.

Machine Learning CS7641

Questions and Answers

What are the key elements required in breaking down a classification problem?

What is the main purpose of decision trees in classification learning?

According to the ID3 algorithm, the 'information gain' from a particular attribute A can be calculated as the difference between the overall entropy and the weighted sum of entropies of each subset. The formula is: ______(S, A) = Entropy(S) - Σ (|Sv| / |S|) * Entropy(Sv).

Decision trees for regression problems can rely on information gain to sort attributes.

What is the general approach to ensemble learning algorithms?

What is the reason behind taking the average of weak learners trained on subsets of the data to outperform a single learner trained on the entire dataset?

Ensemble learning algorithms aim to prevent overfitting by learning rules over subsets of data.

In boosting, what is the process of finding the weak learner that achieves the lowest error in each boosting round?

What is supervised learning?

What is the main difference between classification and regression in supervised learning?

What is one of the risks in machine learning related to errors in data?

Cross-validation is a method used to reduce the risk of overfitting in machine learning.

What is the 'Goldilocks zone' in training models?

What is the outcome when the weak learner classified incorrectly?

What is the outcome when the weak learner classified correctly?

What is the purpose of boosting in machine learning?

Boosting is a method that tends to overfit.

What does a support vector machine try to find in the data?

What is the purpose of the classification function f(x) in Support Vector Machines?

What is the significance of the Kernel Trick in SVMs?

The kernel trick involves finding the explicit mapping function Φ in order to apply different kernels in SVMs.

In the example provided, the kernel function K(xi, xj) = (1 + xTi xj)2 is a _________ product in a higher-dimensional space.

What does the choice of K(·) represent in SVMs?

What is the formula for finding the least squares approximation of the solution in linear regression?

What is the basic model of an artificial neuron powered by?

The perceptron is the foundational building block of neural networks.

In the binary AND operation, if x1 = 1 and x2 = 1 with w1 = 1 and w2 = -2, the output is __.

Match the following terms with their definitions:

What is the error metric defined based on the difference between the expected output (y) and the actual output (a)?

What is the derivative of the error (E) with respect to a weight (wi)?

What function is introduced as the activation function that allows using gradient descent?

Neural networks prefer simpler explanations over complex ones.

Which optimization method allows gradient descent to 'gain speed' when descending down steep areas?

What is the main downside of kNN according to the text?

What is the contrasting characteristic of linear regression compared to kNN?

What are the two important biases discussed regarding kNN representation of data?

Which of the following factors affect inductions in a learning problem? (Select all that apply)

A good question in a binary yes-or-no scenario ideally reduces the number of possibilities by half.

What is the hypothesis space denoted by 'H' in machine learning?

What is the version space in machine learning?

A consistent learner is one that produces the correct result for all of the training samples: _______.

Training error is the fraction of training examples correctly classified by the candidate hypothesis 'h'.

Study Notes

Supervised Learning

Techniques

Classification

Decision Trees

Asking Questions: The ID3 Algorithm

Considerations

Stopping Point

Ensemble Learning

Bagging

Boosting

AdaBoost

Support Vector Machines

Extending SVMs: The Kernel Trick

Kernel Functions

Regression

Studying That Suits You

Description

More Quizzes Like This

Machine Learning Quiz: Test Your Knowledge with Online Quizzes

Mastering Machine Learning Performance Metrics

The Bias-Variance Tradeoff in Machine Learning Quiz

Graded Quiz: Intro to Machine Learning with Flashcards