Machine Learning Classification Methods

Classification

• Refers to identifying the category or class of a new observation based on a training dataset with known categories.
• A supervised learning method where an algorithm learns from labeled data to make predictions or decisions.
• Aims to map input features to a specific label by learning the relationships and boundaries separating different classes.
• The primary goal is creating a function to accurately predict the category of new, unseen data points.
• Classification involves defining a decision boundary that effectively separates input data into distinct classes.
• Purposes include data categorization, automating decision-making, and predictive modeling.

Interpreting Classification Model Output

• Models often provide probabilities indicating confidence in predictions, expressing the certainty of a given input belonging to a specific class.
• Output is often represented as ŷ, a result from a function parameterized by θ, expressed as ŷ = f(x) = h_θ(x).
• x represents the input feature vector, and θ represents model parameters.
• h_θ(x) is a function that outputs ŷ.
• The prediction ŷ outputs a vector of probabilities belonging to a probability simplex.
• A probability simplex represents the set of all possible probability distributions over a finite number of categories.
• For c classes, the probability simplex A_c is a set of c-dimensional non-negative vectors (y₁, y₂, ..., y_c) satisfying non-negativity (y_i ≥ 0 for all i) and normalization (∑y_i = 1).
• In a c-class case, A_c forms a (c-1)-dimensional polytope.

Making a Classification Decision

• To make a final decision, the model uses the argmax function to select the class with the highest probability for a given input.
• argmax_i f(i) = {i | f(i) = f(s) for all j}. In this case, f(i) = g_i, so C = argmax_i g_i.

Example: Email Classification

• A model is developed to determine if an email is spam or not.
• Features like the number of suspicious words are used.
• The model outputs a probability distribution over spam (class 1) and not spam (class 0).
• An example output might be ŷ = [0.85, 0.15], indicating 85% confidence the email is not spam and 15% confidence it is spam.

Binary Classification

• A supervised learning approach to assigning data to one of two distinct classes.
• The target variable y takes values in {0, 1}, corresponding to the two classes (e.g., 0 for negative outcome, 1 for positive outcome).

Example: Logistic Regression with Sigmoid Function

• A common way to model binary classification.
• The sigmoid function (σ(z) = 1 / (1 + e^-z)) maps any real-valued number to a value between 0 and 1.
• This can transform the output to a probability.
• The decision boundary is a surface that separates the feature space into different regions, where points with a probability of class membership equal to a threshold are included in the given class.

Likelihood and Log-Likelihood Functions

• Likelihood (L) is a measure of how well a model explains observed data, expressed as L = ∏ P(y_i|f(x_i)).
• Log-likelihood (log L) is a measure of how well predicted probabilities align with actual outcomes. log L(θ) = ∑ [y_i log f_θ(x_i) + (1 - y_i) log(1 - f_θ(x_i))].

Negative Log-Likelihood

• Negative log-likelihood(NLL) is a loss function to quantify model fit.
• Minimizing NLL maximizes the likelihood of observed data being valid.

Multi-Class Classification

• Models are trained to assign instances to one of three or more classes.

Hyperplanes

• A hyperplane in N-dimensional space is a (N-1)-dimensional subspace.
• They are used to create decision boundaries in classification tasks, especially when classes are linearly separable.
• The margin is the distance between the hyperplane and the closest data points from each class; Larger margins indicate better class separation.

Calibration

• Measures how well predicted probabilities align with actual outcomes.
• A validation set, confidence bins, predicted confidence, and the number of samples in each bin are used to determine a model's calibration.
• Expected Calibration Error (ECE) is used to measure calibration by dividing the range (0-1) of predicted scores into bins.