Generative vs Discriminative Classifiers

• Logistic Regression is a machine learning concept

Generative Classifiers

• A Generative Classifier distinguishes between cat and dog images in order to build a model to be used in classification
• Classifiers identify key identifier in data
• Classifiers assign a probability to any image to determine how cat-like it appears
• Models are run to new images to determine a fit

Discriminative Classifiers

• A Discriminative Classifier is used to distinguish cat v dog images
• Classifiers distinguish dogs from cats by identifiers such as collars

Generative vs Discriminative Classifiers

• Discriminative models directly estimate P(y|x) to establish a decision boundary

• Generative models estimate P(x|y) to deduce P(y|x) and identify probability distributions of the data

• Regression and SVMs are examples of discriminative models

• GDA and Naive Bayes are examples of generative models

• Generative Classifiers (Naive Bayes) assumes conditional independence using a functional form

• Generative Classifiers estimate parameters of P(D|h), P(h) directly from training

• Bayes rule is used to calculate P(h|D) with a generative classifier.

• Unlike Generative Classifiers, Discriminative Classifiers (Logistic Regression) assume a functional form for P(h|D) or for the decision boundary

• Discriminative Classifiers estimate parameters of P(h|D) directly from training data

• The Naive Bayes formula is YNB = argmaxₕ P(D|h) · P(h)

• The Logistic Regression formula is YLR = argmaxₕ P(h|D)

Learning a Logistic Regression Classifier

• Use feature representation to classify input.
• Use a classification function that computes y, the estimated class, via P(y|x), using the sigmoid or softmax functions.
• Maximize optimization with cross-entropy loss
• An algorithm is used to optimize the objective function like stochastic gradient ascent/descent.

Logistic Regression Explained

• Logistic Regression assumes a functional form for P(y|x):
• P(y = 1|x) = 1 / (1 + e^-(∑j wjxj+b))
• P(y = 1|x) = e^(∑j wjxj+b) / (e^(∑j wjxj+b) + 1)
• P(y = 0|x) = 1 − (e^(∑j wjxj+b) / (e^(∑j wjxj+b) + 1)) = 1 / (e^(∑j wjxj+b) + 1)
• P(y = 1|x) / P(y = 0|x) = e&jWjxj+b > 1
• ∑j wjxj + b > 0. Logistic Regression is a linear classifier
• Turning a probability into a classifier using the logistic function:
• YLR = 1 if P(y = 1 | x) ≥ 0.5
• YLR = 0 otherwise
• Logistic regression is used on movie reviews to assign sentiment class positive = 1 or negative = 0

Sentiment Classification with Logistic Regression

• Features include counts positive lexicon words, counts negative lexicon words, occurences of the word no
• Features include counts of 1st and 2nd pronouns, occurance of "!", and the word count of doc
• The weights corresponding to the 6 features are [2.5, -5.0, -1.2, 0.5, 2.0, 0.7], b = 0.1
• P(+ve|x) = P(y = 1|x) = 0.70 and P(-ve|x) = P(y = 0|x) = 1 − P(y = 1|x) = 0.30
• Since, P(+ve|x) > P(-ve|x), the output sentiment class is positive

Training Logistic Regression

• Focus on binary classification, parameterizing (wj, b) as 0:
• y = 0|x, θ) = 1/e^(∑j wjxj+b)+1
• P(y = 1|x, θ) = e^(∑j wjxj+b)/e^(∑j wjxj+b) + 1
• Learn parameters θ by minimizing the cross-entropy loss.

Cross-Entropy Loss

• It measures the difference classifier output ŷ from true output y or L(ŷ, y)
• Only 2 discrete outcomes (0 or 1) express the probability P(y|x) from classifier: P(y|x) = ŷy · (1 – ŷ)^(1-y)
• LCE (ŷ, y) = −log P(y|x) = -[ylog ŷ + (1 −y) log(1 – ŷ)], minimizing the cross-entropy loss

Minimizing Cross-Entropy Loss

• The minimum loss function is a convex optimization problem.
• Because a convex function has a global minimum
• A concave function has a global maxima

Optimizing a Convex/Concave Function

• The maximum of a concave function is equivalent to the minimum of a convex function
• Gradient Ascent is for finding the maximum of a concave function.
• Gradient Descent finds the minimum of a convex function

Gradients

• The gradient of a function is a vector pointing in the direction of the greatest increase
• Gradient Ascent finds the gradient of the function at the current point to move in the same direction.
• Gradient Descent finds the gradient of the function at the current point and moves in the opposite direction

Gradient Descent for Logistic Regression

• ŷ = f (x; 0)
• dL(f(x; 0),y) dL(f(x; 0),y)
• ∇ƉL(f(x; 0), y) = db dw1 ... dwd
• Use the update rule of Δθ = η · VoL(f(x; 0), y) to iterate gradient descent algorithm until a minimum delta

Learning Rate in Training

• η is a hyperparameter.
• Large η = Fast convergence and larger residual error/ possible oscillations.
• Small η = Slow convergence and small residual error.

Sentiment Classification examples

• Some sentiment features include word counts, the presents of specific adjectives and the use of words like no , yes , great , good

Continuing a sentiment analysis example

• The features x = [x0,x1,x2,x3,x4,x5] and θ = [b,w1,w2,w3,w4,w5]
• The algorithm performs a series of computations, and the process it continues until convergence.

Understanding the Sigmoid

• Large weights can lead to overfitting within the data and create a skew
• Penalizing larger weights can reduce overfitting

Regularization Methods

• Regularization is used to avoid overfitting by identifying and removing features that do not appear in the majority of responses
• Features correlate with the class which causes overfitting from the data, producing a poor generalization
• Avoid overfitting by adding a regularization term R(θ) to the loss function:

L2 and L1 Regularization.

• L2 Regularization is also called Ridge Regression and uses the equation R(Θ) = ||Θ||₂ = ∑ Θ²ᵢ
• L1 Regularization is calling Lasso Regression and uses the equation; R(Θ) = ||Θ||₁ = ∑ |Θᵢ|

Batch Training

• Stochastic gradient descent is stochastic with it chooses a single random example at a time to improve performance on an example.
• Choppy movement means we can compute the gradient over batches of training instances rather than a single instance.

Expressions

• Training data = {xi, y}i=1, ⋯, n where, xi = (xi1, xi2, ⋯, xid) and n is the instances in a batch and d is the dimension of an instance.
• θt+1 = θt − η/n * ∑ (xij * [e^-∑wᵢxᵢ +b)- yi])

Maximum Likelihood Estimate:

• The formula for maximum conditional likelihood estimate:
  • θMCLE = argmaxθ ∏ P(yi|xi, θ)
• Express the the Conditional Log Likelihood: -L(θ) = 1/n * log(∏ P(yi|xi,θ) = 1/n * log (e^yi ⋅∑wᵢxᵢ + b)/ e∑wᵢxᵢ + b+ 1

Gradient Ascent

• The L(0) Gradient Ascent for Logistic Regression can be expressed by: . = argmax * ∑ [ y log (1 + ex) - y * (w/x+b) ]
• Iteration of Gradient ascent will continue until the ∇θ is less than a set value (usually 1).

MCLE vS MCAP

• LCE is prone to overfitting, where as MCAP is an attempt to prevent this with penalizing a data set towards -0.25, for every gradient that does not meet its standards.

Data on emails using logistic regression

• Apply logistic regression assuming n=3.0 and beginning with b = (0,0,0,0,0,0).
• 1 entails a word is within the data, and 0 is absent from the data.

Testing Phase

• Once complete iterate over all values again applying conditions based on outcome parameters

Multinomial Logistic Regression

• Multinomial logistic regression generalizes the loss function by performing regression between 2 and K classes.
• True class data y has other elements that class that are denoted as 0.
• Classifiers have to perform an accurate estimation to create valid data to use. LCE ^(y,y) = -∑^K (yk log yk),.

Continued Multinomial Logistic regression

exp(zi) softmax(zi)= * (∑_(j=1)^k exp(zj)) P(y = class1| x) output = p(y=class2|x) p(y = class3| x ) P(Y=classk|x )

Other data

• Additional data, input features are often measured differently than output features.

Concusion

• Logistic Regression, used to discrimate and classify, can have its parameters easily modified to minimize the function of lose.