Naive Bayes Classifier

Introduction to Naive Bayes Classifier

• Naive Bayes is a significant theorem in probability with practical applications in machine learning.
• It calculates the probability of an event occurring given the knowledge of another event's occurrence.
• Naive Bayes extends this by making simplifying assumptions when dealing with multiple events to ease calculations.

Spam Detection Example

• A spam detector is to be built using a dataset of 100 emails, with 25 identified as spam and 75 as non-spam.
• The appearance of specific words in emails is correlated with their classification as spam or not spam.
• The word "buy" is studied: 20 spam emails and 5 non-spam emails contain the word.

Conditional Probability and Bayes' Theorem

• The probability of an email being spam if it contains the word "buy" is calculated from the data which yields 80%.
• From the emails containing "buy", 80% are spam, showing association between word and email classification.
• This directly applies Bayes' Theorem, which can be represented as a formula.

Application to a Second Word: "Cheap"

• The word "cheap" is studied to determine its correlation with spam emails.
• It appears in 15 spam emails and 10 non-spam emails.
• When an email contains the word "cheap", there's a 60% probability it is spam

Combining Two Words and the Challenge of Data Sparsity

• An attempt is made to determine the probability of an email being spam if it contains both words "buy" and "cheap".
• 12 spam emails have both words, while zero non-spam emails contain both words.
• Initial calculation suggests a 100% probability of spam, but that seems too strong.

Addressing Data Sparsity with Naive Assumption

• One solution is to gather more data to find non-spam emails containing both "buy" and "cheap".
• With a dataset of 100 emails, 5 contain "buy" and 10 contain "cheap", but none have both.
• The concept of independence is introduced where the appearance of "buy" does not affect the appearance of "cheap".

The Naive Bayes Assumption of Independence

• The naive assumption is made that the words "buy" and "cheap" are independent, simplifying the math.
• An estimated 0.5% of emails containing both words came from the basis of independent events.
• Allows easier calculation without the need for a large expanded unique dataset.

Applying the Naive Bayes Classifier

• Based on existing data of 25 spam emails: 20 contained "buy" (4/5) of the emails, and 15 contained "cheap" (3/5) of emails.
• Based on existing data of 75 non-spam emails: 5 contained "buy", and 10 contained "cheap".
• These probabilities allow estimation of the likelihood of both words appearing together.

Calculating Probabilities with the Independence Assumption

• The product of the individual probabilities (4/5 and 3/5) is used to estimate the combined probability (12/25).
• This is expanded to estimate that 12 spam emails would contain both "buy" and "cheap".
• Assuming "buy" appears in 1/15 and "cheap" in 2/15 of non-spam emails, the product is 2/225.

Final Probability Calculation

• Among emails containing "buy" and "cheap", the split is 12 spam emails to 2/3 of a non-spam email.
• The probability of an email being spam if it contains both words is calculated as approximately 94.73%.
• This is found by dividing the split between the spam numbers (12) and the total (12+2/3).

Summary of the Naive Bayes Approach

• The technique involves filling a table based on available information, using calculations in place of missing ones.
• The frequency of "buy", "cheap", and "work" is tracked in both spam and non-spam emails.
• When direct data is lacking, probabilities are calculated by the assumption of independent words.

Incorporating Additional Features

• Incorporating the word 'work': it appears five times in spam emails and 30 times in non-spam.
• "work" appears less in spam emails, it doesn't help the Spam classifier.
• 1/5 of spam emails contain the word "work".

Impact of Combined Features

• Adding the word "work" decreases the probability of an email being spam to 90%.
• The Naive Bayes classifer helps combine multiple features to estimate probability of an email being spam.
• Requires creating a model with direct satisfaction of all variable.

Formal Definition of Naive Bayes

• Key formula of the Naive Bayes theorem is the probability of an email being spam if it contains the word “buy”.
• Formula calculates ratio for probability, probability of Spam given word Buy, that is also Ham given word Buy.
• This ratio is key to calculating probability.

The Essence of Naive Bayes

• Key assumption of the Naive Bayes theorem is that the product of probabilities of buy and cheap appearing is the probability.
• Probability of Spam given Buy is used with the formula calculating probabilities, calculating ratios, and estimating probability.
• All the steps contribute to an accurate method that yields a 94.73% accurate method.