Understanding Conditional Probability

Conditional Probability

• Conditional probability is the likelihood of an event occurring under a specific condition
• The sample space reduces to the conditional element based on the condition
• Conditional probability is important for classification in machine learning and prediction in data science
• Decisions are typically made based on evidence in these tasks
• P(A|B) is the probability of A given B

Real-world Example

• Weather forecasting is a real-life example of conditional probability
• The event "rain" is the outcome of interest, and "cloudy sky" is the condition
• Data might show rain occurs 60% of the time when the sky is cloudy
• P(Rain | Cloudy) = 60%, meaning a 60% chance of rain if the sky is cloudy
• P(Rain) might be 30%, meaning it rains on 30% of all days

Conditional Probability Calculation

• With the initial die being a 3, the possible outcomes are (3, 1), (3, 2), (3, 3), (3, 4), (3, 5), and (3, 6)
• Each outcome has a probability of 1/6
• Let E be the event that the sum of the dice is 8, and F be the event that the first die is a 3
• P(E|F) is the conditional probability of E given F
• P(E|F) = P(E∩F) / P(F)

Conditional vs. Unconditional Probability

• P(E|F) is not generally equal to P(E)
• The occurrence of F changes the chances of E's occurrence
• E is independent of F when P(E|F) = P(E)
• Knowing that F has occurred does not change the probability that E occurs
• If E is independent of F
• P(E∩F) = P(E)P(F) leading to P(E|F) = P(E)

Law of Total Probability

• A concept in probability theory used to calculate the probability of an event by considering all possible ways it can occur
• Establishes a relationship between marginal and conditional probabilities
• If the sample space consists of disjoint events probability of any event can be expressed by
• Summing the probabilities of that event given each possible outcome in a partition, weighted by the probability of each outcome occurring

Formula

• P(A) = Σ [P(A|Bi) * P(Bi)] , where the sum is taken over all i.
• P(A) is the marginal probability
• P(A|Bi) is conditional probability
• P(Bi) the weighted probability

Bayes' Theorem

• Bayes' Theorem is an extension of conditional probability
• It helps determine the probability of A given B, denoted by P(A|B)
• If one knows P(B|A), then P(A|B) can be determined if P(A) and P(B) are known
• Bayes' theorem is a formula that updates the probabilities of hypotheses when given evidence
• Bayes' theorem is used to build a learner that predicts the probability of a variable belonging to a class given new attributes/parameters
• Naïve Bayes classifier

Bayes Rule Terms

• P(A|B) is the posterior probability
• P(B|A) is the likelihood
• P(A) is the prior probability
• P(B) is the evidence
• Formula: Posterior = Likelihood * Prior / Evidence
• In terms of fire and smoke: P(Fire|Smoke) = P(Smoke|Fire) * P(Fire) / P(Smoke)

Practical Example

• In a neighborhood where fires are rare but barbecues cause smoke, you want to know the probability of a fire given smoke
• P(Fire|Smoke): Probability of a fire given you see smoke
• P(Smoke|Fire): Probability of seeing smoke if there's a fire
• P(Fire): Overall probability of a fire in the neighborhood regardless of smoke
• P(Smoke): Overall probability of seeing smoke in the neighborhood whether it's from a fire or a barbecue

Understanding Naive Bayes

• A classification technique based on Bayes' Theorem assuming independence among predictors
• The presence of a certain feature in a class is unrelated to the presence of any other feature in a Naive Bayes classifier
• A fruit can be considered an apple, if it is red, round, with a diameter three inches
• Each property independently contributes to the probability that the fruit is an apple, even if these features depend on each other
• Naive Bayes models are easy to build and are particularly useful for very large datasets.
• Naive Bayes outperforms sophisticated classification methods
• Bayes theorem computes posterior probability P(c|x) from P(c), P(x), and P(x|c)

Naive Bayes Terms

• P(c|x) is the posterior probability of class (c, target) given predictor (x, attributes)
• P(c) is the prior probability of class
• P(x|c) is the likelihood, the probability of the predictor given class
• P(x) is the prior probability of the predictor

Naive Bayes Classifiers Calculations

• P(yi | x1, x2, ..., xn) = P(x1, x2, ..., xn | yi) * P(yi) / P(x1, x2, ..., xn)
• The denominator P(x1, x2, ..., xn) is often removed since it serves as a constant for normalization
• Given independent features
• P(yi | x1, x2, ..., xn) = P(x1|yi) * P(x2|yi) * ... * P(xn|yi) * P(yi)
• Each class label is calculated, and the label with the highest probability is chosen as the classification
• This rule is the maximum a posteriori (MAP) decision rule

Naive Bayes Assumptions

• Every pair of observations/features/random variables being classified is independent of each other
• Assumes that each feature makes an "independent" and "equal" contribution to the outcome

Naive Bayes Implementation

• Convert the dataset into a frequency table where
• P(yi) = examples with yi / total examples
• Create a likelihood table containing all P(xi|yi) for K classes and compute k * n different probabilities
• The naive Bayesian equation is used to calculate the posterior probability for each class, highest being the outcome of prediction

Different Data Types

• A different approach is required depending on the data type of each feature:
• Binary data
• Use Binomial distribution
• applies Binomial Naive Bayes
• Categorical variables
• Use Multinomial distribution. -applies Multinomial Naive Bayes
• Numerical data
• Use Gaussian distribution -applies Gaussian Naive Bayes

Naive Bayes Models

• Assumptions about the distributions of features are called the "event model"
• Multinomial and Bernoulli distributions are fitting for discrete features like spam filtering.
• A normal (or Gaussian) distribution is assumed for continuous data