Bayes Rule

• Bayes rule is expressed as P(E/F) = P(E,F) / P(F)

• P(E,F) can also be expressed as P(E/F) * P(F)

• P(E,F) can also be expressed as P(F/E) * P(E)

• Equating the two expressions for P(E,F) gives P(E/F) * P(F) = P(F/E) * P(E)

• Bayes Rule is denoted P(E/F) = P(F/E) * P(E) / P(F) which finds the probability of event E given measurement F.

• F can be noisy in the process.

• One should measure probability which quantifies the uncertainty in F.

• P(Ci/F) represents the chances of labeling Ci for a particular measurement F.

• P(Ci/F) = P(Ci,F) / P(F) = [P(F/Ci) * P(Ci)] / P(F)

• In the equation, P(F/Ci) is the likelihood and P(Ci) is the prior.

• S = C1 U C2, which is also expressed as S = U(Ci), represents the union of mutually exclusive events (classes).

• F = U F(Ci)

• P(F) = Σ P(F, Ci)

• P(F) = Σ P(Ci) * P(F/Ci)

• The distribution of F (e.g., measurements) over the sample space S gives the total probability.

• Likelihood P(F|Ci) assumes F -> Ci.

• it answers what the chances are of something corresponding

• Prior outlines the chances of that class actually being present.

Independence

• Independence is PCA/B) = P(A)
• A is not dependent on B.
• P(A,B) = P(A)P(B)

Decomposition

• Decomposition of a joint distribution in terms of a 1D conditional distribution.
• Practical example in DS: P(F/Ci) is the likelihood of one attribute conditioned over a class.
• In the case of multiple attributes P(F1, F2, ..., Fn/C), it indicates the likelihood of multiple attributes conditioned on a class.
• If F1, F2, ..., Fn are independent, P(F1, F2, ..., Fn/C) = P(F1/C) * P(F2/C) * ... * P(Fn/C)
• Decomposition transforms a high-dimensional likelihood into low-dimensional likelihoods.
• This concept is known as conditional independence.

Independence of Complement

• If A and B are independent, then A and B' (complement of B) are also independent.
• P(A, B') = P(A) * P(B').
• The proof: A = (A ∩ B) ∪ (A ∩ B')
• P(A) = P(A ∩ B) + P(A ∩ B')
• P(A) = P(A) * P(B) + P(A ∩ B')
• P(A ∩ B') = P(A) * (1 - P(B))
• P(A ∩ B') = P(A) * P(B')

Independence of Multiple Events

• Example for three events, A, B, C: P(A ∩ B ∩ C) = P(A) * P(B) * P(C)
• P(A ∩ B) = P(A) * P(B)
• P(B ∩ C) = P(B) * P(C)
• P(A ∩ C) = P(A) * P(C)
• Independence of n events A1, A2, ..., An implies independence of all subsets of {A1, ..., An}.

Distributing (Simple Examples)

• Independence represents a distribution of two independent events (yes/no).

• P(A = 0) = p

• P(A = 1) = q = 1 - p

• This distribution is known as the Bernoulli's distribution

• The experiment, which has two outcomes, is repeated multiple times.

• One way to consider probabilities in such a scenario is given by the Binomial distribution.

Binomial Distribution

• If an experiment with two outcomes, with probabilities p and q (yes/no), is repeated k times, the probability of k successes in n apps is given by the Binomial distribution.

• To derive this, some background in country is needed

• Simple country relates involves complex ways counting involving relationships, arrangements, and selections of different variables.

• Two important counting principles include permutations and combinations.

• Permutation deals with arrangements.

• The number of ways in which k objects can be arranged among n positions is denoted as P(k,n) = n(n-1)(n-2)...(n-k+1).

• For k = n, P(k, n) = n(n-1) ... 1 = n!

• P(k, n) = n! / (n-k)!

• A famous example of permutation is the birthday pairing problem:

• Number of possible states for each person: 365

• Number of people: k

• Total number of stales: 365k

• Denominator in computing probability

• Total possible number of outcomes

• To compute the number of cases where at least 2 share a birthday.

• Compute the number of cases where no 2 people share th birthday

• Assumption: k <= 365

• P(k, 365) is the number of cases where no two people share birthdays.

• Probability = P(k, 365) / 365k

• Probability of at least 2 people sharing a birthday = 1 - [P(k, 365) / 365k], k <= 365

Combination: Selection

• Combination refers to the selection of items.

• It deals with how many ways you can select k objects from n objects.

• For example, for two objects a and b, (a, b) or (b, a) is the same selection.

• But a -> b is a different arrangement.

• Permutation has an aspect of arrangements in addition to selection

• Permutation = (Selection of k) * (Arrangements of k).

• P(k, n) = (C(k, n)) * k!

• C(k, n) = P(k, n) / k! = n! / [(n-k)! * k!]

• Bernoulli distribution gives rise to Binomial distribution.

• If the Yes/No trial are repeated n times, then we can have K Yes and n-k No

• The probability of having k successes (yes) in n trials, where the probability of a single success (yes) is p (which comes from the Bernoulli distribution).

• Assumption: Each trial is independent.

• Probability of: P(n,k,p) = (p)(p)(p)(1-p)(1-p)...(1-p) = (p)^k (1-p)^(n-k)

• p(n,k,p) = Probability p to the power k times (1 - p) to the power of (n - k) all divided by 6.

• Other scenarios: different ways of selecting k out of n is like p(1-p) p (1-p) ...

• C(k,n) which is also called (n k)

• is another notation of Combination

• Overall probability when considering all possible combinations.

• P(p,n,k) = (C k,n)* p^k * (1-p)^n-k

• Where P(P,n,k) is probability as a function

• C k,n idicates the number of possible combinations of yes/no

• p^k *(1-p)^n-k indicates the probability for one instance of combination of yes/no