Information Theory: Mutual Information

What does mutual information measure?

What is the relationship between entropy and mutual information?

Which expression defines mutual information using joint probability?

Which of the following equations represents the chain rule in the context of mutual information?

What does H(X | Y) represent?

What is the value of I(X; X)?

How is the intersection of information in sets X and Y depicted?

In a probability mass function for blood type and skin cancer risk, what does a lower probability indicate?

Mutual Information

Quantifies the amount of information one random variable shares with another.
Mathematically defined as:
- ( I(X; Y) = \sum_{x \in X} \sum_{y \in Y} p(x, y) \log \frac{p(x, y)}{p(x)p(y)} )

Interpretation and Formula

Represents the reduction in uncertainty of one variable due to knowledge of another.
Relation with entropy:
- ( I(X; Y) = H(Y) - H(Y|X) )
Proof of the mutual information provided important transformations using conditional probabilities.

Chain Rule Application

Utilizes the Chain Rule for expressing entropy:
- ( H(X, Y) = H(X) + H(Y|X) )
This enables rewriting mutual information:
- ( I(X; Y) = H(Y) - [H(X, Y) - H(X)] )
- Also expressed as ( I(X; Y) = H(X) + H(Y) - H(X, Y) )

Self-Mutual Information

The mutual information of a variable with itself is equal to its entropy:
- ( I(X; X) = H(X) - H(X|X) = H(X) )

Visual Representation

Venn diagram conceptually shows that ( I(X; Y) ) is the shared area of information between X and Y.

Example Computation

Consider variables X (blood type) and Y (risk of skin cancer).
Probability mass function given for various blood types and associated cancer risks.
To compute H(X), H(Y), H(X, Y), H(Y|X), H(X|Y), and I(X; Y), apply the definitions of entropy and conditional probability based on provided data.

Explore the concept of Mutual Information in Information Theory through this quiz. Understand how it measures the amount of information one random variable contains about another and its relationship with entropy. Test your knowledge on key formulas and principles.