Information Theory: Entropy Function

Questions and Answers

What does the entropy of a random variable measure?

The probability distribution of the variable

The maximum value the variable can take

The actual values of the variable

The average uncertainty of the variable (correct)

Which formula correctly represents the entropy of a discrete random variable X?

H(X) = ∑ P(x) log(K)

H(X) = ∑ P(x) log(P(x))

H(X) = -∑ P(x) log(P(x)) (correct)

H(X) = -∑ P(x) log(K)

When is the entropy maximized for a random variable X that can take on K values?

When X is uniformly distributed among K values (correct)

When X has a mean of zero

When X is normally distributed

When X is skewed towards one value

What is the value of entropy H(X) when a random variable has a probability distribution of either P=0 or P=1?

<p>0 bits</p> Signup and view all the answers

What is the significance of the base 'b' in the entropy formula?

<p>It determines the scale of measurement for entropy.</p> Signup and view all the answers

Study Notes

Entropy Function

Entropy measures the uncertainty of a random variable.
For a discrete random variable ( X ) with alphabet ( x ) and probability mass function ( P(x) = \text{Pr}{X = x} ).
Entropy ( H(X) ) is defined as: [ H(X) = -\sum_{x \in X} P(x) \log_b P(x) ]
( b ) denotes the base of the logarithm, influencing units of measurement (bits, Hartleys, or nats).

Characteristics of Entropy

Represents the average information contained in a random variable ( X ).
Depends solely on the distribution of ( X ), not the specific values ( X ) can take.
Always greater than or equal to zero, ( H(X) \geq 0 ).

Example of Binary Entropy

For random variable ( X ) taking values:
- ( 0 ) with probability ( 1-p )
- ( 1 ) with probability ( p )
Entropy given by: [ H(X) = -p \log p - (1-p) \log (1-p) ]
Denoted sometimes as ( H(p, 1-p) ).
Maximum entropy occurs when ( p = 0.5 ), and entropy is zero when ( p = 0 ) or ( p = 1 ).

Uniform Distribution and Maximum Entropy

For a random variable ( X ) taking ( K ) values, entropy is maximized when ( X ) is uniformly distributed among these values.
In this case: [ H(X) = \log K ]

Calculating Entropy

Scenarios where: [ H(X) = -\sum_{x \in X} P(x) \log \left(\frac{1}{P(x)}\right) ]
Using Jensen's inequality for convex functions: [ \sum \lambda f(x) \leq f\left(\sum \lambda x\right) ]
Resulting inequality shows: [ H(X) \leq \log K ]
Indicates that maximum entropy ( H(X) ) occurs at ( \log K ) when uniformly distributed.

Description

This quiz covers the fundamentals of entropy in information theory, focusing on its definition and calculation involving discrete random variables. Participants will explore the concept of uncertainty and how it is quantified through the entropy function. Test your understanding of how probability mass functions relate to entropy.