Information Theory: Conditional Entropy

Conditional Entropy

• Conditional Entropy H(X|Y) quantifies the uncertainty of random variable X given that another variable Y is known.
• Mathematically defined as:
  • H(X|Y) = Σ p(y) H(X|Y=yi) for all y in Y
  • Equivalently expressed as H(X|Y) = -Σ Σ p(x, y) log p(x|y) for all x in X and y in Y.
• Specific case for conditional entropy of X given a particular outcome of Y (yi):
  • H(X|Y=yi) = -Σ p(xj|yi) log p(xj|yi).

Chain Rule

• The chain rule for joint entropy illustrates the relationship between individual entropies and conditional uncertainties:
  • H(X, Y) = H(X) + H(Y|X).
• A symmetrical expression also exists:
  • H(X, Y) = H(Y) + H(X|Y).
• This rule clarifies that joint uncertainty can be decomposed into the uncertainty of individual variables and their dependencies.

Example Calculations

• Two variables X (weather: sunny or rainy) and Y (temperature: above or below 70 degrees) are utilized to compute conditional entropies.
• Marginal probabilities derived from the example are:
  • P(X): {3/4 rainy, 1/4 sunny}
  • P(Y): {3/4 above 70, 1/4 below 70}
• Conditional entropy H(X|Y) calculated at 0.207 hartlys.

Joint Probability Mass Function

• A second example presents a joint probability mass function for variables X and Y with several discrete outcomes.
• The joint distribution is defined with probabilities summing to 1 over all possible X and Y combinations.
• Calculated entropies:
  • H(X|Y) yields a result of approximately 1.284 hartlys.
  • H(Y|X) results in approximately 0.324 hartlys.

Assignments

• Tasks include calculations of H(X), H(Y), and H(X,Y) in hartlys based on joint distributions and marginal probabilities provided from previous sections.