lec(8)mutual information.pptx

Information Theory Mutual Information Mutual Information Measure of the amount of information that one random variable contains about another random variable p ( x, y ) I ( X ; Y )   p( x, y ) log xX yY p( x) p( y ) Mutu...

Information Theory Mutual Information Mutual Information Measure of the amount of information that one random variable contains about another random variable p ( x, y ) I ( X ; Y )   p( x, y ) log xX yY p( x) p( y ) Mutual Information Reduction in the uncertainty of one random variable due to the knowledge of the other. Relationship between entropy and mutual information : I ( X ; Y ) H (Y )  H (Y | X ) Proof: p ( x, y ) I ( X ; Y )    p( x, y ) log xX yY p( x) p( y ) p( y | x)    p( x, y ) log x X yY p( y)    p( x, y )[log p( y | x)  log p( y )] x X yY    p( x, y ) log p ( y | x)    p( x, y) log p( y) x X yY x X yY    p( x) p( y | x) log p( y | x)   p( y) log p( y) x X yY yY  H (Y | X )  H (Y ) H (Y )  H (Y | X ) Mutual information & Chain Rule I ( X ; Y ) H ( X )  H ( X | Y ) use _ chain _ Rule : H ( X , Y ) H ( X )  H (Y | X )  H (Y | X ) H ( X , Y )  H ( X )  I ( X ; Y ) H (Y )  [ H ( X , y )  H ( X )] I ( X ; Y ) H ( X )  H (Y )  H ( X , Y ) I ( X ; X ) H ( X )  H ( X | X ) H ( X ) Vein diagram I(X;Y) is intersection of information in X with with information in Y. Example Let X represent blood type and Y represent chance for skin cancer. Compute H(X) and H(Y), H(X,Y) H(X| Y),H(Y|X) and I(X;Y). Probability mass function defined as bellow: P(Y) O AB B A X Y 1/4 1/32 1/32 1/16 1/8 Very low 1/4 1/32 1/32 1/8 1/16 low 1/4 1/16 1/16 1/16 1/16 medium 1/4 0 0 0 1/4 high 1 1/8 1/8 1/4 1/2 P(X) Sol. X:marginal {1/2,1/4,1/8,1/8} X:marginal {1/4,1/4,1/4,1/4} 1 H ( X )   p ( x ) log x X p( x) 1 1 1 1  log 2 2  log 2 4  log 2 8  log 2 8 2 4 8 8 1 1 3 3     1.75bits 2 2 8 8 Sol. Cont. 1 H (Y )  p ( y ) log yY p( y) 1 ( log 2 4) * 4 2bits 4 1 1 3 3     1.75bits 2 2 8 8 1 H ( X , Y )  p ( x, y ) log xX yY p ( x, y ) 1 1 1 1 ( log 2 8) * 2  ( log 2 16) * 6  ( log 2 32) * 4  log 2 4 8 16 32 4 6 6 5 1 27      3.375bits 8 4 8 2 8 Sol. Cont. I ( X ; Y ) H ( X )  H (Y )  H ( X , Y ) 1.75  2  3.375 0.375bits H ( X | Y ) H ( X )  I ( X ; Y ) 1.75  0.375 1.375bits H (Y | X ) H (Y )  I ( X ; Y ) 2  0.375 1.625bits