lec(2)Introduction to -Information Theory(2) .pptx

Introduction to Information Theory Shannon Theory The original 1948 Shannon Theory contains: 1. Measurement of Information 2. Source Coding Theory 3. Channel Coding Theory Measurement of Information Shannon’s first question is “How to measure information...

Introduction to Information Theory Shannon Theory The original 1948 Shannon Theory contains: 1. Measurement of Information 2. Source Coding Theory 3. Channel Coding Theory Measurement of Information Shannon’s first question is “How to measure information in terms of bits?” = ? bits = ? bits All events are probabilistic! Using Probability Theory, Shannon showed that there is only one way to measure information in terms of number of bits: called the entropy function For example Tossing a dice: – Outcomes are 1,2,3,4,5,6 – Each occurs at probability 1/6 – Information provided by tossing a dice is  Wait! It is nonsense! The number 2.585-bits is not an integer!! What does you mean? Shannon’s First Source Coding Theorem Shannon showed: “To reliably store the information generated by some random source X, you need no more/less than, on the average, H(X) bits for each outcome.” Meaning: If I toss a dice 1,000,000 times and record values from each trial 1,3,4,6,2,5,2,4,5,2,4,5,6,1,…. In principle, I need 3 bits for storing each outcome as 3 bits covers 1-8. So I need 3,000,000 bits for storing the information. Using ASCII representation, computer needs 8 bits=1 byte for storing each outcome The resulting file has size 8,000,000 bits But Shannon said: You only need 2.585 bits for storing each outcome. So, the file can be compressed to yield size 2.585x1,000,000=2,585,000 bits Optimal Compression Ratio is: Let’s Do Some Test! File Size Compression Ratio No 8,000,000 bits 100% Compression Shannon 2,585,000 bits 32.31% Winzip 2,930,736 bits 36.63% WinRAR 2,859,336 bits 35.74%  The Winner is I had mathematically claimed my victory 50 years ago!