lec(3)Introduction to -Information Theory(3) (1).pptx

Introduction to Information Theory (3) Follow-up Story Later in 1952, David Huffman, while was a graduate student in MIT, presented a systematic method to achieve the optimal (1925-1999) compression ratio guaranteed by Shannon. The coding technique is the...

Introduction to Information Theory (3) Follow-up Story Later in 1952, David Huffman, while was a graduate student in MIT, presented a systematic method to achieve the optimal (1925-1999) compression ratio guaranteed by Shannon. The coding technique is therefore called “Huffman code” in honor of his achievement. Huffman codes are used in nearly every application that involves the compression and transmission of digital data, such as fax machines, modems, computer networks, and high-definition television (HDTV), to name a few.  So far… but how about? How? Done Source Channel Data Encoding Encoding Channel Source Channel User Decoding Decoding The Simplest Case: Computer Network Communications over computer network, ex. Internet The major channel impairment herein is Packet Loss Binary Erasure Channel Impairment like “packet loss” can be viewed as Erasures. Data that are erased mean they are lost during transmission… 1-p 0 0 p Erasure p 1 1 1-p p is the packet loss rate in this network  Once a binary symbol is erased, it can not be recovered… Ex: Say, Alice sends 0,1,0,1,0,0 to Bob But the network was so poor that Bob only received 0,?,0,?,0,0 So, Bob asked Alice to send again Only this time he received 0,?,?,1,0,0 and Bob goes CRAZY! What can Alice do? What if Alice sends 0000,1111,0000,1111,0000,0000 Repeating each transmission four times! What Good Can This Serve? Now Alice sends 0000,1111,0000,1111,0000,0000 The only cases Bob can not read Alice are for example ????,1111,0000,1111,0000,0000 all the four symbols are erased. But this happens at probability p4  Thus if the original network has packet loss rate p=0.25, by repeating each symbol 4 times, the resulting system has packet loss rate p4=0.00390625 But if the data rate in the original network is 8M bits per second 8Mbps Alice p=0.25 Bob With repetition, Alice can only transmit at 2 M bps 8Mbps 2 Mbps X4 Alice Bob p=0.00390625 Shannon challenged: Is repetition the best Alice can do? And he thinks again… Shannon’s Channel Coding Theorem Shannon answered: “Give me a channel and I can compute a quantity called capacity, C for that channel. Then reliable communication is possible only if your data rate stays below C.” ? ? ? ? What does Shannon mean?  Shannon means In this example: 8Mbps p=0.25 Alice Bob He calculated the channel capacity C=1-p=0.75 And there exists coding scheme such that: 8Mbps ? Alice 8 x (1-p) p=0 Bob =6 Mbps Unfortunately… I do not know exactly HOW? Neither do we… But With 50 Years of Hard Work We have discovered a lot of good codes: – Hamming codes – Convolutional codes, – Concatenated codes, – Low density parity check (LDPC) codes – Reed-Muller codes – Reed-Solomon codes, – BCH codes, – Finite Geometry codes, – Cyclic codes, – Golay codes, – Goppa codes – Algebraic Geometry codes, – Turbo codes – Zig-Zag codes, – Accumulate codes and Product-accumulate codes, – … We now come very close to the dream Shannon had 50 years ago! ☺ Nowadays… Source Coding Theorem has applied to Image Audio/Video MPEG Compression Compression Data Audio Compression MP3 Compression Channel Coding Theorem has applied to VCD/DVD – Reed-Solomon Codes Wireless Communication – Convolutional Codes Optical Communication – Reed-Solomon Codes Computer Network – LT codes, Raptor Codes Space Communication And Information Theory has Applied to All kinds of Communications, Stock Market, Economics Game Theory and Gambling, Quantum Physics, Cryptography, Biology and Genetics, and many more…