lec(9)source coding.pptx

Information Theory Source coding Definition A source code C for random variable X is a mapping from x to D* ( the set of finite length strings of symbols from a D-ary alphabet. Let C(x) denote the codeword corresponding to x. Let l(x) denote the length of c(x). Example F...

Information Theory Source coding Definition A source code C for random variable X is a mapping from x to D* ( the set of finite length strings of symbols from a D-ary alphabet. Let C(x) denote the codeword corresponding to x. Let l(x) denote the length of c(x). Example For example: c(read)=00, C(blue)=11 is a source code for X={red, blue} with alphabet D={0,1} l(C(blue))=2, l(C(red))=2 Average Codeword Length The expected length L(C) of source code C(X) for a random variable X with probability mass function p(x) is given by: L (C )   p ( x)l (C ( x )) x X Where l(x) is the length of codeword associated with x. Example Let X be a random variable with the following distribution and codeword assignment: P(X=1)=1/2 codeword C(1)=0 P(X=2)=1/4 codeword C(2)=10 P(X=3)=1/8 codeword C(3)=110 P(X=4)=1/8 codeword C(4)=111 Sol. L(C )   p( x)l (C ( x)) xX 1 1 1 1  *1  * 2  * 3  * 3 1.75bits 2 4 8 8 Efficient Code We now increasingly move stringent conditions for efficient codes: n Let X denote (x1,x2,…..,xn) 1. A code is said to be Non-singular if every element of the range of X maps into a different string in D*. That is : x  x  C ( x) C ( x' ) e.g x4 x3 x2 x1 X 10 01 1 0 C(X) Efficient Code Cont. Motivated by the necessity to send sequences of symbols X we define the extension of a code as follows: The extension C* of a code C is the mappings from finite length strings of X to finite length strings of D. Defined by C(x1 x2 ….xn) = C(x1) C(x2)….C(xn) Efficient Code Cont. 2. A code is called uniquely decodable if it’s extension is non singular. In other words : any encoded string in a uniquely decodable code has only one possible source string producing it. x4 x3 x2 x1 X 10 01 11 00 C(X) Efficient Code Cont. 3. A code is called a prefix code or an instantaneous code if no codeword is a prefix of any other codeword. Example: C(1)=0 C(2)=10 C(3)=110 C(3)=111 Efficient Code Cont. 4. Satisfy Kraft inequality: Code with codeword lengths l1 , l2 , …, lk satisfy kraft inequality if and only if: k  D  li 1 i1 Theorem The codeword length L of a prefix code is lower bounded by the entropy of source L H ( X ) Optimum code Defining an optimum code as one with minimum codeword length not exceed H(X)+1 L H ( X )  1 Theorem Given a variable X, the average codeword length for an optimal prefix code satisfies: H ( X ) L H ( X )  1 Example Let C is a code with symbols {x1,x2,x3,x4} C P(X) X 00 0.45 x1 01 0.25 x2 100 0.20 x3 1010 0.10 x4 Is this code C is optimal prefix Code? Sol. H(X)=1.815 L=2.4 H(X)+1=2.815 Code C is optimal prefix code