Information Theory

Sixth Assignment: due Mon. Oct. 29

I will add more later. I will also hand out a final version to you on Wednesday, since this is so hard to read.
  1. It can be shown that H_X(g(X,Y)) <= H_X(Y) .

    a. Give a simple example where the inequality is strict.

    b. Give conditions which guarantee equality.

  2. Find a really simple example where H({(f(X),g(Y))}) < H(X x Y) .
  3. Prove H_X(Y) + H_{X x Y}(Z) = H_X(Y x Z) by some means other than that used in class.
  4. Use the chain rule to show that if m < n, H(X_1,...,X_m) <= H(X_1,...,X_n) and determine when equality occurs.
  5. Prove directly that H_X(Z) >= H_{X x Y}(Z) . Interpret.

  6. Suppose that 75% of women are dark haired and 25% are blond. Suppose also that 70% of men are dark haired and 30% are blond.

    a. If all blond women marry dark haired men, how much information does a womans hair color give about her husband's?

    b. Answer the same question if all blond women marry blond men.

    c. Comment.

  7. (Ash, p. 25) Suppose that in a certain city, 3/4 of the male high-school students pass and 1/4 fail. Of those who pass, 10% own cars, while 50% of the failing students own cars. All of the car owning students belong to fraternities, while 40% of those who do not own cars but pass, as well as 40% of those who do not own cars but fail, belong to fraternities.

    a. How much information is conveyed about a student's academic standing by specifying whether or not he owns a car?

    b. How much information is conveyed about a student's academic standing by specifying whether or not he belongs to a fraternity?

    c. How much information is conveyed about a student's academic standing by specifying both whether he owns a car and whether he belongs to a fraternity?

    d. If a student is a member of a fraternity, how much information is conveyed about his academic standing by specifying whether or not he owns a car?

    e. If a students academic standing, car-owning status, and fraternity status are transmitted by three successive bits, how much information is conveyed by each bit?

  8. Use the maximum entropy principle to find the probabilities for the roll of a die in the following cases.

    a. The roll of 6 is twice as likely as the roll of a 1.

    b. Additionally, suppose that the average roll (after lots of trials) is 4.