information_theory

Given a noisy typewriter (each input produces one of its neighboring outputs, e.g. \(C \to \{B, C, D\}\)), what is the capacity of the channel? One way is to compute the capacity directly : \[ C = \max_{P_X} I(X;Y) = \max_{P_X} \bigl( H(Y) - H(Y\mid X) \bigr). \]

But there is a cute trick:

  • Choose an input set such that the outputs never overlap — for example: \(\{A, D, G, \ldots\},\) and assign each of these usable symbols probability \(1/9\). Because this removes all ambiguity at the output, the channel becomes effectively noise-free on this reduced alphabet. Thus \(I(X;Y) = H(Y) = \log_2 9\)