Cost-Constrained, Gaussian, and Feedback Channels — Examples

Write $q_x=W(1\mid x)\in (0,1)$ and $r={\sum }_x{p}_x{q}_x$. There is some $x_0$ with $p_{x_0}>0$, so $r\ge p_{x_0}{q}_{x_0}>0$ and $1-r\ge p_{x_0}(1-q_{x_0})>0$. The output entropy is $h_2(r)\le h_2(1\mathrm{/}2)=\ln 2$: concavity and symmetry in [F2] give $h_2(1\mathrm{/}2)\ge \frac{1}{2}{h}_2(r)+\frac{1}{2}{h}_2(1-r)=h_2(r)$. Every row has entropy $h_2(q_x)>0$, and at least one term has positive weight, so ${\sum }_x{p}_x{h}_2(q_x)>0$. By [F1], $I(p;W)=h_2(r)-{\sum }_x{p}_x{h}_2(q_x)<\ln 2$ for every input law.

For each $k$, choose the law assigning probability $1\mathrm{/}2$ to $2k$ and $1\mathrm{/}2$ to $2k-1$. Its output is uniform because the two row probabilities sum to one, while its conditional entropy is $h_2({ϵ}_k)$ by symmetry. Hence its mutual information is $\ln 2-h_2({ϵ}_k)$. Since ${ϵ}_k\downarrow 0$ and binary entropy is continuous with $h_2(0)=0$ by [F2]–[F3], these values tend to $\ln 2$.

Step 1.1 bounds every input law strictly below $\ln 2$, while step 1.2 gives a sequence of feasible values tending to $\ln 2$. Therefore the supremum is finite and equals $\ln 2$, but no input law attains it.

Direct multiplication gives $U^{T}U=I_2$ and $U^T\mathrm{diag}(1,4)U=\frac{1}{2}\left(\begin{array}{cc}5& 3\\ 3& 5\end{array}\right)=K$, hence $UK{U}^T=\mathrm{diag}(1,4)$, so $K$ is positive definite with eigenvalues $1,4$. The diagonalization argument of [F1], applying [F2] to the underlying independent standard-normal vector, gives that $UZ$ has independent coordinates with variances $1,4$. For every input, $\mathrm{\parallel }UX{\mathrm{\parallel }}_2^2=X^T{U}^{T}UX=\mathrm{\parallel }X{\mathrm{\parallel }}_2^2$ and $U(X+Z)=UX+UZ$.

Since $P=2<4-1=3$, the water level is $T=1+P=3$: the first branch receives $p_1=T-1=2$ and the second receives $p_2=0$. Substitution in [F3] gives capacity $\frac{1}{2}\ln (1+2\mathrm{/}1)+\frac{1}{2}\ln (1+0\mathrm{/}4)=\frac{1}{2}\ln 3$. The orthogonal code correspondence from step 1.1 transfers this value to the original correlated channel.

At $P=0$, the only feasible allocation is $(0,0)$. If $0<P\le 3$, choosing $T=1+P\le 4$ makes $p_1=T-1=P$ and $p_2=(T-4{)}^{+}=0$, whose sum is $P$. If $P\ge 3$, choosing $T=(P+5)\mathrm{/}2\ge 4$ gives $p_1=(P+3)\mathrm{/}2$, $p_2=(P-3)\mathrm{/}2$, and their sum is $P$. At $P=3$ the formulas agree. In each positive-power regime [F1] therefore confirms the displayed water level and nonnegative powers.

Substituting the first allocation into the rate formula [F2] gives $\frac{1}{2}\ln (1+P)$; substituting the second gives $\frac{1}{2}\ln (T)+\frac{1}{2}\ln (T\mathrm{/}4)=\ln (T\mathrm{/}2)=\ln ((P+5)\mathrm{/}4)$. At the common endpoint $P=3$, both are $\ln 2$.

An input law is feasible exactly when $0\le p\le \min \{\mathrm{\Gamma },1\}$, and [F2] makes its mutual information $h_2(p)$. If $0\le p\le q\le 1\mathrm{/}2$ and $p<1\mathrm{/}2$, put $\theta =(1-p-q)\mathrm{/}(1-2p)\in [1\mathrm{/}2,1]$. Then $q=\theta p+(1-\theta )(1-p)$; concavity and symmetry in [F3] imply $h_2(q)\ge \theta h_2(p)+(1-\theta )h_2(1-p)=h_2(p)$. Thus $h_2$ is nondecreasing on $[0,1\mathrm{/}2]$.

For any $p\in [0,1]$, concavity and symmetry give $h_2(1\mathrm{/}2)\ge \frac{1}{2}{h}_2(p)+\frac{1}{2}{h}_2(1-p)=h_2(p)$. Therefore the maximum over the feasible interval occurs at $p=\min \{\mathrm{\Gamma },1\mathrm{/}2\}$ and equals the displayed formula. Substitution yields $C(0)=0$, $C(1\mathrm{/}2)=\ln 2$, and $C(1)=\ln 2$.

The feedback theorem in [F4] makes the feedback capacity equal to the same ordinary constrained capacity. This identifies the values for all three requested budgets.