LemmaStatement: AI-adaptedProof: AI-adaptedPipeline-generated

How statement and proof provenance work

The first chip identifies the source of the statement or construction; the second identifies the source of its local proof or verification.

Literature-sourced: the exact statement appears in a cited source; only wording and notation differ.
AI-adapted: a semantically identical restatement of literature-sourced material, modulo indexing, notation, and boundary cases adopted by the library.
AI-generated: a genuinely novel statement formulated by AI, with no source for the claim itself.

These labels describe origin, not correctness: citations and verification chips remain separate evidence.

Fano bound for finite messages and density observations

Statement

Let $M$ be uniform on a finite message set $M$ of size $m \geq 2$ , let $S$ take values in a finite set $S$ , and let $Y$ take values in $R^{d}$ ; a complex observation space $C^{k}$ is identified with $R^{2 k}$ . Suppose $Y ∣ (M, S)$ has a Lebesgue density. For every measurable estimator $\hat{M} = ϕ (Y, S)$ with $ϕ : R^{d} \times S \to M$ and error probability $ϵ$ , define $H (M ∣ Y, S) : = \int H (p_{M ∣ y, s}) d P_{Y, S} (y, s)$ using its finite posterior vector. Then $H (M ∣ Y, S) \leq h_{b} (ϵ) + ϵ \ln (m - 1) \leq \ln 2 + ϵ \ln m,$ where $h_{b} (t) = - t \ln t - (1 - t) \ln (1 - t)$ and $0 \ln 0 = 0$ . For $m = 1$ , the conditional entropy is zero.

Facts & Assumptions

Given: A finite uniform message, finite side information, conditional observation densities and a measurable estimator.

[F1]

Finite distributions, entropy, divergence, and stochastic matrices: $H (p) = - \sum_{i} p_{i} \ln p_{i}$ , with $0 \ln 0 = 0$ .

[F2]

Finite KL divergence is nonnegative with equality exactly at equal laws: finite relative entropy is nonnegative; in particular a $k$ -point law has entropy at most $\ln k$ by comparison with the uniform law.

Proof

1.1givenF1algebra

If $π_{s} (m) = \Pr [M = m ∣ S = s]$ and $f_{m, s}$ is the conditional density of $Y$ , then $f_{s} (y) = \sum_{m} π_{s} (m) f_{m, s} (y)$ and $p_{M ∣ y, s} (m) = π_{s} (m) f_{m, s} (y) / f_{s} (y)$ whenever $f_{s} (y) > 0$ . This is a finite posterior probability vector, so [F1] makes its entropy well-defined and bounded by $\ln m$ .

2.1givenF1step 1.1algebra

Put $E = 1 {\hat{M} \neq M}$ . For each fixed $(y, s)$ , $E$ is a function of $M$ ; expanding the finite entropy sum first over $M$ and then over $E$ gives $H (M ∣ y, s) = H (E ∣ y, s) + H (M ∣ E, y, s)$ .

3.1F1F2step 2.1algebracases

Let $q (y, s) = \Pr [E = 1 ∣ Y = y, S = s]$ . If $0 < ϵ < 1$ , [F2] applied to the two-point laws $(q, 1 - q)$ and $(ϵ, 1 - ϵ)$ gives $q \ln (q / ϵ) + (1 - q) \ln ((1 - q) / (1 - ϵ)) \geq 0$ . Rearranging and averaging, using $E q = ϵ$ , yields $E h_{b} (q) \leq h_{b} (ϵ)$ . If $ϵ = 0$ or $1$ , then $q = ϵ$ almost surely and both binary entropies are zero.

3.2F2step 2.1

On $E = 0$ , $M = ϕ (Y, S)$ and so its conditional entropy is zero; on $E = 1$ and fixed $(Y, S)$ , at most $m - 1$ message values remain, so [F2] bounds that entropy by $\ln (m - 1)$ . Therefore $H (M ∣ E, Y, S) \leq ϵ \ln (m - 1)$ .

4.1F1F2step 2.1step 3.1step 3.2algebracases∎

Averaging the identity of step 2.1 and applying steps 3.1 and 3.2 proves the sharp bound. Since $h_{b} (ϵ) \leq \ln 2$ and $\ln (m - 1) \leq \ln m$ , the weaker displayed bound follows. If $m = 1$ , $M$ is constant.

Depends on

Used by

Dependency tree · two levels

4 results within two dependency steps of this one, each drawn at its shortest distance from it. An arrow runs from a result to what uses it, so the chart reads left to right and ends at this result, which carries a heavier outline. Every node is a link to that result. Click elsewhere on the chart to enlarge it.

Sources

Thomas M. Cover and Joy A. Thomas, Elements of Information Theory, 2nd ed., Chapter 2, §2.10 (standard reference, not scraped)
Amos Lapidoth, A Foundation in Digital Communication, Chapter 3, Fano's inequality (standard reference, not scraped)