Self-mutual-information entropy on countable and non-atomic laws

For each positive integer n, there are at most n atoms with mass at least $1\mathrm{/}n$, since any n+1 distinct such atoms would have total mass greater than one. For each atom of mass $p>0$, choose n with $1\mathrm{/}n<p$ by [F5]; thus S is a countable union of finite sets and is countable.

Suppose $P(S)=1$ and enumerate the distinct atoms as $(x_i)$ with masses $p_i>0$. By [F2], each singleton $(x_i,x_i)$ is measurable, so the function $r$ that equals $1\mathrm{/}{p}_i$ at $(x_i,x_i)$ and zero elsewhere is measurable. For each finite $N$, let $r_N={\sum }_{i\le N}{p}_i^{-1}{\mathbf{1}}_{\{(x_i,x_i)\}}$; these functions increase to $r$. The rectangle formula gives $(P\otimes P)\{(x_i,x_i)\}=p_i^2$. Hence for every measurable $C\subseteq E\times E$, [F6] gives ${\int }_{C}rd(P\otimes P)={\lim }_N{\sum }_{i\le N:(x_i,x_i)\in C}\frac{1}{p_i}{p}_i^2={\sum }_{i:(x_i,x_i)\in C}{p}_i=P_{XX}(C)\mathrm{.}$ Thus $r$ is the Radon–Nikodym density. Since $p_i\le 1$, $\log r\ge 0$ on the $P_{XX}$-full set of diagonal atoms; the partial sums $g_N={\sum }_{i\le N}\ln (1\mathrm{/}{p}_i){\mathbf{1}}_{\{(x_i,x_i)\}}$ increase to $\log r$ there. Another application of [F6] gives the KL integral ${\sum }_i{p}_i\ln (1\mathrm{/}{p}_i)$, including an infinite sum.

Suppose $P(S)<1$. The set $C=\mathrm{\Delta }\setminus (S\times S)$ is measurable, where $\mathrm{\Delta }=\{(x,x):x\in E\}$ by [F2]. The diagonal joint law assigns C mass $P(E\setminus S)>0$. Its product-measure section at $x\in S$ is empty; at $x\mathrm{\notin }S$ it is $\{x\}$, whose P-mass is zero. By [F3], $(P\otimes P)(C)=0$. Hence $P_{XX}\ll ̸P\otimes P$ and [F1] gives $H(X)=+\mathrm{\infty }$.

The two cases $P(S)=1$ and $P(S)<1$ exhaust all probability laws. Step 1.2 gives the nonnegative countable sum, which agrees on every finite subset with the finite Shannon formula [F4]; step 2.1 gives infinite divergence when there is positive non-atomic mass.