Mutual information for a finite uniform hypothesis experiment

Let $M\ge 2$ be an integer, let $P_1,\dots ,P_M$ be probability measures on one measurable sample space, and let $J$ be uniform on $[M]=\{1,\dots ,M\}$. Conditional on $J=j$, let the observation $X$ have law $P_j$. Define the mixture law $\bar{P}:=\frac{1}{M}{\sum }_{j=1}^M{P}_j$ and the mutual information of this finite experiment by $I(J;X):=\frac{1}{M}{\sum }_{j=1}^{M}D(P_j\mathrm{\parallel }\bar{P})\mathrm{.}$ The set function $\bar{P}$ is a probability measure: for disjoint measurable sets $(A_m)$, countable additivity of the $P_j$ gives $\bar{P}\text{\hspace{0.17em}⁣}\left({\bigcup }_m{A}_m\right)=\frac{1}{M}{\sum }_{j=1}^M{\sum }_m{P}_j(A_m)\mathrm{.}$ The finite sum commutes with the increasing partial-sum limits of these nonnegative series, so $\frac{1}{M}{\sum }_{j=1}^M{\sum }_m{P}_j(A_m)={\sum }_m\frac{1}{M}{\sum }_{j=1}^M{P}_j(A_m)={\sum }_m\bar{P}(A_m),$ so $\bar{P}$ is countably additive; also $\bar{P}(\mathrm{\varnothing })=0$ and $\bar{P}(X)=1$. Here $D$ is the directional KL divergence of Kullback–Leibler divergence, and logarithms are natural. For every measurable $A$, $P_j(A)\le M\bar{P}(A)$, so $P_j\ll \bar{P}$. Let $r_j=d{P}_j\mathrm{/}d\bar{P}$ be a finite-valued nonnegative density representative provided by Kullback–Leibler divergence. If $\bar{P}(\{r_j>M\})>0$, then for some integer $q\ge 1$ the set $E_q:=\{r_j\ge M+1\mathrm{/}q\}$ has positive $\bar{P}$-measure. The change-of-measure identity Likelihood-ratio change of measure would give $P_j(E_q)={\int }_{E_q}{r}_{j}d\bar{P}\ge (M+1\mathrm{/}q)\bar{P}(E_q)>M\bar{P}(E_q),$ contradicting $P_j(E_q)\le M\bar{P}(E_q)$. Hence $r_j\le M$ almost everywhere. The densities also satisfy $M^{-1}{\sum }_{j=1}^M{r}_j=1$ $\bar{P}$-almost everywhere: for each measurable $A$, $\bar{P}(A)=\frac{1}{M}{\sum }_{j=1}^M{P}_j(A)={\int }_A\frac{1}{M}{\sum }_{j=1}^M{r}_{j}d\bar{P},$ so $M^{-1}{\sum }_j{r}_j$ and the constant function $1$ are both densities of $\bar{P}$ with respect to itself. They agree almost everywhere by A sigma-finite signed measure that is absolutely continuous with respect to a sigma-finite positive measure has a unique almost-everywhere density. For $0\le t\le M$, the positive part of $t\log t$ is at most $M\log M$, while its negative part is bounded by $1\mathrm{/}e$ in Kullback–Leibler divergence. Thus every term in the finite sum is well-defined and finite.

The uniform prior is part of this definition. A nonuniform prior requires its own weights in the mixture and in the KL average.