Measure-theoretic mutual information

Let $(E,\mathcal{E})$ and $(F,\mathcal{F})$ be measurable spaces and let $P$ be a probability measure on $(E\times F,\mathcal{E}\otimes \mathcal{F})$. Its marginals are $P_E(A)=P(A\times F)$ and $P_F(B)=P(E\times B)$. The product probability $P_E\otimes P_F$ exists by the finite product-measure theorem. For the absolutely continuous branch $P\ll P_E\otimes P_F$, assume AC as required by Kullback–Leibler divergence; the non-absolutely-continuous branch below remains $+\mathrm{\infty }$. Define the measure-theoretic mutual information of the joint law $P$ by $I(P):=D(P\mathrm{\parallel }{P}_E\otimes P_F)\in [0,+\mathrm{\infty }]\mathrm{.}$ For random elements $X,Y$ with joint law $P_{XY}$, write $I(X;Y):=I(P_{XY})$. The logarithm convention is the natural logarithm, so finite values are in nats as fixed by Information units and logarithm-base convention. The divergence is directional in its two law arguments and is $+\mathrm{\infty }$ when $P_{XY}\ll ̸P_X\otimes P_Y$.

This definition uses only the joint law, not a choice of density or a difference of entropy quantities.