Finite conditional mutual information

Let $p_{XYZ}$ be a joint probability vector on finite alphabets and write $p_Z(z)$ for its marginal. It is the law of finite coordinate random elements on the canonical finite probability space from Finite probability spaces, outcome weights, events, and event probabilities and Random elements and real random variables. For each $z$ with $p_Z(z)>0$, the conditional law $p_{XY\mathrm{\mid }z}$ and its marginals $p_{X\mathrm{\mid }z},p_{Y\mathrm{\mid }z}$ are probability vectors by Marginals and positive-mass conditionals of a finite joint law are probability vectors. Define conditional mutual information in nats by

$I(X;Y\mathrm{\mid }Z):={\sum }_{z:p_Z(z)>0}{p}_Z(z)D(p_{XY\mathrm{\mid }z}\mathrm{\parallel }{p}_{X\mathrm{\mid }z}\otimes p_{Y\mathrm{\mid }z}),$

using the finite KL convention of Finite distributions, entropy, divergence, and stochastic matrices, where $(p_{X\mathrm{\mid }z}\otimes p_{Y\mathrm{\mid }z})(x,y):=p_{X\mathrm{\mid }z}(x)p_{Y\mathrm{\mid }z}(y)$. A positive conditional joint mass forces both conditional marginals to be positive, so each displayed KL term is finite. Zero-mass conditioning values contribute no term. The base-$b$ version replaces $\ln$ by ${\log }_b$ inside the conditional KL sums.