Negative expected finite masked-diffusion ELBO is weighted cross-entropy

Assume the Axiom of Choice. Let $V$ be finite and nonempty, $m\mathrm{\notin }V$, $L,N\ge 1$, $1={\alpha }_0>\cdots >{\alpha }_N=0$, and let $X\sim \mu$ for a probability vector $\mu$ on $V^L$. Put $S=(V\cup \{m\}{)}^L$. Conditional on $X=x$, let $q_x$ be the finite forward masking path law started at $Z_0=x$: at step $j$, each clean coordinate survives with probability ${\alpha }_j\mathrm{/}{\alpha }_{j-1}$ and otherwise becomes the absorbing mask $m$, independently across coordinates. For every position $i$, reverse step $j$, state $z\in S$ with $z_i=m$, and token $v\in V$, assume $h_{i,j}(z)(v)>0$, where each $h_{i,j}(z)$ is a probability vector on $V$. Put $c_j=({\alpha }_{j-1}-{\alpha }_j)\mathrm{/}(1-{\alpha }_j)$ and $d_j=(1-{\alpha }_{j-1})\mathrm{/}(1-{\alpha }_j)$. Define the reverse kernel $P_j$ by copying every unmasked coordinate and, independently across masked positions conditional on $z_j$, assigning mass $d_j$ to $m$ and mass $c_j{h}_{i,j}(z_j)(v)$ to each $v\in V$. Use the point mass at $m^L$ as terminal prior and the decoder $\mathrm{Dec}(x\mid z_0)={\mathbf{1}}_{\{x=z_0\}}$ when $z_0\in V^L$, while $\mathrm{Dec}(x\mid z_0)=\mathrm{\mid }V{\mathrm{\mid }}^{-L}$ otherwise. Define $p(x,z_0,\dots ,z_N)={\mathbf{1}}_{\{z_N=m^L\}}({\prod }_{j=1}^N{P}_j(z_j,z_{j-1}))\mathrm{Dec}(x\mid z_0),p(x)={\sum }_{z_0,\dots ,z_N}p(x,z_0,\dots ,z_N)\mathrm{.}$ For each $x\in V^L$, let ${\pi }_x(z)=p(x,z)\mathrm{/}p(x)$ whenever $p(x)>0$ and define the finite ELBO $\mathcal{E}(x)={\sum }_{z:q_x(z)>0}{q}_x(z)\log (p(x,z)\mathrm{/}{q}_x(z))$. Here admissibility means $q_x$ is supported where $p(x,\cdot )>0$ and the finite weighted logarithm sums required for the ELBO are finite. Let $P_{j,i}^{\mathrm{m}\mathrm{a}\mathrm{s}\mathrm{k}}$ be the conditional law of $(Z_j,X_i)$ given $Z_{j,i}=m$, and let $R_{j,i}=\mathbb{E}[-\log h_{i,j}(Z_j)(X_i)\mid Z_{j,i}=m]$, with $-\log 0=+\mathrm{\infty }$. For probability vectors $u,v$ on a finite set, write $D_{\mathrm{K}\mathrm{L}}(u\mathrm{\parallel }v)={\sum }_{a:u(a)>0}u(a)\log (u(a)\mathrm{/}v(a))$, with zero source terms set to zero and positive source mass against zero reference mass set to $+\mathrm{\infty }$. Then $p(x)>0$, $q_x$ is admissible, and for every $x$, $\log p(x)=\mathcal{E}(x)+D_{\mathrm{K}\mathrm{L}}(q_x\mathrm{\parallel }{\pi }_x),\mathcal{E}(x)\le \log p(x)\mathrm{.}$ Moreover, averaging over $X\sim \mu$ gives $\mathbb{E}[-\mathcal{E}(X)]={\sum }_{j=1}^N{\sum }_{i=1}^L({\alpha }_{j-1}-{\alpha }_j)R_{j,i}\mathrm{.}$ The reconstruction term and terminal-prior KL vanish here because the endpoints are exactly clean and all-masked; with approximate endpoints, the ordinary ELBO retains those terms and this corollary does not discard them.

Finite masked-diffusion generative model: defines $P_j$, the terminal point mass, decoder $D$, and finite joint path mass.

Forward path law is admissible for the finite masked model: full support of $h$ makes $p(x)>0$ and the forward path law admissible.

Masked-diffusion KL reduces to weighted cross-entropy: each averaged finite reverse-KL contribution equals the weighted conditional masked-token population risk.

Finite masking path law factors backward by its posterior rows: the positive forward path law factors in reverse order by its one-step posterior rows.

Evidence lower bound is log evidence minus reverse KL: under AC and admissibility, log evidence is ELBO plus the reverse KL and therefore bounds the ELBO above.