Latent-variable EM and likelihood ascent

Inputs. Supply a measurable parameter space $(\mathrm{\Theta },\mathcal{T})$, sigma-finite measure spaces $(\mathsf{X},\mathcal{A},\nu )$ and $(\mathsf{Z},\mathcal{C},\mu )$, and a jointly measurable complete-data density $p(\theta ,x,z)$ relative to $\nu \otimes \mu$ that integrates to one on each parameter fiber, as in Latent-variable joint and observed-data factorization. This joint measurability is supplied explicitly and is not inferred from separate likelihood-density versions (Likelihood densities). Also supply fixed observed data $x$, an initial value ${\theta }_0$, a mode exact or generalized, and a candidate selector that returns either a candidate or unavailable in finite time at each call. Require $0<m_{{\theta }_0}(x)<\mathrm{\infty }$ and $\int q_{{\theta }_0}^x(z)\mathrm{\mid }\log p({\theta }_0,x,z)\mathrm{\mid }\mu (dz)<\mathrm{\infty }\mathrm{.}$

Write $m_{\theta }(x):={\int }_{\mathsf{Z}}p(\theta ,x,z)\mu (dz),q_{\theta }^x(z):=\frac{p(\theta ,x,z)}{m_{\theta }(x)}$ only when $0<m_{\theta }(x)<\mathrm{\infty }$. By The observed density is the latent marginal under Tonelli, $m_{\theta }$ is the observed-data density and $q_{\theta }^x$ is a probability density on every such fiber. The observed log likelihood is ${\mathrm{\ell }}_x(\theta ):=\log m_{\theta }(x)$ on those fibers.

At each valid iterate ${\theta }_t$, set $q_t:=q_{{\theta }_t}^x$ and define $Q(\theta \mid {\theta }_t):={\int }_{\mathsf{Z}}\log p(\theta ,x,z)q_t(z)\mu (dz)$ only for candidates where this integral is finite. Use $\log 0=-\mathrm{\infty }$ and set $q_t\mathrm{\mid }\log p\mathrm{\mid }=0$ wherever $q_t=0$. The Q integral itself is choice-free. To test finite KL when $q_t\mu \ll q_{\theta }^x\mu$, assume AC as required by The Axiom of Choice and Kullback–Leibler divergence; when absolute continuity fails, $D=+\mathrm{\infty }$ by that definition without AC. An admissible candidate $\theta$ also satisfies $0<m_{\theta }(x)<\mathrm{\infty }$, $q_t\mu \ll q_{\theta }^x\mu$, and $D(q_t\mu \mathrm{\parallel }{q}_{\theta }^x\mu )<\mathrm{\infty },$ where the direction of $D$ is that of Kullback–Leibler divergence. Require $\int q_t\mathrm{\mid }\log p(\theta ,x,\cdot )\mathrm{\mid }d\mu <\mathrm{\infty }$ for each candidate and $\int q_t\mathrm{\mid }\log p({\theta }_t,x,\cdot )\mathrm{\mid }d\mu <\mathrm{\infty }$ for the current iterate.

Output. Emit the iterate sequence beginning with ${\theta }_0$. If a stopping condition occurs after $T$ successful updates, return the finite list $({\theta }_0,\dots ,{\theta }_T)$ and a status recording the first unavailable condition or selector result; do not append a failed candidate. If no stopping condition occurs, the emitted sequence is infinite and there is no finite stopping status.

Procedure. For $t=0,1,2,\dots$, check that $0<m_{{\theta }_t}(x)<\mathrm{\infty }$; if not, return the current list with status stopped: invalid current marginal. Form $q_t$ and $Q(\cdot \mid {\theta }_t)$ on admissible candidates. If $Q({\theta }_t\mid {\theta }_t)$ is not finite, return with status stopped: current Q is not finite. Call the supplied selector. In exact mode, its candidate set must consist of admissible candidates and contain ${\theta }_t$; it returns an attained maximizer of $Q$ on that set, or unavailable if no attained maximum is available. In generalized mode, it returns any admissible ${\theta }_{t+1}$ with $Q({\theta }_{t+1}\mid {\theta }_t)\ge Q({\theta }_t\mid {\theta }_t)$, or unavailable if no such update is supplied. If the selector returns unavailable, return with status stopped: no required update supplied; otherwise append its candidate and continue. The procedure asserts no convergence or global-optimality property and makes no assertion for a candidate that loses current latent support.

Assume the Axiom of Choice. Let $p(\theta ,x,z)$ be a jointly measurable normalized complete-data density as in Latent-variable joint and observed-data factorization, and fix observed data $x$. Put $m_{\theta }(x)=\int p(\theta ,x,z)\mu (dz)$. Let ${\theta }_t$ and $\theta$ satisfy $0<m_{{\theta }_t}(x),m_{\theta }(x)<\mathrm{\infty }$. Define $q_t(z)=p({\theta }_t,x,z)\mathrm{/}{m}_{{\theta }_t}(x)$ and $q_{\theta }^x(z)=p(\theta ,x,z)\mathrm{/}{m}_{\theta }(x)$. Assume $q_t\mu \ll q_{\theta }^x\mu ,\int q_t(z)(\mathrm{\mid }\log p({\theta }_t,x,z)\mathrm{\mid }+\mathrm{\mid }\log p(\theta ,x,z)\mathrm{\mid })\mu (dz)<\mathrm{\infty },$ and $D(q_t\mu \mathrm{\parallel }{q}_{\theta }^x\mu )<\mathrm{\infty }$. Use $\log 0=-\mathrm{\infty }$ and set $q_t\mathrm{\mid }\log p\mathrm{\mid }=0$ wherever $q_t=0$. Then, with $Q(\theta \mid {\theta }_t)=\int q_t(z)\log p(\theta ,x,z)\mu (dz)$, ${\mathrm{\ell }}_x(\theta )-{\mathrm{\ell }}_x({\theta }_t)=Q(\theta \mid {\theta }_t)-Q({\theta }_t\mid {\theta }_t)+D(q_t\mu \mathrm{\parallel }{q}_{\theta }^x\mu )\mathrm{.}$ Consequently, any admissible update with $Q(\theta \mid {\theta }_t)\ge Q({\theta }_t\mid {\theta }_t)$ does not decrease ${\mathrm{\ell }}_x$. An exact M-step has this property when its maximum is attained over an admissible candidate set containing ${\theta }_t$. No equality-of-parameter conclusion is asserted when the KL divergence is zero.

Assume the Axiom of Choice and the model, fiber, support, integrability, and finite-KL hypotheses of EM does not decrease the observed log likelihood. If an admissible candidate ${\theta }_{t+1}$ satisfies $Q({\theta }_{t+1}\mid {\theta }_t)\ge Q({\theta }_t\mid {\theta }_t),$ then ${\mathrm{\ell }}_x({\theta }_{t+1})\ge {\mathrm{\ell }}_x({\theta }_t)$. The Q maximizer need not be attained for this generalized-EM conclusion.

Under the hypotheses of EM does not decrease the observed log likelihood and Generalized EM does not decrease the observed log likelihood under Q-ascent, including AC, these are one-step inequalities for a fixed data fiber. They do not prove that the likelihood sequence is bounded, that parameter iterates converge, that a limit point is stationary, or that a stationary point is globally optimal. The exact M-step also requires an attained Q maximum; ascent itself does not supply one. These are limits of the stated ascent result, not additional convergence theorems.

The examples An EM likelihood with two strict local maxima ↗ and Gaussian mixture likelihood is unbounded under covariance collapse ↗ exhibit tied maxima and an unbounded observed likelihood.