Expectation-maximization

Inputs. Supply a measurable parameter space $(\mathrm{\Theta },\mathcal{T})$, sigma-finite measure spaces $(\mathsf{X},\mathcal{A},\nu )$ and $(\mathsf{Z},\mathcal{C},\mu )$, and a jointly measurable complete-data density $p(\theta ,x,z)$ relative to $\nu \otimes \mu$ that integrates to one on each parameter fiber, as in Latent-variable joint and observed-data factorization. This joint measurability is supplied explicitly and is not inferred from separate likelihood-density versions (Likelihood densities). Also supply fixed observed data $x$, an initial value ${\theta }_0$, a mode exact or generalized, and a candidate selector that returns either a candidate or unavailable in finite time at each call. Require $0<m_{{\theta }_0}(x)<\mathrm{\infty }$ and $\int q_{{\theta }_0}^x(z)\mathrm{\mid }\log p({\theta }_0,x,z)\mathrm{\mid }\mu (dz)<\mathrm{\infty }\mathrm{.}$

Write $m_{\theta }(x):={\int }_{\mathsf{Z}}p(\theta ,x,z)\mu (dz),q_{\theta }^x(z):=\frac{p(\theta ,x,z)}{m_{\theta }(x)}$ only when $0<m_{\theta }(x)<\mathrm{\infty }$. By The observed density is the latent marginal under Tonelli, $m_{\theta }$ is the observed-data density and $q_{\theta }^x$ is a probability density on every such fiber. The observed log likelihood is ${\mathrm{\ell }}_x(\theta ):=\log m_{\theta }(x)$ on those fibers.

At each valid iterate ${\theta }_t$, set $q_t:=q_{{\theta }_t}^x$ and define $Q(\theta \mid {\theta }_t):={\int }_{\mathsf{Z}}\log p(\theta ,x,z)q_t(z)\mu (dz)$ only for candidates where this integral is finite. Use $\log 0=-\mathrm{\infty }$ and set $q_t\mathrm{\mid }\log p\mathrm{\mid }=0$ wherever $q_t=0$. The Q integral itself is choice-free. To test finite KL when $q_t\mu \ll q_{\theta }^x\mu$, assume AC as required by The Axiom of Choice and Kullback–Leibler divergence; when absolute continuity fails, $D=+\mathrm{\infty }$ by that definition without AC. An admissible candidate $\theta$ also satisfies $0<m_{\theta }(x)<\mathrm{\infty }$, $q_t\mu \ll q_{\theta }^x\mu$, and $D(q_t\mu \mathrm{\parallel }{q}_{\theta }^x\mu )<\mathrm{\infty },$ where the direction of $D$ is that of Kullback–Leibler divergence. Require $\int q_t\mathrm{\mid }\log p(\theta ,x,\cdot )\mathrm{\mid }d\mu <\mathrm{\infty }$ for each candidate and $\int q_t\mathrm{\mid }\log p({\theta }_t,x,\cdot )\mathrm{\mid }d\mu <\mathrm{\infty }$ for the current iterate.

Output. Emit the iterate sequence beginning with ${\theta }_0$. If a stopping condition occurs after $T$ successful updates, return the finite list $({\theta }_0,\dots ,{\theta }_T)$ and a status recording the first unavailable condition or selector result; do not append a failed candidate. If no stopping condition occurs, the emitted sequence is infinite and there is no finite stopping status.

Procedure. For $t=0,1,2,\dots$, check that $0<m_{{\theta }_t}(x)<\mathrm{\infty }$; if not, return the current list with status stopped: invalid current marginal. Form $q_t$ and $Q(\cdot \mid {\theta }_t)$ on admissible candidates. If $Q({\theta }_t\mid {\theta }_t)$ is not finite, return with status stopped: current Q is not finite. Call the supplied selector. In exact mode, its candidate set must consist of admissible candidates and contain ${\theta }_t$; it returns an attained maximizer of $Q$ on that set, or unavailable if no attained maximum is available. In generalized mode, it returns any admissible ${\theta }_{t+1}$ with $Q({\theta }_{t+1}\mid {\theta }_t)\ge Q({\theta }_t\mid {\theta }_t)$, or unavailable if no such update is supplied. If the selector returns unavailable, return with status stopped: no required update supplied; otherwise append its candidate and continue. The procedure asserts no convergence or global-optimality property and makes no assertion for a candidate that loses current latent support.