Latent-variable EM and likelihood ascent — Examples

Each $f_j$ integrates to one, so $p_{\pi }$ is a probability density on $\mathsf{X}\times \mathsf{Z}$ because ${\sum }_j{\pi }_j=1$. At $x=1\mathrm{/}4$, $f_1(x)=2$ and $f_2(x)=1$, hence the observed density is $m_{\pi }(x)={\sum }_j{\pi }_j{f}_j(x)=4\mathrm{/}3$.

The latent conditional weights are $q_{\pi }(j\mid x)={\pi }_j{f}_j(x)\mathrm{/}{m}_{\pi }(x)$ by [F2] and [F3], giving $q_{\pi }(1\mid x)=(2\mathrm{/}3)\mathrm{/}(4\mathrm{/}3)=1\mathrm{/}2$ and $q_{\pi }(2\mid x)=(2\mathrm{/}3)\mathrm{/}(4\mathrm{/}3)=1\mathrm{/}2$.

Using this conditional law in the definition [F5] yields $Q(\lambda \mid \pi )=\frac{1}{2}\log (2{\lambda }_1)+\frac{1}{2}\log ({\lambda }_2)$. The two coefficients are exactly the current responsibilities, so this is the complete-data log density averaged over the latent label. By [F4], these weights are also the conditional probabilities for the observation fiber.

For every $\theta \in (0,1)$, summing $p_{\theta }(x,z)$ over $(x,z)\in \{0,1{\}}^2$ gives $(\theta +(1-\theta ){)}^2=1$. At $x=1$, $m_{\theta }(1)=\theta$ and the conditional law of $Z$ is Bernoulli$(\theta )$, so the current law is Bernoulli$(1\mathrm{/}2)$. Every candidate has positive finite marginal, the same two-point latent support, and $D(q_t\mathrm{\parallel }{q}_{\theta })=\frac{1}{2}\log (1\mathrm{/}(2\theta ))+\frac{1}{2}\log (1\mathrm{/}(2(1-\theta )))$ is finite by [A1, F3]. Their complete-data log values are finite, so every $\theta \in (0,1)$ is admissible by [F4].

Averaging $\log p_{\theta }(1,Z)$ under that current conditional law gives $Q(\theta \mid 1\mathrm{/}2)=\frac{3}{2}\log \theta +\frac{1}{2}\log (1-\theta )=\frac{1}{2}\log ({\theta }^3(1-\theta ))$. At $\theta =1\mathrm{/}2$, this is $\log (1\mathrm{/}4)$; at $\theta =7\mathrm{/}10$, it is $\frac{1}{2}\log (1029\mathrm{/}10000)$. Their difference is $\frac{1}{2}\log (1029\mathrm{/}625)>0$, so the candidate strictly raises $Q$.

For $0<\theta <1$, $\frac{27}{256}-{\theta }^3(1-\theta )=(\theta -3\mathrm{/}4{)}^2((\theta +1\mathrm{/}4{)}^2+1\mathrm{/}8)\ge 0$, with equality only at $\theta =3\mathrm{/}4$. Thus the exact maximizer of $Q$ is $3\mathrm{/}4$, so $7\mathrm{/}10$ is a genuinely nonmaximizing generalized step. The observed log-likelihood difference is $\log (7\mathrm{/}10)-\log (1\mathrm{/}2)=\log (7\mathrm{/}5)>0$.

Since $0\le ({\theta }^2-1{)}^2\le 9$ on $[-2,2]$, $1\mathrm{/}4\le r(\theta )\le 1\mathrm{/}2$. Thus $p_{\theta }$ is nonnegative and sums to one over $(x,z)\in \{0,1{\}}^2$ for every $\theta$. The observed density at $x=1$ is $m_{\theta }(1)=r(\theta )$.

The squared term $({\theta }^2-1{)}^2$ vanishes exactly at $\theta =\pm 1$ and is positive elsewhere. Therefore $r(\theta )$, and hence the observed likelihood and log likelihood, have strict local maxima at those two distinct parameter values; both are global maximizers with equal value, so neither is a strict global maximum.

For every current value ${\theta }_t=\pm 1$, the latent conditional law is uniform on $Z$. For any candidate $\theta$, $Q(\theta \mid {\theta }_t)=\log r(\theta )-\log 2$, so its exact maximizer set is again $\{-1,1\}$. Starting from ${\theta }_0=-1$, choosing ${\theta }_{t+1}=-{\theta }_t$ at each step is an attained exact M-step; all fibers have positive finite marginals and identical latent support, with finite Q and KL. The algorithm therefore emits the infinite alternating sequence, which has no limit while its log likelihood is constant. This verifies the limitation described in [F5] without contradicting the one-step ascent result [F4].

For each fixed label pair the displayed density is continuous in $(ϵ,y_1,y_2)$ on $(0,\mathrm{\infty })\times {\mathbb{R}}^2$; the finite label space therefore makes it jointly measurable. For each observation coordinate, ${\sum }_{j=1}^2{\int }_{\mathbb{R}}\frac{1}{2}{\phi }_{a_j,s_j}(y)dy=1$ by [F1]. Lebesgue measure and finite counting measure are sigma-finite, so Tonelli [F5] gives integral $1\cdot 1=1$ for the nonnegative two-coordinate product density. This verifies the requirements of [F2] directly. Summing over the two labels in each coordinate gives the observed joint density $g_{ϵ}(y_1)g_{ϵ}(y_2)$, so its value at $(0,1)$ is $L_{ϵ}$.

At the two observations, $g_{ϵ}(0)\ge \frac{1}{2\sqrt{2\pi }ϵ}$ and $g_{ϵ}(1)\ge \frac{1}{2\sqrt{2\pi }}$. Thus $L_{ϵ}\ge 1\mathrm{/}(8\pi ϵ)$. For every $M>0$, choosing $0<ϵ<1\mathrm{/}(8\pi M)$ gives $L_{ϵ}>M$, so the likelihood is unbounded while each fixed-$ϵ$ value is finite. No finite parameter attains a global maximum.

This is compatible with [F3]–[F4]: the ascent theorem only compares consecutive admissible iterates; it neither bounds the likelihood sequence nor supplies a parameter attaining its unbounded supremum.

For each $\theta \in [0,1]$, the complete-data masses sum to $\theta +(1-\theta )=1$, and the observed marginal is $m_{\theta }(x_0)=1$. The current and candidate latent laws are therefore $q_t=(1\mathrm{/}2,1\mathrm{/}2)$ and $q_0=(1,0)$.

The singleton set $\{1\}$ has $q_0$-mass zero and $q_t$-mass $1\mathrm{/}2$, so $q_t\ll ̸q_0$ and $D(q_t\mathrm{\parallel }{q}_0)=+\mathrm{\infty }$ by [F2]. The extended diagnostic integral is $\frac{1}{2}\log (1-0)+\frac{1}{2}\log 0=-\mathrm{\infty }$; it is not a finite value of $Q(0\mid 1\mathrm{/}2)$ under [F3].

The observed log likelihood is zero at both parameters, but the candidate violates the support-compatible admissibility condition of [F3]. It is not an update allowed by the defined procedure.