Exact Gaussian-mixture EM does not decrease observed log likelihood

Let ${\mathrm{\Theta }}_K$ be the set of mixture parameters in [F1], let $Y=({\mathbb{R}}^p{)}^n$ with product Lebesgue measure $\nu ={\lambda }_p^{\otimes n}$, and let $Z=\{1,\dots ,K{\}}^n$ with counting measure $\mathrm{\#}(B):=\mathrm{\mid }B\mathrm{\mid }$ for $B\subseteq Z$. The parameter domain is Borel: the positive-weight simplex is relatively open in its affine hyperplane, and the SPD cone is open as follows. For a symmetric positive-definite matrix $A$, the spectral theorem [F15] gives an orthonormal eigenbasis. Every eigenvalue $\lambda$ has a unit eigenvector $u$ with $\lambda =u^{\mathsf{T}}Au>0$, so the least eigenvalue ${\lambda }_{\min }(A)$ is positive. For a symmetric perturbation $E$, define $\mathrm{\parallel }E{\mathrm{\parallel }}_F:=({\sum }_{i,j}{e}_{ij}^2{)}^{1\mathrm{/}2}$. Applying [F19] to the finite lists $(e_{ij})$ and $(v_i{v}_j)$ gives $\mathrm{\mid }{v}^{\mathsf{T}}Ev\mathrm{\mid }\le \mathrm{\parallel }E{\mathrm{\parallel }}_F({\sum }_{i,j}{v}_i^2{v}_j^2{)}^{1\mathrm{/}2}=\mathrm{\parallel }E{\mathrm{\parallel }}_F\mathrm{\parallel }v{\mathrm{\parallel }}_2^2$. Therefore, whenever $\mathrm{\parallel }E{\mathrm{\parallel }}_F<{\lambda }_{\min }(A)$, $v^{\mathsf{T}}(A+E)v\ge ({\lambda }_{\min }(A)-\mathrm{\parallel }E{\mathrm{\parallel }}_F)\mathrm{\parallel }v{\mathrm{\parallel }}_2^2>0$ for every nonzero $v$, so $A+E$ is positive definite. This supplies a relative open neighborhood of each $A$ in the symmetric-matrix subspace. For each fixed $z\in Z$, the formula below is continuous in $(\eta ,y)\in {\mathrm{\Theta }}_K\times Y$: determinants and inverses are continuous on the SPD cone by [F15], and the remaining operations are sums, products, and exponentials. A continuous map on this Borel subspace has Borel preimages relative to it; those relative Borel sets are ambient Borel. The product-Borel identification for Euclidean factors [F16] and finite discreteness of $Z$ then make the function jointly measurable for the required product sigma-algebra.

Define the complete-data density, relative to $\nu$ times counting measure on $Z$, by $p(\eta ,y,z):={\prod }_{i=1}^n{\pi }_{z_i}{\varphi }_p(y_i;{\mu }_{z_i},{\mathrm{\Sigma }}_{z_i})\mathrm{.}$ Repeated use of [F9] constructs the finite product measures, including $\nu \otimes \mathrm{\#}$, and preserves sigma-finiteness. AC supplies the countable-choice premise of [F11] by [F10]; repeated use of [F11] then identifies $\nu ={\lambda }_p^{\otimes n}$ with Euclidean Lebesgue measure on ${\mathbb{R}}^{pn}$. For each component, [F1,F2] and finite-sum additivity [F18] give $\int f_{\eta }(y_i){\lambda }_p(d{y}_i)={\sum }_k{\pi }_k=1$. Distributing the finite sum over labels gives ${\sum }_{z\in Z}p(\eta ,y,z)={\prod }_i{f}_{\eta }(y_i)$. Repeated Tonelli integration [F12] therefore gives ${\sum }_{z\in Z}{\int }_{Y}p(\eta ,y,z)\nu (dy)={\int }_Y{\prod }_i{f}_{\eta }(y_i)\nu (dy)={\prod }_i{\int }_{{\mathbb{R}}^p}{f}_{\eta }(y_i){\lambda }_p(d{y}_i)=1.$ Thus $p$ is normalized on each parameter fibre as required by [F3].

For a fixed observed sample $x_{1:n}$, summing the complete density over the finite label set yields $m_{\eta }(x_{1:n})={\sum }_{z\in Z}p(\eta ,x_{1:n},z)={\prod }_i{\sum }_k{\pi }_k{\varphi }_p(x_i;{\mu }_k,{\mathrm{\Sigma }}_k)={\prod }_i{f}_{\eta }(x_i)\mathrm{.}$ Every factor is finite and strictly positive by [F1,F2], so this marginal is positive and finite. Its logarithm is exactly the likelihood $\mathrm{\ell }(\eta ;x_{1:n})$ in the statement. By [F4], every normalized posterior $q_{\eta }^{x_{1:n}}(z)=p(\eta ,x_{1:n},z)\mathrm{/}{m}_{\eta }(x_{1:n})$ is a probability mass function. More explicitly it is $q_{\eta }^{x_{1:n}}(z)={\prod }_i{\gamma }_{i,z_i}(\eta ),$ and every one of its finitely many masses is positive.

At any current parameter $\theta$ and its exact update ${\theta }^{\mathrm{\prime }}$, both posterior measures obtained from the probability masses in step 3.1 by [F17] are strictly positive on the same finite label set. Hence $q_{\theta }^{x_{1:n}}\ll q_{{\theta }^{\mathrm{\prime }}}^{x_{1:n}}$, and the directional KL divergence is a sum of finitely many finite terms by [F8], so it is finite. For each $z$, $p(\theta ,x_{1:n},z)$ and $p({\theta }^{\mathrm{\prime }},x_{1:n},z)$ are strictly positive and finite; their logarithms are finite. The posterior expectations of the absolute log densities are finite sums and therefore finite. Thus every admissibility condition in [F5,F6] holds for this current/candidate pair.

The posterior in step 3.1 factors over observations. Taking the finite expectation of the complete log density gives ${\sum }_{z\in Z}{q}_{\theta }^{x_{1:n}}(z)\log p(\vartheta ,x_{1:n},z)={\sum }_{i=1}^n{\sum }_{k=1}^K{\gamma }_{ik}(\theta )\log \text{\hspace{0.17em}⁣}({\rho }_k{\varphi }_p(x_i;m_k,V_k))=Q(\vartheta \mid \theta ),$ where $Q$ is the function in [F7]. Indeed, expand $\log p$ as the sum over $i$, sum first over $z_i=k$, and use ${\sum }_{z_j=1}^K{\gamma }_{j,z_j}(\theta )=1$ for every $j\mathrm{\ne }i$. This is a finite-sum identity, and all terms are finite by step 3.1.

The local exact M-step result [F7] says ${\theta }^{\mathrm{\prime }}$ globally maximizes this same Q over the full mixture domain. Step 4.1 verifies that both $\theta$ and ${\theta }^{\mathrm{\prime }}$ are admissible, so the two-point candidate set $\{\theta ,{\theta }^{\mathrm{\prime }}\}$ is admissible and contains the current parameter; the global maximizer ${\theta }^{\mathrm{\prime }}$ is therefore an attained maximizer over this set as required by the exact mode of [F5]. Consequently $Q({\theta }^{\mathrm{\prime }}\mid \theta )\ge Q(\theta \mid \theta )$. Applying the identity and nonnegativity conclusion of [F6], with the verified conditions from step 4.1, yields $\mathrm{\ell }({\theta }^{\mathrm{\prime }};x_{1:n})-\mathrm{\ell }(\theta ;x_{1:n})=Q({\theta }^{\mathrm{\prime }}\mid \theta )-Q(\theta \mid \theta )+D(q_{\theta }^{x_{1:n}}\mathrm{\parallel }{q}_{{\theta }^{\mathrm{\prime }}}^{x_{1:n}})\ge 0.$ Thus one exact update does not decrease observed log likelihood.

Initially ${\theta }_0$ is in the mixture domain by hypothesis. If ${\theta }_t$ is in that domain, step 3.1 supplies positive finite marginals and positive responsibilities, and [F7] supplies the next positive weights and SPD covariances; hence ${\theta }_{t+1}$ is again in the domain. Induction gives valid parameters through $T$, and step 5.1 applies at each $t<T$, proving every displayed likelihood inequality. This proves the claim without asserting strict increase or convergence.