Computational foundations: factorization, KL, and optimization — Examples

Apply [F1] to the stated laws; its hypotheses are exactly $a,b\in \mathbb{R}$ and $\tau >0$, and its AC hypothesis is [A1].

Since $(a-b{)}^2\ge 0$ and $2{\tau }^2>0$, the displayed value is nonnegative and equals zero when $a=b$. In that case the two Gaussian laws coincide, so the equality clause is consistent with [F2].

Factor $f(x)=(x-1{)}^2(x+1{)}^2\ge 0$ for every real $x$, with equality at $x=1$ and $x=-1$. Thus these are distinct global minimizers and the minimum value is zero.

For $\mathrm{\mid }x\mathrm{\mid }<1$, $f(x)=(1-x^2{)}^2\le 1$, with strict inequality for $x\mathrm{\ne }0$; since $f(0)=1$, $x=0$ is a strict local maximum.

Steps 1.1–1.2 exhibit a smooth objective with two global minima and an intervening local maximum. The procedures in [F1]–[F2] are defined by local updates and make no global-minimization or initialization-independent claim.

The map $p(x,j)=w_j{f}_j(x)$ is nonnegative and product measurable. With counting measure on the finite label set, ${\int }_{\mathsf{X}\times \mathsf{Z}}pd(\nu \otimes \mathrm{\#})={\sum }_{j=1}^J{w}_j{\int }_{\mathsf{X}}{f}_{j}d\nu ={\sum }_{j=1}^J{w}_j=1.$ It is therefore a complete-data probability density of the type in [F1].

Integrating over the label coordinate gives $m(x)={\int }_{\mathsf{Z}}p(x,j)\mathrm{\#}(dj)={\sum }_{j=1}^J{w}_j{f}_j(x)$. For $0<m(x)<\mathrm{\infty }$, the conditional label mass is $q(j\mid x)=w_j{f}_j(x)\mathrm{/}m(x)$ and its finite sum is ${\sum }_{j=1}^{J}q(j\mid x)=\frac{{\sum }_{j=1}^J{w}_j{f}_j(x)}{m(x)}=1.$

The joint density, its observed marginal, and its normalized positive finite fibres therefore have exactly the finite-mixture factorization claimed.

The only nonzero summand is $p(0,0)=1$, so ${\sum }_{x,z}p(x,z)=1$; this is a normalized joint density on the finite product space.

Summing over $z$ gives $m(0)=1$ and $m(1)=0$. At $x=1$, both $p(1,z)$ and $m(1)$ are zero, so the quotient $p(1,z)\mathrm{/}m(1)$ is undefined.

Since $m(1)=0$, the observation $x=1$ is outside the positive finite fibre set in [F1]. The formula supplies no pointwise conditional latent density there.

Take $A=I_d$ in [F1]. Then $\mathrm{\nabla }f(x)=x-b$ and $H_f(x)=I_d$. This Hessian is positive semidefinite, so [F2] proves convexity.

Direct expansion gives, for all $x,y$, $f(y)=f(x)+⟨x-b,y-x⟩+\frac{1}{2}\mathrm{\parallel }y-x{\mathrm{\parallel }}^2\mathrm{.}$ Thus the strong-convexity inequality holds with $\mu =1$. The gradient satisfies $\mathrm{\parallel }\mathrm{\nabla }f(x)-\mathrm{\nabla }f(y)\mathrm{\parallel }=\mathrm{\parallel }x-y\mathrm{\parallel }\le 2\mathrm{\parallel }x-y\mathrm{\parallel }$, so $L=2$ is valid. The minimum is $f^{\ast }=0$ at $x=b$.

Whenever the algorithm makes a fixed-step update with step $1\mathrm{/}2$, $x_{k+1}=x_k-\frac{1}{2}(x_k-b)=b+\frac{1}{2}(x_k-b)$. Induction along the generated iterates gives $\mathrm{\parallel }{x}_k-b{\mathrm{\parallel }}^2=4^{-k}\mathrm{\parallel }{x}_0-b{\mathrm{\parallel }}^2$, so $f(x_k)-f^{\ast }=4^{-k}(f(x_0)-f^{\ast })\le 2^{-k}(f(x_0)-f^{\ast })$ for each generated $k\ge 1$; for $k=0$ the gap is unchanged. A caller stop ends this finite prefix and does not assert later iterates. This is the bound from [F3] with $\mu \mathrm{/}L=1\mathrm{/}2$.