Finite Gaussian mixture with positive-definite component covariances

Assume AC. Let $p\ge 1$, $\mu \in {\mathbb{R}}^p$, and let $\mathrm{\Sigma }\in {\mathbb{R}}^{p\times p}$ be symmetric positive definite, meaning $v^{\mathsf{T}}\mathrm{\Sigma }v>0$ for every nonzero $v\in {\mathbb{R}}^p$. Define the component function

${\varphi }_p(x;\mu ,\mathrm{\Sigma }):=(2\pi {)}^{-p\mathrm{/}2}(\det \mathrm{\Sigma }{)}^{-1\mathrm{/}2}\exp \text{\hspace{0.17em}⁣}(-\frac{1}{2}(x-\mu {)}^{\mathsf{T}}{\mathrm{\Sigma }}^{-1}(x-\mu )),x\in {\mathbb{R}}^p\mathrm{.}$

The component law is $N_p(\mu ,\mathrm{\Sigma })$, the $p$-dimensional normal law with mean $\mu$ and covariance matrix $\mathrm{\Sigma }$. The separate local normalization result proves that ${\varphi }_p$ is its Lebesgue density.

A finite $K$-component Gaussian mixture has an integer $K\ge 1$, weights ${\pi }_1,\dots ,{\pi }_K>0$ with ${\sum }_{k=1}^K{\pi }_k=1$, means ${\mu }_k\in {\mathbb{R}}^p$, and symmetric positive-definite covariance matrices ${\mathrm{\Sigma }}_k\in {\mathbb{R}}^{p\times p}$. Its density is

$f_{\theta }(x):={\sum }_{k=1}^K{\pi }_k{\varphi }_p(x;{\mu }_k,{\mathrm{\Sigma }}_k),$

where $\theta =(({\pi }_k,{\mu }_k,{\mathrm{\Sigma }}_k){)}_{k=1}^K$. For a positive integer $n$ and an ordered sample $x_1,\dots ,x_n$ modeled as conditionally independent observations from this mixture, define the likelihood $L(\theta ;x_{1:n}):={\prod }_{i=1}^n{f}_{\theta }(x_i)$ and log likelihood $\mathrm{\ell }(\theta ;x_{1:n}):={\sum }_{i=1}^n\log f_{\theta }(x_i)$. The responsibility weight of component $k$ for observation $x_i$ is

${\gamma }_{ik}(\theta ):=\frac{{\pi }_k{\varphi }_p(x_i;{\mu }_k,{\mathrm{\Sigma }}_k)}{{\sum }_{j=1}^K{\pi }_j{\varphi }_p(x_i;{\mu }_j,{\mathrm{\Sigma }}_j)}\mathrm{.}$

Multivariate normal law, including singular covariance — $N_p(\mu ,\mathrm{\Sigma })$ is the probability law whose scalar projections along $u\in {\mathbb{R}}^p$ are normal with mean $u^{\mathsf{T}}\mu$ and variance $u^{\mathsf{T}}\mathrm{\Sigma }u$; AC supplies its stated construction.

Moments, variance, and covariance on a probability space — covariance is the expectation of the product of centered coordinates.

For $n\ge 1$, the determinant over a commutative ring by the Leibniz formula, and $\mathrm{\mid }\det A\mathrm{\mid }$ for a real matrix — $\det \mathrm{\Sigma }$ is the determinant of the square matrix $\mathrm{\Sigma }$.

Invertible matrices and the general linear group ${\mathrm{GL}}_n(F)$ — ${\mathrm{\Sigma }}^{-1}$ denotes the two-sided inverse, which exists for positive-definite $\mathrm{\Sigma }$.

Real spectral theorem: a self-adjoint endomorphism of a finite-dimensional real inner product space has an orthonormal eigenbasis — real symmetric matrices admit orthonormal eigenbases; positive definiteness makes all eigenvalues positive.

Positive-definite multivariate Gaussian density is normalized ↗ — the displayed component function is normalized and is the Lebesgue density of $N_p(\mu ,\mathrm{\Sigma })$.