Mean vector and covariance matrix of a square-integrable random vector

Let $(\mathrm{\Omega },\mathfrak{F},\mathbb{P})$ be a probability space (Probability measures and probability spaces), let $m\ge 1$, and let $X:\mathrm{\Omega }\to {\mathbb{R}}^m$ be a random element with target $({\mathbb{R}}^m,\mathcal{B}({\mathbb{R}}^m))$ (Random elements and real random variables). Use the statistical one-based coordinate labels: for every $v\in {\mathbb{R}}^m$ write $v_i:=v(i-1)$ for $1\le i\le m$, so $X_i(\omega )=X(\omega )(i-1)$. Each $X_i$ is a real random variable because the coordinate projection $x\mapsto x(i-1)$ is continuous and therefore has Borel preimages of Borel sets (A continuous map has Borel preimages of Borel sets). For every integer $r\ge 1$ and matrix $A\in M_{r\times m}(\mathbb{R})$, the notation $A_{ij}$ below means its zero-based matrix entry $A(i-1,j-1)$ for $1\le i\le r$ and $1\le j\le m$; likewise ${\mathrm{\Sigma }}_{ij}$ means the entry at $(i-1,j-1)$ in the covariance matrix below in the matrix space of The vector space $M_{m\times n}(F):=F^{m\times n}$ of $m$ by $n$ matrices over a field, with entrywise operations. These are labels for the root library's zero-based function entries, not a different matrix convention. If $U\subseteq {\mathbb{R}}^m$ is open, $h:U\to {\mathbb{R}}^r$ has all first partials at $a\in U$, and $A=Jh(a)$, then $A_{ij}$ in this notation is ${\mathrm{\partial }}_{j-1}{h}_{i-1}(a)$, with the zero-based Jacobian convention of The Jacobian matrix of partial derivatives and the gradient in the scalar-valued case. Assume $X$ is square-integrable: $E\left[\mathrm{\mid }X{\mathrm{\mid }}^2\right]={\sum }_{i=1}^{m}E\left[X_i^2\right]<+\mathrm{\infty },$ where $\mathrm{\mid }\cdot \mathrm{\mid }$ is the Euclidean norm of The Euclidean inner product $⟨x,y⟩={\sum }_{k<n}{x}_k{y}_k$ on ${\mathbb{R}}^n$. Its mean vector is $\mu =E[X]:=(E[X_1],\dots ,E[X_m])\in {\mathbb{R}}^m,$ the coordinates being the expectations of Moments, variance, and covariance on a probability space. Its covariance matrix is the $m\times m$ real matrix of The vector space $M_{m\times n}(F):=F^{m\times n}$ of $m$ by $n$ matrices over a field, with entrywise operations $\mathrm{\Sigma }=\mathrm{Cov}(X):=({\mathrm{\Sigma }}_{ij}{)}_{i,j=1}^m,{\mathrm{\Sigma }}_{ij}:=E[(X_i-{\mu }_i)(X_j-{\mu }_j)]\in \mathbb{R}\mathrm{.}$ For the random vector $X$ the entries ${\mathrm{\Sigma }}_{ij}$ are also written $\mathrm{Cov}(X_i,X_j)$, and ${\mathrm{\Sigma }}_{ii}=\mathrm{Var}(X_i)$.

Well-definedness and elementary properties

Integrability of the entries. Square-integrability gives $E[X_i^2]<+\mathrm{\infty }$ for each $i$. Since the underlying measure is a probability measure, the elementary inequality $\mathrm{\mid }{X}_i\mathrm{\mid }\le \frac{1}{2}(1+X_i^2)$ gives $E\mathrm{\mid }{X}_i\mathrm{\mid }\le \frac{1}{2}(1+E[X_i^2])<+\mathrm{\infty }$ by monotonicity and homogeneity of the nonnegative integral (Monotonicity and nonnegative homogeneity of the nonnegative integral); hence $X_i$ is integrable and ${\mu }_i=E[X_i]$ is finite (Moments, variance, and covariance on a probability space). Also $(X_i-{\mu }_i{)}^2\le 2X_i^2+2{\mu }_i^2,$ so $X_i-{\mu }_i$ is square-integrable; the probability-space inequality $\mathrm{\mid }{X}_i-{\mu }_i\mathrm{\mid }\le \frac{1}{2}(1+(X_i-{\mu }_i{)}^2)$ then makes it integrable as well. Finally $\mathrm{\mid }(X_i-{\mu }_i)(X_j-{\mu }_j)\mathrm{\mid }\le \frac{1}{2}((X_i-{\mu }_i{)}^2+(X_j-{\mu }_j{)}^2),$ and the right-hand side is integrable by additivity, monotonicity and homogeneity of the nonnegative integral (Additivity of the nonnegative Lebesgue integral, Monotonicity and nonnegative homogeneity of the nonnegative integral); hence every ${\mathrm{\Sigma }}_{ij}$ is a finite real number.

Symmetry. For all $i,j$ the real numbers $(X_i-{\mu }_i)(X_j-{\mu }_j)$ and $(X_j-{\mu }_j)(X_i-{\mu }_i)$ are equal pointwise, so ${\mathrm{\Sigma }}_{ij}={\mathrm{\Sigma }}_{ji}$; thus ${\mathrm{\Sigma }}^{\mathsf{T}}=\mathrm{\Sigma }$.

For $M\in M_{m\times m}(\mathbb{R})$ and $z\in {\mathbb{R}}^m$, write $z^{\mathsf{T}}Mz:={\sum }_{i=1}^m{\sum }_{j=1}^m{z}_i{M}_{ij}{z}_j\mathrm{.}$ For a real symmetric $m\times m$ matrix $M$, positive semidefinite means $z^{\mathsf{T}}Mz\ge 0$ for every $z\in {\mathbb{R}}^m$, and positive definite means $z^{\mathsf{T}}Mz>0$ for every nonzero $z\in {\mathbb{R}}^m$. For a real random variable $Z$, almost surely constant means that $Z=c$ outside a measurable $\mathbb{P}$-null set for some $c\in \mathbb{R}$ (Measure-null sets and almost-everywhere statements relative to a measure).

Positive semidefiniteness. Let $z=(z_1,\dots ,z_m)\in {\mathbb{R}}^m$. The random variable $W:={\sum }_{i=1}^m{z}_i{X}_i$ is a finite linear combination of square-integrable random variables, hence square-integrable, and bilinearity of covariance (Variance and covariance identities for random variables) gives $z^{\mathsf{T}}\mathrm{\Sigma }z={\sum }_{i,j=1}^m{z}_i{z}_j\mathrm{Cov}(X_i,X_j)=\mathrm{Cov}(W,W)=\mathrm{Var}(W)=E[(W-E[W]{)}^2]\ge 0,$ the inequality holding because $(W-E[W]{)}^2\ge 0$ pointwise and expectation is monotone (Linearity, monotonicity, and the modulus bound for expectation). If $z^{\mathsf{T}}\mathrm{\Sigma }z=\mathrm{Var}(W)=0$, Chebyshev's inequality gives $\mathbb{P}(\mathrm{\mid }W-E[W]\mathrm{\mid }\ge 1\mathrm{/}r)=0$ for every integer $r\ge 1$; the union of these events is $\{W\mathrm{\ne }E[W]\}$, so $W=E[W]$ almost surely (Chebyshev's inequality for random variables). Conversely, if $W=c$ almost surely, invariance of the integral under almost-everywhere equality gives $E[W]=c$ and $E[(W-E[W]{)}^2]=0$ (Two integrable functions are equal almost everywhere exactly when all of their indefinite integrals agree). Hence $z^{\mathsf{T}}\mathrm{\Sigma }z=0$ exactly when $W$ is almost surely constant, and $\mathrm{\Sigma }$ is positive definite exactly when for every $z\mathrm{\ne }0$ the random variable ${\sum }_{i=1}^m{z}_i{X}_i$ fails to be almost surely constant.

No independence hypothesis is used anywhere in this definition; $\mathrm{\Sigma }$ records the pairwise covariances of the coordinates.