Gaussian activation covariance and tangent recursion preserve strict positivity

On every compact interval, [F1] gives differentiability of $\sigma$ outside a null set. Taking a countable union over intervals $[-n,n]$ gives one Lebesgue-null exceptional set on $\mathbb{R}$. The continuous difference quotients $s_n(x)=n(\sigma (x+1\mathrm{/}n)-\sigma (x))$ are bounded in modulus by the Lipschitz constant; their pointwise upper limit is a bounded Borel function and equals ${\sigma }^{\mathrm{\prime }}(x)$ wherever the derivative exists. Thus a representative $s$ as in the Statement exists.

Let $A=C^{1\mathrm{/}2}$ and $G$ be the independent standard-normal vector in [F2]. Positive definiteness of $C$ makes $A$ invertible: if $Av=0$, then $v^{T}Cv=v^T{A}^{2}v=\mathrm{\parallel }Av{\mathrm{\parallel }}_2^2=0$, so $v=0$. The product standard-normal density is positive on every nonempty open box; since $A^{-1}$ maps nonempty open sets to nonempty open sets, $Z=AG$ assigns positive probability to every nonempty open set in ${\mathbb{R}}^m$.

By [F2], each $Z_i$ has a scalar normal law with variance $C_{ii}>0$, hence a density. Therefore changing $s$ on the exceptional null set changes none of the random variables $s(Z_i)$ almost surely.

Fix a nonzero vector $c\in {\mathbb{R}}^m$ and set $F(z)={\sum }_{i=1}^m{c}_i\sigma (z_i)$. This function is continuous. If $\mathbb{E}[F(Z{)}^2]=0$, then for every integer $k\ge 1$, $F(Z{)}^2\ge k^{-2}{\mathbf{1}}_{\{\mathrm{\mid }F(Z)\mathrm{\mid }\ge 1\mathrm{/}k\}}$ forces $\mathbb{P}(\mathrm{\mid }F(Z)\mathrm{\mid }\ge 1\mathrm{/}k)=0$; the countable union gives $F(Z)=0$ almost surely. By the full-support conclusion of step 1.2, continuity forces $F(z)=0$ for every $z\in {\mathbb{R}}^m$. Choose $u,v$ with $\sigma (u)\mathrm{\ne }\sigma (v)$. For each $j$, evaluate this identity at vectors whose coordinates other than $j$ are zero and whose $j$th coordinates are $u$ and $v$, then subtract; it gives $c_j(\sigma (u)-\sigma (v))=0$, so $c_j=0$. This contradicts $c\mathrm{\ne }0$. Hence $\mathbb{E}[F(Z{)}^2]>0$ and $c^{T}Sc=\mathbb{E}[F(Z{)}^2]+{\beta }^2({\sum }_i{c}_i{)}^2>0$.

For any $c\in {\mathbb{R}}^m$, $c^{T}Dc=\mathbb{E}[({\sum }_i{c}_{i}s(Z_i){)}^2]\ge 0$; the expectation is finite because $s$ is bounded. Thus $D$ is positive semidefinite.

Apply [F3] to $T$. Each eigenvalue $\lambda$ is nonnegative because for a unit eigenvector $q$, $\lambda =q^{T}Tq\ge 0$. If $q_1,\dots ,q_m$ is an orthonormal eigenbasis with eigenvalues ${\lambda }_1,\dots ,{\lambda }_m$, the vectors $u_i=(\sqrt{{\lambda }_1}(q_1{)}_i,\dots ,\sqrt{{\lambda }_m}(q_m{)}_i)$ satisfy $T_{ij}=⟨u_i,u_j⟩$. For $D$, the same finite-dimensional factorization applies because step 3.1 proves it is positive semidefinite, so there are vectors $v_i$ with $D_{ij}=⟨v_i,v_j⟩$.

Then $(T\circ D{)}_{ij}=⟨u_i\otimes v_i,u_j\otimes v_j⟩$, so $T\circ D$ is positive semidefinite. Adding the strictly positive quadratic form of $S$ from step 2.2 shows $T\circ D+S$ is strictly positive definite.