The recursively defined population Gram matrices are finite, symmetric, and positive semidefinite

The input matrix $K^{(0)}$ is symmetric, and for $v\in {\mathbb{R}}^n$, $v^T{K}^{(0)}v=\mathrm{\parallel }{\sum }_i{v}_i{x}_i{\mathrm{\parallel }}_2^2\ge 0$. Its diagonal entries are $\mathrm{\parallel }{x}_i{\mathrm{\parallel }}_2^2=1$, giving the base case.

Assume the induction hypothesis that $K^{(h-1)}$ is finite, symmetric positive semidefinite, and has diagonal $1$, for some $1\le h<H$. By [F2]–[F3], take $G\sim N_n(0,K^{(h-1)})$; each pair $(G_i,G_j)$ has covariance $A_{ij}^{(h)}$. The growth bound $\mathrm{\mid }\sigma (z)\mathrm{\mid }\le \mathrm{\mid }\sigma (0)\mathrm{\mid }+L\mathrm{\mid }z\mathrm{\mid }$ and finite Gaussian second moments put each $\sigma (G_i)$ in $L^2$; [F8] then makes every product $\sigma (G_i)\sigma (G_j)$ integrable. Thus $K^{(h)}$ is finite and symmetric, and for every $v$, $v^T{K}^{(h)}v=c_{\sigma }\mathbb{E}[({\sum }_i{v}_i\sigma (G_i){)}^2]\ge 0$. Because $G_i\sim N(0,1)$, its diagonal is $c_{\sigma }\mathbb{E}[\sigma (Z{)}^2]=1$ by [F5]. This establishes the induction step.

For the fixed $H$, let $P(h)$ mean that $K^{(h)}$ is finite, symmetric positive semidefinite, and has diagonal $1$, for $0\le h<H$. Define $Q(k)$ to mean $P(j)$ for every $j\le \min (k,H-1)$. Step 1.1 gives $Q(0)$. If $Q(k)$ and $k+1<H$, step 2.1 applied with $h=k+1$ gives $P(k+1)$ and hence $Q(k+1)$; if $k+1\ge H$, $Q(k+1)$ adds no new index. By [F6], $Q(H-1)$ holds, so $K^{(H-1)}$ is finite, symmetric positive semidefinite, and has diagonal $1$; when $H=1$, this is exactly the base matrix $K^{(0)}$. By [F4], take independent $G,G^{\mathrm{\prime }}$ both distributed as $N_n(0,K^{(H-1)})$. The vector with coordinates $Y_i=\sqrt{c_{\sigma }}{G}_i^{\mathrm{\prime }}{\sigma }^{\mathrm{\prime }}(G_i)$ has finite second moments; ${\sigma }^{\mathrm{\prime }}$ is bounded because it is the derivative of a Lipschitz differentiable function. Factoring expectations of independent variables gives $\mathbb{E}[Y_i{Y}_j]=c_{\sigma }{K}_{ij}^{(H-1)}\mathbb{E}[{\sigma }^{\mathrm{\prime }}(G_i){\sigma }^{\mathrm{\prime }}(G_j)]=K_{ij}^{(H)}$. Therefore $K^{(H)}$ is finite and symmetric, and $v^T{K}^{(H)}v=\mathbb{E}[({\sum }_i{v}_i{Y}_i{)}^2]\ge 0$ for every $v$. This proves the fully connected terminal claim and discharges that induction.

For the residual base layer let $G_0\sim N_n(0,S^{(0)})$ and $V^{(1)}=\sqrt{c_{\sigma }}\sigma (G_0)$, with activation applied coordinatewise. Then $E{V}^{(1)}=b^{(1)}$ and $E[V^{(1)}(V^{(1)}{)}^T]=S^{(1)}$ by [F1]. The growth bound used in step 2.1 proves finite second moments. For every $v$, $v^T{S}^{(1)}v=E[(v^T{V}^{(1)}{)}^2]\ge 0$, while $S^{(1)}-b^{(1)}(b^{(1)}{)}^T$ is its covariance and has quadratic form $E[(v^T(V^{(1)}-E{V}^{(1)}){)}^2]\ge 0$. Thus both matrices are symmetric positive semidefinite.

Suppose a finite-second-moment vector $V^{(h-1)}$ has mean $b^{(h-1)}$ and second-moment matrix $S^{(h-1)}$. Let $\mu$ be the Borel law of this generally non-Gaussian vector and let $\nu =N_n(0,S^{(h-1)})$, which exists by [F2]. Both are probability measures and hence sigma-finite. By [F7], on $({\mathbb{R}}^n\times {\mathbb{R}}^n,\mu \times \nu )$ the coordinate projections $V$ and $G$ have laws $\mu$ and $\nu$ and are independent: for Borel sets $A,B$, the probability of $\{V\in A,G\in B\}$ is $\mu (A)\nu (B)$. Put $V^{(h)}=V+\alpha \sigma (G)$. Finite second moments follow from the growth bound and $(u+v{)}^2\le 2u^2+2v^2$. Independence gives $E{V}^{(h)}=b^{(h-1)}+\alpha {\mu }^{(h-1)}$ and $E[V_i^{(h)}{V}_j^{(h)}]=S_{ij}^{(h-1)}+\alpha (b_i^{(h-1)}{\mu }_j^{(h-1)}+b_j^{(h-1)}{\mu }_i^{(h-1)})+{\alpha }^{2}E[\sigma (G_i)\sigma (G_j)]=S_{ij}^{(h)}$. For every $v$, $v^T{S}^{(h)}v=E[(v^T{V}^{(h)}{)}^2]\ge 0$, and $S^{(h)}-b^{(h)}(b^{(h)}{)}^T$ is the covariance of $V^{(h)}$, so its quadratic form is also an expectation of a square. Thus both matrices are finite, symmetric positive semidefinite at this stage, which supplies the covariance hypothesis for the next stage. Iterating through the finitely many layers is justified by [F6].

For $H\ge 2$, take independent $G,G^{\mathrm{\prime }}\sim N_n(0,S^{(H-1)})$ and set $Y_i=(c_{\mathrm{r}\mathrm{e}\mathrm{s}}\mathrm{/}H)G_i^{\mathrm{\prime }}{\sigma }^{\mathrm{\prime }}(G_i)$. The boundedness $\mathrm{\mid }{\sigma }^{\mathrm{\prime }}\mathrm{\mid }\le L$ gives finite second moments, and independence gives $E[Y_i{Y}_j]=K_{\mathrm{r}\mathrm{e}\mathrm{s},ij}^{(H)}$. Consequently this terminal matrix is symmetric positive semidefinite. At $H=1$ it equals the already proved fully connected matrix by definition. This proves all residual assertions.