Jacot–Gabriel–Hongler width-normalized fully connected networks

Assume AC. Fix integers $L,n_0,\dots ,n_L\ge 1$, a bias scale $\beta >0$, and an activation $\sigma :\mathbb{R}\to \mathbb{R}$. For each layer $\mathrm{\ell }=0,\dots ,L-1$, let $W^{(\mathrm{\ell })}\in {\mathbb{R}}^{n_{\mathrm{\ell }+1}\times n_{\mathrm{\ell }}}$ and $b^{(\mathrm{\ell })}\in {\mathbb{R}}^{n_{\mathrm{\ell }+1}}$. At initialization all entries of all matrices and bias vectors are mutually independent and each has the standard normal law $N(0,1)$. These finite collections of entries have the corresponding finite-dimensional multivariate normal product law.

For an input $x\in {\mathbb{R}}^{n_0}$, put $z^{(0)}(x)=x$ and, for $\mathrm{\ell }=0,\dots ,L-1$, define the preactivation $u^{(\mathrm{\ell }+1)}(x)=\frac{1}{\sqrt{n_{\mathrm{\ell }}}}{W}^{(\mathrm{\ell })}{z}^{(\mathrm{\ell })}(x)+\beta b^{(\mathrm{\ell })}\mathrm{.}$ For $\mathrm{\ell }+1<L$, set $z^{(\mathrm{\ell }+1)}(x)=\sigma (u^{(\mathrm{\ell }+1)}(x))$ coordinatewise; the output layer is linear and $f_{\theta }(x):=u^{(L)}(x)\in {\mathbb{R}}^{n_L}$. The parameter vector $\theta$ consists of every entry of the matrices and biases. The width $n_L$ is the fixed output dimension; the hidden widths are $n_1,\dots ,n_{L-1}$. A sequential hidden-width limit takes $n_1\to \mathrm{\infty }$ first, then $n_2\to \mathrm{\infty }$, and so on through $n_{L-1}\to \mathrm{\infty }$.

Jacot, Gabriel, and Hongler, §2, source layer parametrization and initialization

Every family of nonempty sets has a choice function

Standard normal and normal laws: The standard normal law has density $(2\pi {)}^{-1\mathrm{/}2}{e}^{-x^2\mathrm{/}2}$

Finite-dimensional multivariate normal laws allow singular covariance