A positive activation mean creates a high-variance common initialization mode

Softplus is positive and nonconstant. It is analytic, has derivative in $(0,1)$, and its second derivative is at most $1\mathrm{/}4$, so it satisfies the activation regularity assumptions. By [F2] and [F4], $\mu$ is finite; because $\sigma (Z)\ge \log 2$ when $Z\ge 0$ and the standard normal density is positive on $(0,1)$, $\mu >0$. Since softplus is nonconstant, choose $z_0$ with $\sigma (z_0)\mathrm{\ne }\mu$; continuity and the positive Gaussian density give $\mathbb{E}[(s-\mu {)}^2]>0$. Expanding this finite square and using $\mathbb{E}s=\mu$ yields $\mathbb{E}[s^2]>{\mu }^2$. Therefore $0<\rho <1$.

Let $U=\mathrm{span}\{x_1,\dots ,x_n\}$. By [F5], ${\mathbb{R}}^d=U\oplus U^{\perp }$, and [F6] supplies an orthonormal basis of $U^{\perp }$. Joining it to $x_1,\dots ,x_n$ gives the columns of an orthogonal matrix $Q$. By [F7], $W^{(1)}Q$ has the same iid Gaussian law as $W^{(1)}$, so all coordinates $W_{r\cdot }^{(1)}{x}_i$ are independent standard normals. Thus $\mathbb{E}[(\mathrm{\Phi }{\mathrm{\Phi }}^T{)}_{ii}]=c_{\sigma }\mathbb{E}[s^2]=1$, while for $i\mathrm{\ne }j$, $\mathbb{E}[(\mathrm{\Phi }{\mathrm{\Phi }}^T{)}_{ij}]=c_{\sigma }{\mu }^2=\rho$. Therefore $\mathbb{E}[\mathrm{\Phi }{\mathrm{\Phi }}^T]=(1-\rho )I_n+\rho {\mathbf{11}}^T$, and its quadratic form in the unit vector $e$ is $1+(n-1)\rho$.

Conditional on the hidden weights, $e^{T}u=e^T\mathrm{\Phi }a$ is a centered normal random variable with variance $S=e^T\mathrm{\Phi }{\mathrm{\Phi }}^{T}e\ge 0$, by [F1] and [F3]. The nonnegative random variable $S$ has expectation $1+(n-1)\rho >\rho n\mathrm{/}2$, so the event $S>\rho n\mathrm{/}2$ has positive probability; otherwise its expectation would be at most $\rho n\mathrm{/}2$. On this event, for any fixed $C>0$, the conditional probability of $\mathrm{\mid }{e}^{T}u\mathrm{\mid }>C\sqrt{n}$ is at least $P\{\mathrm{\mid }Z\mathrm{\mid }>C\sqrt{2\mathrm{/}\rho }\}>0$. Averaging over the hidden weights proves positive unconditional probability. Since $e$ is a unit vector, write $u=(e^{T}u)e+(u-(e^{T}u)e)$; the summands are orthogonal, so $\mathrm{\parallel }u{\mathrm{\parallel }}_2^2=\mathrm{\mid }{e}^{T}u{\mathrm{\mid }}^2+\mathrm{\parallel }u-(e^{T}u)e{\mathrm{\parallel }}_2^2\ge \mathrm{\mid }{e}^{T}u{\mathrm{\mid }}^2$. This proves the asserted absence of one confidence-uniform constant by a local projection calculation.