ReLU forward preactivation-variance recurrence

Let $L\ge 2$ and let $d_0,\dots ,d_L$ be positive integers. For each $1\le \mathrm{\ell }\le L$, let $W_{\mathrm{\ell }}\in {\mathbb{R}}^{d_{\mathrm{\ell }}\times d_{\mathrm{\ell }-1}}$ have mutually independent, identically distributed entries with a symmetric law, mean zero, and finite variance ${\sigma }_{\mathrm{\ell }}^2$, and suppose $W_{\mathrm{\ell }}$ is independent of the input activation vector $X_{\mathrm{\ell }-1}$. Set the fan-in $n_{\mathrm{\ell }}=d_{\mathrm{\ell }-1}$, set every bias to zero, and define $Z_{\mathrm{\ell }}=W_{\mathrm{\ell }}{X}_{\mathrm{\ell }-1}$ and $X_{\mathrm{\ell }}=\rho (Z_{\mathrm{\ell }})$ coordinatewise, where $\rho (t)=\max (t,0)$. Assume for every layer that the coordinates of $X_{\mathrm{\ell }-1}$ have a common finite second moment; they need not be independent or identically distributed. Also assume that each preactivation coordinate $Z_{\mathrm{\ell },j}$ has a symmetric law; an atom at zero is allowed in this forward second-moment recurrence. Then, for every $2\le \mathrm{\ell }\le L$ and $1\le j\le d_{\mathrm{\ell }}$, $\mathrm{Var}(Z_{\mathrm{\ell },j})=\frac{1}{2}{n}_{\mathrm{\ell }}{\sigma }_{\mathrm{\ell }}^2\mathrm{Var}(Z_{\mathrm{\ell }-1,1})\mathrm{.}$ At the raw-input boundary, the exact base relation is $\mathrm{Var}(Z_{1,j})=n_1{\sigma }_1^2\mathbb{E}[X_{0,1}^2];$ if $X_{0,1}$ is centered, its second moment here is its variance. In particular, writing $V_{\mathrm{\ell }}=\mathrm{Var}(Z_{\mathrm{\ell },1})$, the exact finite-depth formula is $V_L=V_1{\prod }_{\mathrm{\ell }=2}^L\frac{n_{\mathrm{\ell }}{\sigma }_{\mathrm{\ell }}^2}{2}\mathrm{.}$ Thus ${\sigma }_{\mathrm{\ell }}^2=2\mathrm{/}{n}_{\mathrm{\ell }}$ for $\mathrm{\ell }\ge 2$ gives $V_L=V_1$, with zero atoms allowed at every depth. This is a statement about ensemble second moments, not about variance along every sampled network or training success.

The equal-second-moment assumption allows the dependence between coordinates that naturally arises after averaging over a random previous layer. For a concrete finite-depth model satisfying all these hypotheses, take every width to be one, $X_0=1$, and mutually independent weights $W_{\mathrm{\ell }}=\pm \sqrt{v_{\mathrm{\ell }}}$ with equal probabilities, where $v_{\mathrm{\ell }}>0$. At each layer the fresh weight is independent of the preceding activation and its random sign makes the preactivation symmetric. The event $W_1<0$ already forces all subsequent preactivations to vanish, yet $V_1=v_1$ and $V_L=v_1{\prod }_{\mathrm{\ell }=2}^L(v_{\mathrm{\ell }}\mathrm{/}2)$ hold exactly. Thus atomlessness is unnecessary for finite-depth forward propagation.