PReLU fan-in and fan-out variance scales

Let $a\in \mathbb{R}$ and consider a finite rectified stack with zero biases, widths $d_0,\dots ,d_L$, $Z_{\mathrm{\ell }}=W_{\mathrm{\ell }}{X}_{\mathrm{\ell }-1}$, and $X_{\mathrm{\ell }}={\varphi }_a(Z_{\mathrm{\ell }})$, where ${\varphi }_a(t)=t$ for $t>0$ and ${\varphi }_a(t)=at$ for $t\le 0$. Put $n_{\mathrm{\ell }}=d_{\mathrm{\ell }-1}$ and ${\widehat{n}}_{\mathrm{\ell }}=d_{\mathrm{\ell }}$. Suppose each $W_{\mathrm{\ell }}$ has independent entries with a common symmetric centered law of finite variance ${\sigma }_{\mathrm{\ell }}^2$ and is independent of $X_{\mathrm{\ell }-1}$. Suppose each layer input has coordinates with a common finite second moment, and all preactivation coordinates are symmetric about zero. Input coordinates may be dependent, and zero atoms are allowed. Then, writing $V_{\mathrm{\ell }}=\mathrm{Var}(Z_{\mathrm{\ell },1})$, $V_1=n_1{\sigma }_1^2\mathbb{E}[X_{0,1}^2],V_{\mathrm{\ell }}=\frac{1+a^2}{2}{n}_{\mathrm{\ell }}{\sigma }_{\mathrm{\ell }}^2{V}_{\mathrm{\ell }-1}(2\le \mathrm{\ell }\le L),V_L=V_1{\prod }_{\mathrm{\ell }=2}^L\frac{1+a^2}{2}{n}_{\mathrm{\ell }}{\sigma }_{\mathrm{\ell }}^2\mathrm{.}$ Thus ${\sigma }_{\mathrm{\ell }}^2=2\mathrm{/}((1+a^2)n_{\mathrm{\ell }})$ preserves the forward preactivation variance across the hidden layers.

For backward signals, take an objective $J$ differentiable as a function of the last activation and set $g_L={\mathrm{\nabla }}_{X_L}J(X_L)$. Specify a finite kink value $c_{\mathrm{\ell }}$ and put ${\eta }_{\mathrm{\ell },j}={\mathbf{1}}_{Z_{\mathrm{\ell },j}>0}+a{\mathbf{1}}_{Z_{\mathrm{\ell },j}<0}+c_{\mathrm{\ell }}{\mathbf{1}}_{Z_{\mathrm{\ell },j}=0}$, $h_{\mathrm{\ell }}={\eta }_{\mathrm{\ell }}\odot g_{\mathrm{\ell }}$, and $g_{\mathrm{\ell }-1}=W_{\mathrm{\ell }}^{\mathsf{T}}{h}_{\mathrm{\ell }}$. They are ordinary chain-rule gradients when all relevant preactivations are nonzero; when $a=1$ and $c_{\mathrm{\ell }}=1$ the activation is linear and this also holds at zeros. Otherwise a specified kink value is a backpropagation convention. If the coordinates of $Z_{\mathrm{\ell }}$ have common zero probability $p_{\mathrm{\ell }}$, the coordinates of $g_{\mathrm{\ell }}$ have common finite second moment $G_{\mathrm{\ell }}$, ${\eta }_{\mathrm{\ell }}$ is independent of $g_{\mathrm{\ell }}$, and $W_{\mathrm{\ell }}$ is independent of $h_{\mathrm{\ell }}$, then $G_{\mathrm{\ell }-1}=\mathrm{Var}(g_{\mathrm{\ell }-1,i})={\widehat{n}}_{\mathrm{\ell }}{\sigma }_{\mathrm{\ell }}^2{\kappa }_{\mathrm{\ell }}{G}_{\mathrm{\ell }},{\kappa }_{\mathrm{\ell }}=\frac{1+a^2}{2}(1-p_{\mathrm{\ell }})+c_{\mathrm{\ell }}^2{p}_{\mathrm{\ell }}\mathrm{.}$ The upstream coordinates have mean zero. If these conditional hypotheses hold for every $r\le \mathrm{\ell }\le L$, then $G_{r-1}=G_L{\prod }_{\mathrm{\ell }=r}^L{\widehat{n}}_{\mathrm{\ell }}{\sigma }_{\mathrm{\ell }}^2{\kappa }_{\mathrm{\ell }}\mathrm{.}$ In the source's atomless case $p_{\mathrm{\ell }}=0$, this retains the original backward factor $(1+a^2){\widehat{n}}_{\mathrm{\ell }}{\sigma }_{\mathrm{\ell }}^2\mathrm{/}2$ and the fan-out choice ${\sigma }_{\mathrm{\ell }}^2=2\mathrm{/}((1+a^2){\widehat{n}}_{\mathrm{\ell }})$. For a noncentered terminal signal $G_L$ is its second moment; for centered signals it is their variance. The source's forward and atomless backward weight variances agree exactly when $n_{\mathrm{\ell }}={\widehat{n}}_{\mathrm{\ell }}$. With atoms the conditional backward-preserving choice is $1\mathrm{/}({\widehat{n}}_{\mathrm{\ell }}{\kappa }_{\mathrm{\ell }})$ when ${\kappa }_{\mathrm{\ell }}>0$; if ${\kappa }_{\mathrm{\ell }}=0$ the upstream signal has zero variance for every finite weight variance. These are conditional backward statements, and no independence assumption or nonzero actual-gradient realization is inferred from initialization.

The atomless half-factor product is retained as the precise conditional calculation in He et al. §2.2; the forward product is valid for actual finite stacks satisfying the forward assumptions, with dependent coordinates and zero atoms allowed. The backward independence hypotheses require separate verification for an actual network. In particular, the ReLU specialization has the explicit nonzero-feasibility obstruction proved in the backward theorem, even at one hidden layer. No mean-field limit or empirical training claim supplies that gap.