Rectifier variance-preserving initialization

Inputs. A finite layered perceptron with positive integer widths $d_0,\dots ,d_L$, a real negative-side slope $a$, a mode $m\in \{\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{w}\mathrm{a}\mathrm{r}\mathrm{d},\mathrm{b}\mathrm{a}\mathrm{c}\mathrm{k}\mathrm{w}\mathrm{a}\mathrm{r}\mathrm{d}\}$, and a finite mutually independent family of real random variables ${\xi }_{\mathrm{\ell }ji}$ with a common symmetric law, mean zero, and variance one. The layer matrix has shape $d_{\mathrm{\ell }}\times d_{\mathrm{\ell }-1}$, with row index $j$ and column index $i$.

Output. Random weight matrices $W_{\mathrm{\ell }}$ and zero bias vectors $b_{\mathrm{\ell }}$.

Instructions. For each $1\le \mathrm{\ell }\le L$, set the fan-in $n_{\mathrm{\ell }}=d_{\mathrm{\ell }-1}$ and fan-out ${\widehat{n}}_{\mathrm{\ell }}=d_{\mathrm{\ell }}$. Set $v_{\mathrm{\ell }}=\frac{2}{(1+a^2)n_{\mathrm{\ell }}}(m=\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{w}\mathrm{a}\mathrm{r}\mathrm{d}),v_{\mathrm{\ell }}=\frac{2}{(1+a^2){\widehat{n}}_{\mathrm{\ell }}}(m=\mathrm{b}\mathrm{a}\mathrm{c}\mathrm{k}\mathrm{w}\mathrm{a}\mathrm{r}\mathrm{d})\mathrm{.}$ For every matrix entry set $W_{\mathrm{\ell },ji}=\sqrt{v_{\mathrm{\ell }}}{\xi }_{\mathrm{\ell }ji}$, and set $b_{\mathrm{\ell }}=0$. The case $a=0$ is ReLU; the piecewise-linear activation for general $a$ is $t$ for $t>0$ and $at$ for $t\le 0$.

The repeated forward recurrence concerns a layer whose inputs are outputs of a rectifier. A raw-input first affine layer has no preceding rectifier factor. If the input coordinates are square-integrable, centered, have common variance $q$, and the first-layer weights are independent of the input vector, then the finite weighted sum has mean zero. In its second moment, the diagonal terms each contribute $v_{1}q$ and the off-diagonal terms vanish because the weights are independent and centered. Thus $\mathrm{Var}(Z_{1,j})=n_1{v}_{1}q$, so $v_1=1\mathrm{/}{n}_1$ preserves $q$. He et al. also use the rectifier scale at that first layer for simplicity; this changes one boundary factor, not the repeated hidden-layer factor. Fan-in mode and fan-out mode target different directions and need not preserve both when layer widths differ. The paper's depth and ImageNet accuracy comparisons are reported experiments, not consequences of this definition and not guarantees of training success or generalization.

The mode names specify the source's forward and conditional backward moment targets. They do not assert that finite-network gradients are independent of their weights or gates. A positive-probability zero preactivation requires a specified kink convention for backpropagation; it does not have an ordinary ReLU derivative. Consequently the backward scale alone supplies no nonzero actual-gradient preservation guarantee.