Initialization and signal propagation

Let $n\ge 1$. Let $W_1,\dots ,W_n$ be mutually independent real random variables with finite second moments, means $\mathbb{E}[W_i]=0$, and variances ${\sigma }_i^2$. Let $X=(X_1,\dots ,X_n)$ be a real random vector whose coordinates have finite second moments, and suppose the random elements $W$ and $X$ are independent. The coordinates of $X$ need not be mutually independent. For $S={\sum }_{i=1}^n{W}_i{X}_i$, the variable $S$ is square-integrable, $\mathbb{E}[S]=0$, and $\mathrm{Var}(S)={\sum }_{i=1}^n{\sigma }_i^2\mathbb{E}[X_i^2]\mathrm{.}$

Let $Z$ be a real random variable whose law is symmetric about zero, meaning that $Z$ and $-Z$ have the same distribution. Write $p_0=\mathbb{P}(Z=0)$. Define the ReLU function $\rho (t)=\max (t,0)$ and, for $a\in \mathbb{R}$, define ${\varphi }_a(t)=t$ for $t>0$ and ${\varphi }_a(t)=at$ for $t\le 0$. Then $\mathbb{P}(Z>0)=\mathbb{P}(Z<0)=(1-p_0)\mathrm{/}2$. If $\mathbb{E}[Z^2]<\mathrm{\infty }$, the following ReLU moment identities hold: $\mathbb{E}[\rho (Z)]=\frac{1}{2}\mathbb{E}[\mathrm{\mid }Z\mathrm{\mid }],\mathbb{E}[\rho (Z{)}^2]=\frac{1}{2}\mathbb{E}[Z^2]\mathrm{.}$ The corresponding PReLU identities are $\mathbb{E}[{\varphi }_a(Z)]=\frac{1-a}{2}\mathbb{E}[\mathrm{\mid }Z\mathrm{\mid }],\mathbb{E}[{\varphi }_a(Z{)}^2]=\frac{1+a^2}{2}\mathbb{E}[Z^2]\mathrm{.}$ In particular, $\mathrm{Var}(\rho (Z))=\frac{1}{2}\mathbb{E}[Z^2]-\frac{1}{4}(\mathbb{E}[\mathrm{\mid }Z\mathrm{\mid }]{)}^2\mathrm{.}$ For any specified value $c\in \mathbb{R}$ at the kink, put ${\eta }_{a,c}(t)={\mathbf{1}}_{t>0}+a{\mathbf{1}}_{t<0}+c{\mathbf{1}}_{t=0}$. Then, without a moment hypothesis on $Z$, $\mathbb{E}[{\eta }_{a,c}(Z{)}^2]=\frac{1+a^2}{2}(1-p_0)+c^2{p}_0\mathrm{.}$ The zero-at-kink ReLU backpropagation gate is ${\eta }_{0,0}$, with second moment $(1-p_0)\mathrm{/}2$. It agrees almost surely with the ordinary ReLU derivative if $p_0=0$. A specified value at zero is a backpropagation convention, and is not an ordinary derivative there unless $a=1$ and $c=1$.

Let $n\ge 1$ and let $Z_1,\dots ,Z_n$ be independent and identically distributed real random variables whose law is symmetric about zero and has no atom at zero. Put $X_i=\mathrm{ReLU}(Z_i)$. For any real random variables $W_1,\dots ,W_n$ on the same probability space, define the next zero-bias preactivation $T={\sum }_{i=1}^n{W}_i{X}_i$. Then $\mathbb{P}(T=0)\ge 2^{-n}>0.$ In particular, this next preactivation cannot be atomless at zero, regardless of the distribution or dependence of the weights. The same conclusion holds under the weaker assumption that the $X_i$ are mutually independent and each $Z_i$ is symmetric with no atom at zero; independence of the $Z_i$ is not required in that formulation.

Inputs. A finite layered perceptron with positive integer widths $d_0,\dots ,d_L$, a real negative-side slope $a$, a mode $m\in \{\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{w}\mathrm{a}\mathrm{r}\mathrm{d},\mathrm{b}\mathrm{a}\mathrm{c}\mathrm{k}\mathrm{w}\mathrm{a}\mathrm{r}\mathrm{d}\}$, and a finite mutually independent family of real random variables ${\xi }_{\mathrm{\ell }ji}$ with a common symmetric law, mean zero, and variance one. The layer matrix has shape $d_{\mathrm{\ell }}\times d_{\mathrm{\ell }-1}$, with row index $j$ and column index $i$.

Output. Random weight matrices $W_{\mathrm{\ell }}$ and zero bias vectors $b_{\mathrm{\ell }}$.

Instructions. For each $1\le \mathrm{\ell }\le L$, set the fan-in $n_{\mathrm{\ell }}=d_{\mathrm{\ell }-1}$ and fan-out ${\widehat{n}}_{\mathrm{\ell }}=d_{\mathrm{\ell }}$. Set $v_{\mathrm{\ell }}=\frac{2}{(1+a^2)n_{\mathrm{\ell }}}(m=\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{w}\mathrm{a}\mathrm{r}\mathrm{d}),v_{\mathrm{\ell }}=\frac{2}{(1+a^2){\widehat{n}}_{\mathrm{\ell }}}(m=\mathrm{b}\mathrm{a}\mathrm{c}\mathrm{k}\mathrm{w}\mathrm{a}\mathrm{r}\mathrm{d})\mathrm{.}$ For every matrix entry set $W_{\mathrm{\ell },ji}=\sqrt{v_{\mathrm{\ell }}}{\xi }_{\mathrm{\ell }ji}$, and set $b_{\mathrm{\ell }}=0$. The case $a=0$ is ReLU; the piecewise-linear activation for general $a$ is $t$ for $t>0$ and $at$ for $t\le 0$.

The repeated forward recurrence concerns a layer whose inputs are outputs of a rectifier. A raw-input first affine layer has no preceding rectifier factor. If the input coordinates are square-integrable, centered, have common variance $q$, and the first-layer weights are independent of the input vector, then the finite weighted sum has mean zero. In its second moment, the diagonal terms each contribute $v_{1}q$ and the off-diagonal terms vanish because the weights are independent and centered. Thus $\mathrm{Var}(Z_{1,j})=n_1{v}_{1}q$, so $v_1=1\mathrm{/}{n}_1$ preserves $q$. He et al. also use the rectifier scale at that first layer for simplicity; this changes one boundary factor, not the repeated hidden-layer factor. Fan-in mode and fan-out mode target different directions and need not preserve both when layer widths differ. The paper's depth and ImageNet accuracy comparisons are reported experiments, not consequences of this definition and not guarantees of training success or generalization.

The mode names specify the source's forward and conditional backward moment targets. They do not assert that finite-network gradients are independent of their weights or gates. A positive-probability zero preactivation requires a specified kink convention for backpropagation; it does not have an ordinary ReLU derivative. Consequently the backward scale alone supplies no nonzero actual-gradient preservation guarantee.

Let $L\ge 2$ and let $d_0,\dots ,d_L$ be positive integers. For each $1\le \mathrm{\ell }\le L$, let $W_{\mathrm{\ell }}\in {\mathbb{R}}^{d_{\mathrm{\ell }}\times d_{\mathrm{\ell }-1}}$ have mutually independent, identically distributed entries with a symmetric law, mean zero, and finite variance ${\sigma }_{\mathrm{\ell }}^2$, and suppose $W_{\mathrm{\ell }}$ is independent of the input activation vector $X_{\mathrm{\ell }-1}$. Set the fan-in $n_{\mathrm{\ell }}=d_{\mathrm{\ell }-1}$, set every bias to zero, and define $Z_{\mathrm{\ell }}=W_{\mathrm{\ell }}{X}_{\mathrm{\ell }-1}$ and $X_{\mathrm{\ell }}=\rho (Z_{\mathrm{\ell }})$ coordinatewise, where $\rho (t)=\max (t,0)$. Assume for every layer that the coordinates of $X_{\mathrm{\ell }-1}$ have a common finite second moment; they need not be independent or identically distributed. Also assume that each preactivation coordinate $Z_{\mathrm{\ell },j}$ has a symmetric law; an atom at zero is allowed in this forward second-moment recurrence. Then, for every $2\le \mathrm{\ell }\le L$ and $1\le j\le d_{\mathrm{\ell }}$, $\mathrm{Var}(Z_{\mathrm{\ell },j})=\frac{1}{2}{n}_{\mathrm{\ell }}{\sigma }_{\mathrm{\ell }}^2\mathrm{Var}(Z_{\mathrm{\ell }-1,1})\mathrm{.}$ At the raw-input boundary, the exact base relation is $\mathrm{Var}(Z_{1,j})=n_1{\sigma }_1^2\mathbb{E}[X_{0,1}^2];$ if $X_{0,1}$ is centered, its second moment here is its variance. In particular, writing $V_{\mathrm{\ell }}=\mathrm{Var}(Z_{\mathrm{\ell },1})$, the exact finite-depth formula is $V_L=V_1{\prod }_{\mathrm{\ell }=2}^L\frac{n_{\mathrm{\ell }}{\sigma }_{\mathrm{\ell }}^2}{2}\mathrm{.}$ Thus ${\sigma }_{\mathrm{\ell }}^2=2\mathrm{/}{n}_{\mathrm{\ell }}$ for $\mathrm{\ell }\ge 2$ gives $V_L=V_1$, with zero atoms allowed at every depth. This is a statement about ensemble second moments, not about variance along every sampled network or training success.

The equal-second-moment assumption allows the dependence between coordinates that naturally arises after averaging over a random previous layer. For a concrete finite-depth model satisfying all these hypotheses, take every width to be one, $X_0=1$, and mutually independent weights $W_{\mathrm{\ell }}=\pm \sqrt{v_{\mathrm{\ell }}}$ with equal probabilities, where $v_{\mathrm{\ell }}>0$. At each layer the fresh weight is independent of the preceding activation and its random sign makes the preactivation symmetric. The event $W_1<0$ already forces all subsequent preactivations to vanish, yet $V_1=v_1$ and $V_L=v_1{\prod }_{\mathrm{\ell }=2}^L(v_{\mathrm{\ell }}\mathrm{/}2)$ hold exactly. Thus atomlessness is unnecessary for finite-depth forward propagation.

Consider a finite rectified stack in a fully connected layered perceptron, with widths $d_0,\dots ,d_L$, zero biases, $Z_{\mathrm{\ell }}=W_{\mathrm{\ell }}{X}_{\mathrm{\ell }-1}$, and $X_{\mathrm{\ell }}=\rho (Z_{\mathrm{\ell }})$, where $\rho (t)=\max (t,0)$. For an objective $J$ differentiable as a function of the last activation, set $g_L={\mathrm{\nabla }}_{X_L}J(X_L)$ and define backward signals by the zero-at-kink convention ${\gamma }_{\mathrm{\ell },j}={\mathbf{1}}_{Z_{\mathrm{\ell },j}>0}$, $h_{\mathrm{\ell }}={\gamma }_{\mathrm{\ell }}\odot g_{\mathrm{\ell }}$, and $g_{\mathrm{\ell }-1}=W_{\mathrm{\ell }}^{\mathsf{T}}{h}_{\mathrm{\ell }}$. At realizations where all relevant preactivations are nonzero, these are ordinary chain-rule gradients. At zeros they are specified backpropagation signals; no ordinary derivative of ReLU at zero is asserted.

Without any independence assumptions, fix a layer $\mathrm{\ell }$ and suppose every $T_{ji}=W_{\mathrm{\ell },ji}{\gamma }_{\mathrm{\ell },j}{g}_{\mathrm{\ell },j}$ is square-integrable. Then $\mathbb{E}[g_{\mathrm{\ell }-1,i}]={\sum }_j\mathbb{E}[T_{ji}],\mathrm{Var}(g_{\mathrm{\ell }-1,i})={\sum }_j\mathrm{Var}(T_{ji})+2{\sum }_{j<k}\mathrm{Cov}(T_{ji},T_{ki})\mathrm{.}$ This is an exact atom-aware identity for the realized network signals. For any specified bounded measurable random gate vector $q_{\mathrm{\ell }}$ in place of ${\gamma }_{\mathrm{\ell }}$, define $T_{ji}=W_{\mathrm{\ell },ji}{q}_{\mathrm{\ell },j}{g}_{\mathrm{\ell },j}$. If every redefined $T_{ji}$ is square-integrable, the same mean and covariance formulas hold.

For the conditional recurrence, additionally assume at the layers in question that the entries of $W_{\mathrm{\ell }}$ are independent with a common symmetric centered law of finite variance ${\sigma }_{\mathrm{\ell }}^2$, that $Z_{\mathrm{\ell },j}$ is symmetric with common zero probability $p_{\mathrm{\ell }}$, and that the coordinates of $g_{\mathrm{\ell }}$ have common finite second moment $G_{\mathrm{\ell }}$. Assume the gate vector ${\gamma }_{\mathrm{\ell }}$ is independent of $g_{\mathrm{\ell }}$ and $W_{\mathrm{\ell }}$ is independent of $h_{\mathrm{\ell }}$. Write the fan-out as ${\widehat{n}}_{\mathrm{\ell }}=d_{\mathrm{\ell }}$. Then $\mathbb{E}[g_{\mathrm{\ell }-1,i}]=0,G_{\mathrm{\ell }-1}=\mathrm{Var}(g_{\mathrm{\ell }-1,i})=\frac{1-p_{\mathrm{\ell }}}{2}{\widehat{n}}_{\mathrm{\ell }}{\sigma }_{\mathrm{\ell }}^2{G}_{\mathrm{\ell }}\mathrm{.}$ If these hypotheses hold for every $r\le \mathrm{\ell }\le L$, then $G_{r-1}=G_L{\prod }_{\mathrm{\ell }=r}^L\frac{1-p_{\mathrm{\ell }}}{2}{\widehat{n}}_{\mathrm{\ell }}{\sigma }_{\mathrm{\ell }}^2\mathrm{.}$ In particular, the source's atomless conditional case $p_{\mathrm{\ell }}=0$ gives $\mathrm{Var}(g_{\mathrm{\ell }-1,i})=\frac{1}{2}{\widehat{n}}_{\mathrm{\ell }}{\sigma }_{\mathrm{\ell }}^2\mathbb{E}[g_{\mathrm{\ell },1}^2]$ and the corresponding product of half factors. If the downstream coordinate is centered, $G_{\mathrm{\ell }}$ is its variance. The conditional fan-out choice ${\sigma }_{\mathrm{\ell }}^2=2\mathrm{/}{\widehat{n}}_{\mathrm{\ell }}$ makes each atomless product factor one.

These conditional identities do not establish that a nonzero actual-network gradient satisfies their hypotheses. Indeed, if $\mathrm{\ell }\ge 2$, $X_{\mathrm{\ell }-1}$ is a ReLU output and ${\sigma }_{\mathrm{\ell }}^2>0$, the assumption $W_{\mathrm{\ell }}\perp h_{\mathrm{\ell }}$ already forces $h_{\mathrm{\ell }}=0$ almost surely. If also $p_{\mathrm{\ell }}<1$ and gate independence holds, it forces $G_{\mathrm{\ell }}=0$. Thus the nonzero atomless backward preservation model with these full independence assumptions is infeasible for such an actual ReLU layer; the conditional implication and product remain valid, and the general covariance identity above supplies the unconditional finite-network formula.

He et al. §2.2, Eqs. (14)–(16), multiply the atomless half factors under the stated weight/incoming-gradient and gate/downstream-gradient independence assumptions. Those are a conditional moment calculation, not a proof of their realization by a nonzero finite ReLU gradient. Zero atoms also invalidate the ordinary derivative at those realizations; choosing the zero-at-kink convention addresses the algorithmic signal but does not establish the independence assumptions.

Let $a\in \mathbb{R}$ and consider a finite rectified stack with zero biases, widths $d_0,\dots ,d_L$, $Z_{\mathrm{\ell }}=W_{\mathrm{\ell }}{X}_{\mathrm{\ell }-1}$, and $X_{\mathrm{\ell }}={\varphi }_a(Z_{\mathrm{\ell }})$, where ${\varphi }_a(t)=t$ for $t>0$ and ${\varphi }_a(t)=at$ for $t\le 0$. Put $n_{\mathrm{\ell }}=d_{\mathrm{\ell }-1}$ and ${\widehat{n}}_{\mathrm{\ell }}=d_{\mathrm{\ell }}$. Suppose each $W_{\mathrm{\ell }}$ has independent entries with a common symmetric centered law of finite variance ${\sigma }_{\mathrm{\ell }}^2$ and is independent of $X_{\mathrm{\ell }-1}$. Suppose each layer input has coordinates with a common finite second moment, and all preactivation coordinates are symmetric about zero. Input coordinates may be dependent, and zero atoms are allowed. Then, writing $V_{\mathrm{\ell }}=\mathrm{Var}(Z_{\mathrm{\ell },1})$, $V_1=n_1{\sigma }_1^2\mathbb{E}[X_{0,1}^2],V_{\mathrm{\ell }}=\frac{1+a^2}{2}{n}_{\mathrm{\ell }}{\sigma }_{\mathrm{\ell }}^2{V}_{\mathrm{\ell }-1}(2\le \mathrm{\ell }\le L),V_L=V_1{\prod }_{\mathrm{\ell }=2}^L\frac{1+a^2}{2}{n}_{\mathrm{\ell }}{\sigma }_{\mathrm{\ell }}^2\mathrm{.}$ Thus ${\sigma }_{\mathrm{\ell }}^2=2\mathrm{/}((1+a^2)n_{\mathrm{\ell }})$ preserves the forward preactivation variance across the hidden layers.

For backward signals, take an objective $J$ differentiable as a function of the last activation and set $g_L={\mathrm{\nabla }}_{X_L}J(X_L)$. Specify a finite kink value $c_{\mathrm{\ell }}$ and put ${\eta }_{\mathrm{\ell },j}={\mathbf{1}}_{Z_{\mathrm{\ell },j}>0}+a{\mathbf{1}}_{Z_{\mathrm{\ell },j}<0}+c_{\mathrm{\ell }}{\mathbf{1}}_{Z_{\mathrm{\ell },j}=0}$, $h_{\mathrm{\ell }}={\eta }_{\mathrm{\ell }}\odot g_{\mathrm{\ell }}$, and $g_{\mathrm{\ell }-1}=W_{\mathrm{\ell }}^{\mathsf{T}}{h}_{\mathrm{\ell }}$. They are ordinary chain-rule gradients when all relevant preactivations are nonzero; when $a=1$ and $c_{\mathrm{\ell }}=1$ the activation is linear and this also holds at zeros. Otherwise a specified kink value is a backpropagation convention. If the coordinates of $Z_{\mathrm{\ell }}$ have common zero probability $p_{\mathrm{\ell }}$, the coordinates of $g_{\mathrm{\ell }}$ have common finite second moment $G_{\mathrm{\ell }}$, ${\eta }_{\mathrm{\ell }}$ is independent of $g_{\mathrm{\ell }}$, and $W_{\mathrm{\ell }}$ is independent of $h_{\mathrm{\ell }}$, then $G_{\mathrm{\ell }-1}=\mathrm{Var}(g_{\mathrm{\ell }-1,i})={\widehat{n}}_{\mathrm{\ell }}{\sigma }_{\mathrm{\ell }}^2{\kappa }_{\mathrm{\ell }}{G}_{\mathrm{\ell }},{\kappa }_{\mathrm{\ell }}=\frac{1+a^2}{2}(1-p_{\mathrm{\ell }})+c_{\mathrm{\ell }}^2{p}_{\mathrm{\ell }}\mathrm{.}$ The upstream coordinates have mean zero. If these conditional hypotheses hold for every $r\le \mathrm{\ell }\le L$, then $G_{r-1}=G_L{\prod }_{\mathrm{\ell }=r}^L{\widehat{n}}_{\mathrm{\ell }}{\sigma }_{\mathrm{\ell }}^2{\kappa }_{\mathrm{\ell }}\mathrm{.}$ In the source's atomless case $p_{\mathrm{\ell }}=0$, this retains the original backward factor $(1+a^2){\widehat{n}}_{\mathrm{\ell }}{\sigma }_{\mathrm{\ell }}^2\mathrm{/}2$ and the fan-out choice ${\sigma }_{\mathrm{\ell }}^2=2\mathrm{/}((1+a^2){\widehat{n}}_{\mathrm{\ell }})$. For a noncentered terminal signal $G_L$ is its second moment; for centered signals it is their variance. The source's forward and atomless backward weight variances agree exactly when $n_{\mathrm{\ell }}={\widehat{n}}_{\mathrm{\ell }}$. With atoms the conditional backward-preserving choice is $1\mathrm{/}({\widehat{n}}_{\mathrm{\ell }}{\kappa }_{\mathrm{\ell }})$ when ${\kappa }_{\mathrm{\ell }}>0$; if ${\kappa }_{\mathrm{\ell }}=0$ the upstream signal has zero variance for every finite weight variance. These are conditional backward statements, and no independence assumption or nonzero actual-gradient realization is inferred from initialization.

The atomless half-factor product is retained as the precise conditional calculation in He et al. §2.2; the forward product is valid for actual finite stacks satisfying the forward assumptions, with dependent coordinates and zero atoms allowed. The backward independence hypotheses require separate verification for an actual network. In particular, the ReLU specialization has the explicit nonzero-feasibility obstruction proved in the backward theorem, even at one hidden layer. No mean-field limit or empirical training claim supplies that gap.