ReLU backward gradient-variance recurrence

Consider a finite rectified stack in a fully connected layered perceptron, with widths $d_0,\dots ,d_L$, zero biases, $Z_{\mathrm{\ell }}=W_{\mathrm{\ell }}{X}_{\mathrm{\ell }-1}$, and $X_{\mathrm{\ell }}=\rho (Z_{\mathrm{\ell }})$, where $\rho (t)=\max (t,0)$. For an objective $J$ differentiable as a function of the last activation, set $g_L={\mathrm{\nabla }}_{X_L}J(X_L)$ and define backward signals by the zero-at-kink convention ${\gamma }_{\mathrm{\ell },j}={\mathbf{1}}_{Z_{\mathrm{\ell },j}>0}$, $h_{\mathrm{\ell }}={\gamma }_{\mathrm{\ell }}\odot g_{\mathrm{\ell }}$, and $g_{\mathrm{\ell }-1}=W_{\mathrm{\ell }}^{\mathsf{T}}{h}_{\mathrm{\ell }}$. At realizations where all relevant preactivations are nonzero, these are ordinary chain-rule gradients. At zeros they are specified backpropagation signals; no ordinary derivative of ReLU at zero is asserted.

Without any independence assumptions, fix a layer $\mathrm{\ell }$ and suppose every $T_{ji}=W_{\mathrm{\ell },ji}{\gamma }_{\mathrm{\ell },j}{g}_{\mathrm{\ell },j}$ is square-integrable. Then $\mathbb{E}[g_{\mathrm{\ell }-1,i}]={\sum }_j\mathbb{E}[T_{ji}],\mathrm{Var}(g_{\mathrm{\ell }-1,i})={\sum }_j\mathrm{Var}(T_{ji})+2{\sum }_{j<k}\mathrm{Cov}(T_{ji},T_{ki})\mathrm{.}$ This is an exact atom-aware identity for the realized network signals. For any specified bounded measurable random gate vector $q_{\mathrm{\ell }}$ in place of ${\gamma }_{\mathrm{\ell }}$, define $T_{ji}=W_{\mathrm{\ell },ji}{q}_{\mathrm{\ell },j}{g}_{\mathrm{\ell },j}$. If every redefined $T_{ji}$ is square-integrable, the same mean and covariance formulas hold.

For the conditional recurrence, additionally assume at the layers in question that the entries of $W_{\mathrm{\ell }}$ are independent with a common symmetric centered law of finite variance ${\sigma }_{\mathrm{\ell }}^2$, that $Z_{\mathrm{\ell },j}$ is symmetric with common zero probability $p_{\mathrm{\ell }}$, and that the coordinates of $g_{\mathrm{\ell }}$ have common finite second moment $G_{\mathrm{\ell }}$. Assume the gate vector ${\gamma }_{\mathrm{\ell }}$ is independent of $g_{\mathrm{\ell }}$ and $W_{\mathrm{\ell }}$ is independent of $h_{\mathrm{\ell }}$. Write the fan-out as ${\widehat{n}}_{\mathrm{\ell }}=d_{\mathrm{\ell }}$. Then $\mathbb{E}[g_{\mathrm{\ell }-1,i}]=0,G_{\mathrm{\ell }-1}=\mathrm{Var}(g_{\mathrm{\ell }-1,i})=\frac{1-p_{\mathrm{\ell }}}{2}{\widehat{n}}_{\mathrm{\ell }}{\sigma }_{\mathrm{\ell }}^2{G}_{\mathrm{\ell }}\mathrm{.}$ If these hypotheses hold for every $r\le \mathrm{\ell }\le L$, then $G_{r-1}=G_L{\prod }_{\mathrm{\ell }=r}^L\frac{1-p_{\mathrm{\ell }}}{2}{\widehat{n}}_{\mathrm{\ell }}{\sigma }_{\mathrm{\ell }}^2\mathrm{.}$ In particular, the source's atomless conditional case $p_{\mathrm{\ell }}=0$ gives $\mathrm{Var}(g_{\mathrm{\ell }-1,i})=\frac{1}{2}{\widehat{n}}_{\mathrm{\ell }}{\sigma }_{\mathrm{\ell }}^2\mathbb{E}[g_{\mathrm{\ell },1}^2]$ and the corresponding product of half factors. If the downstream coordinate is centered, $G_{\mathrm{\ell }}$ is its variance. The conditional fan-out choice ${\sigma }_{\mathrm{\ell }}^2=2\mathrm{/}{\widehat{n}}_{\mathrm{\ell }}$ makes each atomless product factor one.

These conditional identities do not establish that a nonzero actual-network gradient satisfies their hypotheses. Indeed, if $\mathrm{\ell }\ge 2$, $X_{\mathrm{\ell }-1}$ is a ReLU output and ${\sigma }_{\mathrm{\ell }}^2>0$, the assumption $W_{\mathrm{\ell }}\perp h_{\mathrm{\ell }}$ already forces $h_{\mathrm{\ell }}=0$ almost surely. If also $p_{\mathrm{\ell }}<1$ and gate independence holds, it forces $G_{\mathrm{\ell }}=0$. Thus the nonzero atomless backward preservation model with these full independence assumptions is infeasible for such an actual ReLU layer; the conditional implication and product remain valid, and the general covariance identity above supplies the unconditional finite-network formula.

He et al. §2.2, Eqs. (14)–(16), multiply the atomless half factors under the stated weight/incoming-gradient and gate/downstream-gradient independence assumptions. Those are a conditional moment calculation, not a proof of their realization by a nonzero finite ReLU gradient. Zero atoms also invalidate the ordinary derivative at those realizations; choosing the zero-at-kink convention addresses the algorithmic signal but does not establish the independence assumptions.