Initialization and signal propagation — Examples

Compare two one-unit preactivations. In the first, the input is $X=1$, the bias is zero, and the weight $W$ equals $-1$ with probability $2\mathrm{/}3$ and $2$ with probability $1\mathrm{/}3$. In the second, the input $X$ is uniform on $[-2,2]$, the independent weight $W$ equals $-1$ or $1$ with equal probabilities, and the bias is $1$. In each case compute $\mathbb{E}[\mathrm{ReLU}(Z{)}^2]$ and compare it with $\frac{1}{2}\mathrm{Var}(Z)$, where $Z=WX+b$. The first case tests whether a zero-mean but nonsymmetric weight law suffices; the second tests the effect of a nonzero bias despite symmetric weights and input.

Let $Z_1,Z_2$ be independent and identically distributed symmetric real random variables with no atom at zero, set $X_i=\mathrm{ReLU}(Z_i)$, and form a second-layer zero-bias preactivation $T=W_1{X}_1+W_2{X}_2$ with arbitrary real weights. Although neither $Z_i$ has an atom at zero, the event that both rectified inputs vanish has probability $1\mathrm{/}4$ and forces $T=0$.

For a complete finite-network signal calculation, also take a two-layer width-one ReLU stack with $X_0=1$, mutually independent $W_1,W_2\in \{-\sqrt{2},\sqrt{2}\}$ with equal sign probabilities, and an independent random sign $R\in \{-1,1\}$. Take objective $J(X_2)=R{X}_2$, so $g_2=R$, and use zero-at-kink backpropagation. Then $\mathrm{Var}(Z_1)=\mathrm{Var}(Z_2)=2$ but $\mathbb{P}(Z_2=0)=1\mathrm{/}2$ and $\mathrm{Var}(g_1)=1\mathrm{/}2$. The atomless half-factor coefficient would instead give $1$. The general conditional recurrence allows zero atoms, but its weight/gated-signal independence hypothesis fails in this realized stack. Its half-factor special case also requires $p_2=0$, whereas here $p_2=1\mathrm{/}2$.

Compare the source's forward and atomless conditional backward recurrence coefficients for fan-in 64 and fan-out 128. The respective coefficients are $64{\sigma }^2\mathrm{/}2$ and $128{\sigma }^2\mathrm{/}2$. Compute their factors at the two initializer choices ${\sigma }^2=2\mathrm{/}64$ and ${\sigma }^2=2\mathrm{/}128$. For a preceding preactivation variance $V$ the forward result is a multiple of $V$; for a downstream second moment $G$ the conditional backward result is a multiple of $G$ (and of its variance when centered). These coefficient comparisons do not assert a nonzero actual ReLU gradient model satisfying the full backward independence assumptions: at a nondegenerate hidden ReLU layer those assumptions force $G=0$.

The top row represents the forward linear map and ReLU; the lower row shows the reverse-mode maps with $D_{\mathrm{\ell }}=\mathrm{diag}({\mathbf{1}}_{Z_{\mathrm{\ell },j}>0})$. Each box denotes the full vector of the stated dimension, and the arrow labelled $W_{\mathrm{\ell }}$ denotes the $128\times 64$ matrix, not a single scalar edge. Thus the forward fan-in is 64 and the backward fan-out is 128. The diagram illustrates these dimensions; the verification below uses the recurrence formulas directly.

diagram failed to render:
\begin{tikzpicture}[>=stealth]
  \node[draw,rounded corners] (x) at (-4,1) {$X_{\ell-1}\in\mathbb R^{64}$};
  \node[draw,rounded corners] (z) at (0,1) {$Z_\ell\in\mathbb R^{128}$};
  \node[draw,rounded corners] (y) at (4,1) {$X_\ell\in\mathbb R^{128}$};
  \draw[->] (x) -- node[above] {$W_\ell$} (z);
  \draw[->] (z) -- node[above] {$\rho$} (y);
  \node[draw,rounded corners] (g0) at (-4,-1) {$g_{\ell-1}\in\mathbb R^{64}$};
  \node[draw,rounded corners] (h) at (0,-1) {$h_\ell\in\mathbb R^{128}$};
  \node[draw,rounded corners] (g1) at (4,-1) {$g_\ell\in\mathbb R^{128}$};
  \draw[<-] (g0) -- node[below] {$W_\ell^{\mathsf T}$} (h);
  \draw[<-] (h) -- node[below] {$D_\ell$} (g1);
\end{tikzpicture}

For the PReLU source scale calculation, take negative-side slope $a=1\mathrm{/}2$, fan-in $n=5$, and fan-out $\widehat{n}=10$. Compare the exact forward recurrence coefficient $(1+a^2)n{\sigma }^2\mathrm{/}2$ with the atomless conditional backward coefficient $(1+a^2)\widehat{n}{\sigma }^2\mathrm{/}2$. Compute the two unit-coefficient weight variances and determine whether they agree. The backward coefficient has the corollary's independence assumptions; it is not asserted to describe an arbitrary actual-network gradient, and a zero atom with a specified kink gate changes its factor.