Fan-in and fan-out target different signal directions

Compare the source's forward and atomless conditional backward recurrence coefficients for fan-in 64 and fan-out 128. The respective coefficients are $64{\sigma }^2\mathrm{/}2$ and $128{\sigma }^2\mathrm{/}2$. Compute their factors at the two initializer choices ${\sigma }^2=2\mathrm{/}64$ and ${\sigma }^2=2\mathrm{/}128$. For a preceding preactivation variance $V$ the forward result is a multiple of $V$; for a downstream second moment $G$ the conditional backward result is a multiple of $G$ (and of its variance when centered). These coefficient comparisons do not assert a nonzero actual ReLU gradient model satisfying the full backward independence assumptions: at a nondegenerate hidden ReLU layer those assumptions force $G=0$.

The top row represents the forward linear map and ReLU; the lower row shows the reverse-mode maps with $D_{\mathrm{\ell }}=\mathrm{diag}({\mathbf{1}}_{Z_{\mathrm{\ell },j}>0})$. Each box denotes the full vector of the stated dimension, and the arrow labelled $W_{\mathrm{\ell }}$ denotes the $128\times 64$ matrix, not a single scalar edge. Thus the forward fan-in is 64 and the backward fan-out is 128. The diagram illustrates these dimensions; the verification below uses the recurrence formulas directly.