A width-n ReLU block makes n+1 full folds

Set $t_0=0$, $t_{n+1}=1$, and $t_i=(2i-1)\mathrm{/}(2n+1)$ for $1\le i\le n$. Put $y_i=0$ for even $i$ and $y_i=1$ for odd $i$, and define $s_i=(y_{i+1}-y_i)\mathrm{/}(t_{i+1}-t_i)$ for $0\le i\le n$. The piecewise-affine interpolant through $(t_i,y_i)$ maps every interval bijectively onto $[0,1]$. Its first slope is $s_0=2n+1$, while $s_2=(2n+1)\mathrm{/}2$ and $s_3=-(2n+1)\mathrm{/}2$, so $s_3-s_2=-s_0$.

Use hidden units $h_i(x)=\rho (x-t_i)$ for $i\mathrm{\ne }3$ and $h_3(x)=\rho (t_3-x)$. Give unit $i$ output coefficient $a_i=s_i-s_{i-1}$, including $a_3=s_3-s_2=-s_0$, and set output bias $d=-a_3{t}_3$. Before $t_1$ only $h_3$ is active, so the output is $d+a_3(t_3-x)=s_{0}x$. At every $t_i$, the slope changes by $a_i$: for $i\mathrm{\ne }3$ this is the right-facing ReLU slope jump, and for $i=3$ it is the left-facing ReLU slope jump. Thus the network output has the same initial value and slope jumps as the interpolant in step 1.1, so it equals that interpolant on $[0,1]$. This uses exactly $n$ hidden ReLU units and one affine output.

Each fold block's output is affine in the current hidden layer, so it can be substituted into the affine preactivations of the next adjacent layer without adding an intervening layer. At each knot $t_i$, exactly the corresponding hidden unit changes activation, so the $n+1$ intervals have distinct activation patterns. Each existing interval is cut into $n_{\mathrm{\ell }+1}+1$ subintervals, each mapped bijectively onto $[0,1]$; the new layer's patterns distinguish these subintervals, while earlier layer patterns distinguish their parent intervals. Induction over the blocks gives ${\prod }_{\mathrm{\ell }=1}^L(n_{\mathrm{\ell }}+1)$ intervals, each a distinct full-dimensional activation region.