Connected activation-pattern regions

Consider a layered ReLU network with positive integer input dimension $n_0$ and positive integer hidden widths $n_1,\dots ,n_L$. For a binary pattern $s=(s_i^{\mathrm{\ell }})$, set $s_i^{\mathrm{\ell }}=1$ exactly when the corresponding preactivation is strictly positive, and set $s_i^{\mathrm{\ell }}=0$ when it is nonpositive, so a unit with zero output is inactive. Its activation-pattern cell is the set of inputs inducing that pattern. Once a pattern is fixed, every preactivation is affine in the input and the pattern constraints are affine half-space conditions; hence each cell is convex and connected when nonempty. A connected activation-pattern region is a nonempty activation-pattern cell with nonempty interior in ${\mathbb{R}}^{n_0}$. On each such region the hidden pattern is fixed and the network is affine. Zero preactivations are assigned to the inactive side, including a preactivation that vanishes identically on a cell; zero sets are not removed from the input space. Distinct full-dimensional patterns are counted separately even when they give the same affine formula.

ReLU feedforward networks and complexity conventions

Serra et al., §2: activation-pattern regions and zero-inactive convention

Sullivan, §5.2: ReLU activation cells