Signed-square ReLU feature parameterization

Let $d\ge 2$, let ${\rho }_X$ be a Borel probability measure on ${\mathbb{R}}^{d-1}$ with finite second moment, and define $s(t)=t\mathrm{\mid }t\mathrm{\mid }$ and $\sigma (t)=\max (t,0)$. For $\theta =({\theta }_1,\dots ,{\theta }_d)\in {\mathbb{R}}^d$, define the signed-square ReLU feature in $L^2({\rho }_X)$ by

$\mathrm{\Phi }(\theta )(x)=\sigma \text{\hspace{0.17em}⁣}({\sum }_{j=1}^{d-1}s({\theta }_j)x_j+s({\theta }_d))\mathrm{.}$

The two-copy parameterization uses this feature on one copy of ${\mathbb{R}}^d$ and its negative on the other copy; nonnegative measures on the disjoint union represent signed linear combinations of these features. The scalar signed-square map is continuously differentiable with derivative $s^{\mathrm{\prime }}(t)=2\mathrm{\mid }t\mathrm{\mid }$. This definition asserts differentiability of the resulting $L^2$ feature map only when its data law has the additional regularity needed for that conclusion; it does not assert that its differential is locally Lipschitz.

Feature-measure energy and its regularity assumptions

Single-hidden-layer network realization

Chizat and Bach, Section 4.2 and Appendix D.4.2