Mean-field training and feature learning — Examples

The path consisting of the radius-$r_a$ circular arc from $r_a{e}_1$ to $r_a{e}_2$, the radial segment from $r_a{e}_2$ to $r_b{e}_2$, and the radius-$r_b$ circular arc from $r_b{e}_2$ to $r_b{e}_1$ is continuous after linear reparameterization of its three pieces. Its endpoints lie on the required spheres and every point on it has norm at least $r_a>0$.

The path therefore avoids $\mathrm{spt}{\mu }_0=\{0\}$ while joining the two spheres. By the definition of separation, this support does not separate them.

At $(a,\theta )=(1,0)$, $\phi (0)=(1,0)=y$, so [F1] gives risk $R(y)=0$. Since $R$ is a squared norm divided by two, no objective value is negative; hence the moving model has minimum risk zero.

With the feature fixed at $\pi \mathrm{/}2$, $\phi (\pi \mathrm{/}2)=(0,1)$ and $\mathrm{\parallel }a\phi (\pi \mathrm{/}2)-y{\mathrm{\parallel }}^2=a^2+1$. Therefore $R(a\phi (\pi \mathrm{/}2))=(a^2+1)\mathrm{/}2\ge 1\mathrm{/}2$, with equality at $a=0$. This gives the claimed fixed-feature minimum.

On a small neighborhood of $(1,1)$ with $w,b>0$, the root $c(w,b)$ is a smooth function of parameters. For an increment $h=(h_1,h_2)$, $r_{(w,b)+h}(x)-r_{w,b}(x)=2w{h}_{1}x+2b{h}_2+h_1^{2}x+h_2^2\mathrm{.}$ The candidate differential is $D{\mathrm{\Phi }}_{(w,b)}h={\mathbf{1}}_{x>c(w,b)}(2w{h}_{1}x+2b{h}_2)$. Outside the interval between $c(w,b)$ and $c((w,b)+h)$, the two preactivations have the same sign; there the ReLU remainder is either zero or $h_1^{2}x+h_2^2$, whose $L^2(pdx)$ norm is $O(\mathrm{\mid }h{\mathrm{\mid }}^2)$ by the finite second moment in [F1]. The interval between the two roots has length $O(\mathrm{\mid }h\mathrm{\mid })$ and lies in a fixed bounded neighborhood of $-1$; on it the pointwise remainder is $O(\mathrm{\mid }h\mathrm{\mid })$, so its $L^2(pdx)$ norm is $O(\mathrm{\mid }h{\mathrm{\mid }}^{3\mathrm{/}2})$. Thus the full remainder is $o(\mathrm{\mid }h\mathrm{\mid })$, proving Fréchet differentiability and the displayed differential.

At $(1,1)$ the derivative in direction $e_w=(1,0)$ is $2x{\mathbf{1}}_{x>-1}$. At $(1,1+\delta )$ it is $2x{\mathbf{1}}_{x>-(1+\delta {)}^2}$. For $0<\delta \le 1\mathrm{/}4$, their squared $L^2$ distance is $4{\int }_{-(1+\delta {)}^2}^{-1}{x}^{2}p(x)dx\ge 8p(2)\delta ,$ because the interval lies in $[-2,-1]$, where $x^2\ge 1$ and $p(x)\ge p(2)$, and has length $2\delta +{\delta }^2\ge 2\delta$. The operator norm difference of the two differentials is at least their difference on the unit vector $e_w$. Dividing by the parameter distance $\delta$ gives a quotient at least $\sqrt{8p(2)}\mathrm{/}\sqrt{\delta }$, which diverges as $\delta \downarrow 0$. Therefore the differential is not locally Lipschitz at $(1,1)$.