Neural tangent kernels and lazy training — Examples

Differentiating $f=a{w}^2$ gives ${\mathrm{\partial }}_{a}f=w^2$ and ${\mathrm{\partial }}_{w}f=2aw$. By [F1], $K(a,w)=(w^2{)}^2+(2aw{)}^2=w^4+4a^2{w}^2\mathrm{.}$ At $(1,1)$, $f=1$, ${\mathrm{\partial }}_{a}C=f{w}^2=1$, and ${\mathrm{\partial }}_{w}C=2faw=2$, so the negative-gradient vector field has the stated initial velocity.

The vector field $(\dot{a},\dot{w})=(-a{w}^4,-2a^2{w}^3)$ is polynomial, hence locally Lipschitz; [F2] gives a differentiable local solution through $(1,1)$. Along it, $\frac{d}{dt}K=4w^3\dot{w}+8a{w}^2\dot{a}+8a^{2}w\dot{w},$ which equals $-8-8-16=-32$ at $t=0$. Also $\frac{d}{dt}(w^2){\mathrm{\mid }}_0=2w\dot{w}=-4$ and $\frac{d}{dt}(2aw){\mathrm{\mid }}_0=2(\dot{a}w+a\dot{w})=-6$, so both parameter features vary. A differentiable scalar function with nonzero derivative at zero is not constant on any right neighborhood of zero.

Put $a=e^{2u}>0$. Then $\mathrm{\mid }a-1\mathrm{\mid }<a+1$, so $\mathrm{\mid }\sigma (u)\mathrm{\mid }<1$. The derivative rules in [F1] give ${\sigma }^{\mathrm{\prime }}(u)=4e^{2u}\mathrm{/}(e^{2u}+1{)}^2=1-\sigma (u{)}^2$ and ${\sigma }^{\mathrm{\prime }\mathrm{\prime }}(u)=-2\sigma (u){\sigma }^{\mathrm{\prime }}(u)$. Thus $\mathrm{\mid }{\sigma }^{\mathrm{\prime }}\mathrm{\mid }\le 1$ and $\mathrm{\mid }{\sigma }^{\mathrm{\prime }\mathrm{\prime }}\mathrm{\mid }\le 2$. By [F2], $\mathrm{\mid }\sigma (u)-\sigma (v)\mathrm{\mid }\le \mathrm{\mid }u-v\mathrm{\mid }$, so $\sigma$ is globally Lipschitz. Since ${\sigma }^{\mathrm{\prime }}(u)>0$, it is nonconstant; a polynomial of degree $m\ge 1$ with leading coefficient $a_m\mathrm{\ne }0$ satisfies $p(u)\mathrm{/}{u}^m\to a_m$ as $u\to +\mathrm{\infty }$, so its absolute value is unbounded. The bounded nonconstant $\sigma$ is therefore nonpolynomial.

For each finite network, the parameter vector field for the constant direction is smooth and locally Lipschitz, so a local solution exists by [F3]. The output-layer weights and bias have bounded speed because all hidden activations are bounded by $1$, so they remain bounded on every compact time interval. Suppose the parameters in layers $k+1,\dots ,L-1$ are bounded on such an interval. Backpropagating through those layers multiplies the direction by their finite weight matrices and factors ${\sigma }^{\mathrm{\prime }}$ of modulus at most $1$, so each coordinate of the backpropagated direction at layer $k+1$ is bounded. The gradient formulas in [F4], together with $\mathrm{\mid }\sigma \mathrm{\mid }\le 1$, then bound every derivative in layer $k$; integration bounds its weights and biases. Descending induction bounds all parameters on every compact interval. If the maximal local solution had a finite endpoint, the smooth vector field would be bounded on a compact ball containing the trajectory, so the trajectory would be Lipschitz in time and have a limit as the endpoint is approached. Applying [F3] at that limit extends the solution, a contradiction. Hence the path exists on $[0,1]$. On any compact tube around a finite solution segment the vector field is Lipschitz; subtracting the integral equations for two solutions gives $u(t)\le u(0)+M{\int }_0^{t}u(s)ds$ on the tube for a Lipschitz constant $M$; iterating this inequality gives $u(t)\le e^{Mt}u(0)$, proving local continuous dependence on the initial parameters. A finite cover of $[0,1]$ patches these local flow maps. Thus the global path is measurable as a function of initialization and time. For this direction, $\mathrm{\parallel }d(t){\mathrm{\parallel }}_N=1$ at every time, so its integral is $1$.

Step 1.1 supplies the globally Lipschitz $C^2$ activation and bounded second derivative, and step 2.1 supplies a finite-width solution and the required direction bound. Applying [F4] with $T=1$ gives ${\sup }_{t\in [0,1]}\mathrm{\parallel }{K}_t-K_{\mathrm{\infty }}{\mathrm{\parallel }}_{op}\to 0$ in probability under the stated sequential hidden-width limit.

The initial residual is $f_0-y=av$. Since $v$ is a unit eigenvector with eigenvalue $\lambda >0$, [F2] gives $f(t)-y=a{e}^{-t\lambda \mathrm{/}N}v$.

Taking norms and using $\mathrm{\parallel }v{\mathrm{\parallel }}_2=1$ yields $\mathrm{\parallel }f(t)-y{\mathrm{\parallel }}_2=\mathrm{\mid }a\mathrm{\mid }{e}^{-t\lambda \mathrm{/}N}$, which tends to zero because $\lambda \mathrm{/}N>0$. This is the single positive mode asserted in the Example.

Repeating the input makes every matrix entry equal to $\kappa$, so $K=\kappa \left(\begin{array}{cc}1& 1\\ 1& 1\end{array}\right)$. Multiplication gives $K{v}_{-}=0$ for $v_{-}=(1,-1{)}^T\mathrm{/}\sqrt{2}$. Thus $K$ has rank at most one and the target $y=(1,-1{)}^T=\sqrt{2}{v}_{-}$ lies entirely in its nullspace.

Set $r_{-}(t)=v_{-}^T(f(t)-y)$. The flow equation, symmetry of $K$, and $K{v}_{-}=0$ give ${\dot{r}}_{-}(t)=-\frac{1}{2}{v}_{-}^{T}K(f(t)-y)=-\frac{1}{2}(K{v}_{-}{)}^T(f(t)-y)=0$. Since $r_{-}(0)=-\sqrt{2}$, this residual component stays nonzero for every $t\ge 0$, so $f(t)\mathrm{\ne }y$ at every finite time and the flow cannot converge to $y$. This is consistent with the sphere proposition because its strictness conclusion requires pairwise distinct inputs.