A wide tanh network stays lazy on a fixed time horizon

Put $a=e^{2u}>0$. Then $\mathrm{\mid }a-1\mathrm{\mid }<a+1$, so $\mathrm{\mid }\sigma (u)\mathrm{\mid }<1$. The derivative rules in [F1] give ${\sigma }^{\mathrm{\prime }}(u)=4e^{2u}\mathrm{/}(e^{2u}+1{)}^2=1-\sigma (u{)}^2$ and ${\sigma }^{\mathrm{\prime }\mathrm{\prime }}(u)=-2\sigma (u){\sigma }^{\mathrm{\prime }}(u)$. Thus $\mathrm{\mid }{\sigma }^{\mathrm{\prime }}\mathrm{\mid }\le 1$ and $\mathrm{\mid }{\sigma }^{\mathrm{\prime }\mathrm{\prime }}\mathrm{\mid }\le 2$. By [F2], $\mathrm{\mid }\sigma (u)-\sigma (v)\mathrm{\mid }\le \mathrm{\mid }u-v\mathrm{\mid }$, so $\sigma$ is globally Lipschitz. Since ${\sigma }^{\mathrm{\prime }}(u)>0$, it is nonconstant; a polynomial of degree $m\ge 1$ with leading coefficient $a_m\mathrm{\ne }0$ satisfies $p(u)\mathrm{/}{u}^m\to a_m$ as $u\to +\mathrm{\infty }$, so its absolute value is unbounded. The bounded nonconstant $\sigma$ is therefore nonpolynomial.

For each finite network, the parameter vector field for the constant direction is smooth and locally Lipschitz, so a local solution exists by [F3]. The output-layer weights and bias have bounded speed because all hidden activations are bounded by $1$, so they remain bounded on every compact time interval. Suppose the parameters in layers $k+1,\dots ,L-1$ are bounded on such an interval. Backpropagating through those layers multiplies the direction by their finite weight matrices and factors ${\sigma }^{\mathrm{\prime }}$ of modulus at most $1$, so each coordinate of the backpropagated direction at layer $k+1$ is bounded. The gradient formulas in [F4], together with $\mathrm{\mid }\sigma \mathrm{\mid }\le 1$, then bound every derivative in layer $k$; integration bounds its weights and biases. Descending induction bounds all parameters on every compact interval. If the maximal local solution had a finite endpoint, the smooth vector field would be bounded on a compact ball containing the trajectory, so the trajectory would be Lipschitz in time and have a limit as the endpoint is approached. Applying [F3] at that limit extends the solution, a contradiction. Hence the path exists on $[0,1]$. On any compact tube around a finite solution segment the vector field is Lipschitz; subtracting the integral equations for two solutions gives $u(t)\le u(0)+M{\int }_0^{t}u(s)ds$ on the tube for a Lipschitz constant $M$; iterating this inequality gives $u(t)\le e^{Mt}u(0)$, proving local continuous dependence on the initial parameters. A finite cover of $[0,1]$ patches these local flow maps. Thus the global path is measurable as a function of initialization and time. For this direction, $\mathrm{\parallel }d(t){\mathrm{\parallel }}_N=1$ at every time, so its integral is $1$.

Step 1.1 supplies the globally Lipschitz $C^2$ activation and bounded second derivative, and step 2.1 supplies a finite-width solution and the required direction bound. Applying [F4] with $T=1$ gives ${\sup }_{t\in [0,1]}\mathrm{\parallel }{K}_t-K_{\mathrm{\infty }}{\mathrm{\parallel }}_{op}\to 0$ in probability under the stated sequential hidden-width limit.