Local Gram positivity and loss smoothness give confined gradient descent

Write $g=\mathrm{\nabla }L=J^{T}r$. Since $J{J}^T⪰J_{\ast }{J}_{\ast }^T$, $\mathrm{\parallel }g{\mathrm{\parallel }}^2=r^{T}J{J}^{T}r\ge (\lambda \mathrm{/}2)\mathrm{\parallel }r{\mathrm{\parallel }}^2=\lambda L$ in the ball. For any step ${\theta }^{+}=\theta -hg$, $0<h\le \eta$, whose segment stays in the ball, put $\psi (s)=L(\theta -sg)$ for $0\le s\le h$. Then ${\psi }^{\mathrm{\prime }}(0)=-\mathrm{\parallel }g{\mathrm{\parallel }}^2$ and ${\psi }^{\mathrm{\prime }\mathrm{\prime }}(s)=g^T{H}_{s}g$, where $H_s={\mathrm{\nabla }}^{2}L(\theta -sg)$. By Euclidean Cauchy–Schwarz [F4] and the induced norm [F3], $\mathrm{\mid }{g}^T{H}_{s}g\mathrm{\mid }\le \mathrm{\parallel }g\mathrm{\parallel }\mathrm{\parallel }{H}_{s}g\mathrm{\parallel }\le \mathrm{\parallel }g\mathrm{\parallel }\mathrm{\parallel }{H}_s\mathrm{\parallel }\mathrm{\parallel }g\mathrm{\parallel }\le S\mathrm{\parallel }g{\mathrm{\parallel }}^2$; for $g=0$ the same bound is immediate. By [F2], $L^{+}=L-h\mathrm{\parallel }g{\mathrm{\parallel }}^2+{\int }_0^h(h-s){\psi }^{\mathrm{\prime }\mathrm{\prime }}(s)ds\le L-h(1-Sh\mathrm{/}2)\mathrm{\parallel }g{\mathrm{\parallel }}^2\le (1-h\lambda \mathrm{/}2)L$. If $L>0$, nonnegativity of $L^{+}$ gives $h\lambda \mathrm{/}2\le 1$.

For such a segment with $L>0$, $\sqrt{L}-\sqrt{L^{+}}=(L-L^{+})\mathrm{/}(\sqrt{L}+\sqrt{L^{+}})\ge h\mathrm{\parallel }g{\mathrm{\parallel }}^2\mathrm{/}(4\sqrt{L})\ge (\sqrt{\lambda }\mathrm{/}4)h\mathrm{\parallel }g\mathrm{\parallel }$. If $L=0$, $r=0$ and hence $g=0$, so the step is stationary. Telescoping any finite sequence of admissible full or truncated steps gives total parameter path length at most $4\sqrt{L_0}\mathrm{/}\sqrt{\lambda }=2\sqrt{2}\mathrm{\parallel }{r}_0\mathrm{\parallel }\mathrm{/}\sqrt{\lambda }$.

Suppose there is a first proposed full step exiting the ball. Truncate that step at the first intersection with its boundary; this is a segment of length $h\mathrm{\parallel }g\mathrm{\parallel }$ with $h\le \eta$, wholly in the ball. Step 2.1 applies to all previous steps and this truncated step, giving total path length strictly less than $R$. Reaching the boundary from ${\theta }_0$ requires length at least $R$, a contradiction. Thus no exit occurs. Step 1.1 applied at every full step gives the loss rate, and taking the supremum of finite partial path lengths in step 2.1 gives the asserted series bound.