Parameter-ball control of the last-layer Gram and loss Hessian

Write $z_h=W_h{x}_{h-1}$ and $s_1=\sqrt{c_{\sigma }}$ in both architectures, $s_h=\sqrt{c_{\sigma }}$ for fully connected $h\ge 2$, and $s_h=c_{\mathrm{r}\mathrm{e}\mathrm{s}}\mathrm{/}H$ for residual $h\ge 2$. A layer has feature Jacobian $P_h=(s_h\mathrm{/}\sqrt{m})\mathrm{diag}({\sigma }^{\mathrm{\prime }}(z_h))W_h$ in the fully connected case, and $P_h=I+(s_h\mathrm{/}\sqrt{m})\mathrm{diag}({\sigma }^{\mathrm{\prime }}(z_h))W_h$ in residual layers $h\ge 2$. Within the stated ball, $\mathrm{\parallel }{W}_h\mathrm{\parallel }\le (B_w+D\mathrm{/}\sqrt{m})\sqrt{m}$. Products of consecutive $P_h$, $h\ge 2$, therefore have norm at most $A^H$ in the fully connected case and ${\prod }_h(1+C\mathrm{/}H)\le e^C$ in the residual case. All estimates below use only $h\ge 2$ for matrix operator norms; the input is fixed.

Let $e_h=\mathrm{\parallel }{x}_h(\theta )-x_h({\theta }_0)\mathrm{\parallel }$. Lipschitz continuity gives $e_1\le \sqrt{c_{\sigma }}LD\mathrm{/}\sqrt{m}$. For fully connected layers, $e_h\le C{e}_{h-1}+C{A}_0^{H}D\mathrm{/}\sqrt{m}$; for residual layers it gives $e_h\le (1+C\mathrm{/}H)e_{h-1}+CD\mathrm{/}(H\sqrt{m})$, using the fixed upper feature bound in the residual case and the initial exponential envelope in the fully connected case. Summing the geometric recurrences gives $e_h\le TD\mathrm{/}\sqrt{m}$ in the first case and $e_h\le CD\mathrm{/}\sqrt{m}$ in the second. The imposed smallness of $D\mathrm{/}\sqrt{m}$ keeps all feature norms in the required bounds and justifies the recurrences successively. Also $\mathrm{\parallel }{z}_h(\theta )-z_h({\theta }_0)\mathrm{\parallel }\le TD$ or $CD$, since it is bounded by $\mathrm{\parallel }{W}_h(\theta )\mathrm{\parallel }{e}_{h-1}+D\mathrm{\parallel }{x}_{h-1}({\theta }_0)\mathrm{\parallel }$ for $h\ge 2$, and by $D$ for $h=1$.

For a unit parameter direction $v$, let ${\xi }_h=D{x}_h[v]$. At the first layer $\mathrm{\parallel }{\xi }_1\mathrm{\parallel }\le C\mathrm{\parallel }{v}_{W_1}\mathrm{\parallel }\mathrm{/}\sqrt{m}$. Subsequent layers obey ${\xi }_h=P_h{\xi }_{h-1}+(s_h\mathrm{/}\sqrt{m})\mathrm{diag}({\sigma }^{\mathrm{\prime }}(z_h))v_{W_h}{x}_{h-1}$. The product bounds in step 1.1, the feature bounds in step 2.1, and Cauchy–Schwarz over layer blocks give ${\max }_h\mathrm{\parallel }{\xi }_h\mathrm{\parallel }\le T\mathrm{/}\sqrt{m}$ or $C\mathrm{/}\sqrt{m}$. For the residual case use ${\sum }_{h\ge 2}{s}_h\mathrm{\parallel }{v}_{W_h}\mathrm{\parallel }\le c_{\mathrm{r}\mathrm{e}\mathrm{s}}({\sum }_{h\ge 2}\mathrm{\parallel }{v}_{W_h}{\mathrm{\parallel }}^2{)}^{1\mathrm{/}2}\mathrm{/}\sqrt{H}\le C$, retaining the separate first-layer term. Thus $\mathrm{\parallel }D{z}_h[v]\mathrm{\parallel }\le T$ or $C$, uniformly in $h$. Since $D{f}_i[v]=v_a^T{x}_H+a^T{\xi }_H$ and $\mathrm{\parallel }a\mathrm{\parallel }\le C\sqrt{m}$, each output gradient has norm at most $T$ or $C$, proving $\mathrm{\parallel }J\mathrm{\parallel }\le T\sqrt{n}$ or $C\sqrt{n}$. This explicitly retains the output and first-layer terms with no $1\mathrm{/}H$ bound.

Put $d_h=b_h(\theta )-b_h({\theta }_0)$. The backward recurrence is $b_{h-1}=P_h^T{b}_h$, with $d_H=a-a_0$. Decompose its difference as $P_h(\theta {)}^T{d}_h+(P_h(\theta )-P_h({\theta }_0){)}^T{b}_h({\theta }_0)$. The latter is the sum of a weight-difference term bounded by $T{s}_{h}D$ (respectively $C{s}_{h}D$) and a gate-difference term bounded by $C{s}_h{B}_{b}M\mathrm{\parallel }{z}_h(\theta )-z_h({\theta }_0)\mathrm{\parallel }$, because $\mathrm{\parallel }\mathrm{diag}({\sigma }^{\mathrm{\prime }}(z_h)-{\sigma }^{\mathrm{\prime }}(z_{h,0}))b_{h,0}\mathrm{\parallel }\le \beta B_{b}M\mathrm{\parallel }{z}_h-z_{h,0}\mathrm{\parallel }$. Summing the backward recurrence with the product bounds gives $\mathrm{\parallel }{d}_h\mathrm{\parallel }\le TMD$ or $CMD$. Hence $\mathrm{\parallel }{b}_h(\theta ){\mathrm{\parallel }}_{\mathrm{\infty }}\le TM(1+D)$ or $CM(1+D)$. The crude ${\mathrm{\ell }}_2$ bound $\mathrm{\parallel }{b}_h(\theta ){\mathrm{\parallel }}_2\le T\sqrt{m}$ or $C\sqrt{m}$ follows separately from step 1.1 and $\mathrm{\parallel }a{\mathrm{\parallel }}_2\le C\sqrt{m}$.

For the last-hidden-layer Gram, put $X_i=x_i^{(H-1)}$, $Z_i=W_H{X}_i$ and $\gamma =c_{\sigma }$ or $c_{\mathrm{r}\mathrm{e}\mathrm{s}}^2\mathrm{/}{H}^2$. Direct differentiation gives $G_{H,ij}=\gamma ⟨X_i,X_j⟩m^{-1}{\sum }_r{a}_r^2{\sigma }^{\mathrm{\prime }}(Z_{i,r}){\sigma }^{\mathrm{\prime }}(Z_{j,r})$. Subtract its initialized version in three parts: feature-inner-product change, derivative-gate change with $a_0^2$, and output-weight-square change. Step 2.1 bounds the first part by $TD\mathrm{/}\sqrt{m}$ or $CD\mathrm{/}\sqrt{m}$ before multiplying by $\gamma$. For the second part, Cauchy–Schwarz gives $m^{-1}{\sum }_r{a}_{0,r}^2\mathrm{\mid }\mathrm{\Delta }{Z}_{i,r}\mathrm{\mid }\le m^{-1}\mathrm{\parallel }{a}_0{\mathrm{\parallel }}_4^2\mathrm{\parallel }\mathrm{\Delta }{Z}_i{\mathrm{\parallel }}_2\le C\mathrm{\parallel }\mathrm{\Delta }{Z}_i{\mathrm{\parallel }}_2\mathrm{/}\sqrt{m}$; multiply by $\beta L$, and use step 2.1. For the third part, $m^{-1}{\sum }_r\mathrm{\mid }{a}_r^2-a_{0,r}^2\mathrm{\mid }\le m^{-1}\mathrm{\parallel }a-a_0{\mathrm{\parallel }}_2(\mathrm{\parallel }a{\mathrm{\parallel }}_2+\mathrm{\parallel }{a}_0{\mathrm{\parallel }}_2)\le CD\mathrm{/}\sqrt{m}$. Feature norms and $\mathrm{\mid }{\sigma }^{\mathrm{\prime }}\mathrm{\mid }\le L$ bound all remaining factors. The entry error is consequently at most $TD\mathrm{/}\sqrt{m}$ or $CD\mathrm{/}(H^2\sqrt{m})$. The operator norm is at most $n$ times this entry bound.

For two unit parameter directions $v,w$, twice differentiating the output gives the mixed output-weight terms $v_a^{T}D{x}_H[w]+w_a^{T}D{x}_H[v]$, and the sum over layers of $\frac{s_h}{\sqrt{m}}{b}_h^T(\mathrm{diag}({\sigma }^{\mathrm{\prime }\mathrm{\prime }}(z_h))(D{z}_h[v]\odot D{z}_h[w])+\mathrm{diag}({\sigma }^{\mathrm{\prime }}(z_h))(v_{W_h}D{x}_{h-1}[w]+w_{W_h}D{x}_{h-1}[v]))\mathrm{.}$ This identity follows by differentiating every primitive twice and propagating each resulting second-order term through its downstream Jacobians; the linear skip has zero second derivative. The coordinate-product inequality $\mathrm{\parallel }p\odot q{\mathrm{\parallel }}_1\le \mathrm{\parallel }p{\mathrm{\parallel }}_2\mathrm{\parallel }q{\mathrm{\parallel }}_2$, steps 3.1–3.2, and ${\sum }_h{s}_h\le CH$ or $C$ bound the first part by $TM(1+D)\mathrm{/}\sqrt{m}$ or $CM(1+D)\mathrm{/}\sqrt{m}$. The terms involving $v_{W_h}D{x}_{h-1}[w]$ use the ${\mathrm{\ell }}_2$ backward bound and $\mathrm{\parallel }D{x}_{h-1}[w]\mathrm{\parallel }\le T\mathrm{/}\sqrt{m}$ or $C\mathrm{/}\sqrt{m}$; summing with Cauchy–Schwarz over matrix blocks gives the same bound. The output-weight mixed terms are at most $T\mathrm{/}\sqrt{m}$ or $C\mathrm{/}\sqrt{m}$. This proves the output-Hessian bound, with polynomial factors in $H$ absorbed into $A^H$ only for the fully connected case.

The identity ${\mathrm{\nabla }}^{2}L=J^{T}J+{\sum }_i{r}_i{\mathrm{\nabla }}^2{f}_i$ gives $\mathrm{\parallel }{\mathrm{\nabla }}^{2}L\mathrm{\parallel }\le \mathrm{\parallel }J{\mathrm{\parallel }}^2+\sqrt{n}\mathrm{\parallel }r\mathrm{\parallel }{\max }_i\mathrm{\parallel }{\mathrm{\nabla }}^2{f}_i\mathrm{\parallel }$. Integrating the Jacobian along the straight segment to initialization gives $\mathrm{\parallel }r\mathrm{\parallel }\le \mathrm{\parallel }{r}_0\mathrm{\parallel }+T\sqrt{n}D$ or $\mathrm{\parallel }{r}_0\mathrm{\parallel }+C\sqrt{n}D$. Substitution of steps 3.1 and 4.1 proves the claimed loss-Hessian bound, after enlarging $A,C$ to absorb squares of fixed exponential envelopes. For each coordinate the integral Taylor formula is $f_i(\theta +h)-f_i(\theta )-D{f}_i(\theta )h={\int }_0^1(1-t)D^2{f}_i(\theta +th)[h,h]dt$. Step 4.1 bounds its absolute value by $TM(1+D)\mathrm{\parallel }h{\mathrm{\parallel }}^2\mathrm{/}(2\sqrt{m})$ or the corresponding residual constant. Summing coordinate squares gives the stated vector Taylor bound.