Simultaneous initialization estimates for the Du architectures

We give the scalar estimates used below explicitly. If $U,V$ are centered Gaussian variables with variances at most a fixed $Q$, then $\mathrm{\mid }\sigma (z)\mathrm{\mid }\le B+L\mathrm{\mid }z\mathrm{\mid }$ and [F3] imply $\mathrm{\parallel }\sigma (U){\mathrm{\parallel }}_{2p}\le C\sqrt{p}$; [F5] then gives $\mathrm{\parallel }\sigma (U)\sigma (V){\mathrm{\parallel }}_p\le Cp$ for integers $p\ge 2$. The same last bound holds for $a^2{\sigma }^{\mathrm{\prime }}(U){\sigma }^{\mathrm{\prime }}(V)$ when $a$ is an independent standard normal, because $\mathrm{\mid }{\sigma }^{\mathrm{\prime }}\mathrm{\mid }\le L$. After centering, each such variable $X$ satisfies $E\mathrm{\mid }X{\mathrm{\mid }}^p\le p!C^p$, enlarging $C$ using $p^p\le e^{p}p!$. Expanding its exponential series, for $\mathrm{\mid }t\mathrm{\mid }\le 1\mathrm{/}(2C)$ one has $E{e}^{tX}\le 1+{\sum }_{p\ge 2}(C\mathrm{\mid }t\mathrm{\mid }{)}^p\le e^{2C^2{t}^2}$. For $m$ independent copies, [F20] factors the exponential moment of their sum. Applying [F11] to $\exp (t{\sum }_r{X}_r)$ with $t=s\mathrm{/}(4C^2)$, and then to $\exp (-t{\sum }_r{X}_r)$, to obtain $P(\mathrm{\mid }{m}^{-1}{\sum }_r{X}_r\mathrm{\mid }>s)\le 2e^{-cm{s}^2}$ for $0<s\le c$. All constants are uniform over the covariances with these variance bounds.

The Gaussian expectation maps needed in the population recurrences are Lipschitz functions of the three entries of their bivariate covariance on any region $v\le Q_{ii},Q_{jj}\le Q$, where $v>0$ is fixed. This applies to $E[g(U)g(V)]$ for $g=\sigma$ and $g={\sigma }^{\mathrm{\prime }}$, and to the residual marginal map $s\mapsto E\sigma (\sqrt{s}Z)$. At fixed $\rho$, changing $s,t$ changes these expectations by at most $C(\mathrm{\mid }\mathrm{\Delta }s\mathrm{\mid }+\mathrm{\mid }\mathrm{\Delta }t\mathrm{\mid })$: use the Lipschitz constants $L,\beta$, $s,t\ge v$, and [F5] to bound the resulting products. For fixed $s,t$ and $\mathrm{\mid }\rho \mathrm{\mid }<1$, the covariance matrix is positive definite, so [F7] supplies the density ${\varphi }_{s,t,\rho }$. On any compact interval $\mathrm{\mid }\rho \mathrm{\mid }\le r<1$, its eigenvalues lie between $v(1-r)$ and $Q(1+r)$. Direct differentiation of the Gaussian density gives ${\mathrm{\partial }}_{\rho }{\varphi }_{s,t,\rho }=\sqrt{st}{\mathrm{\partial }}_u{\mathrm{\partial }}_v{\varphi }_{s,t,\rho }$. Uniform eigenvalue bounds imply that $\mathrm{\mid }g(u)g(v){\mathrm{\partial }}_{\rho }{\varphi }_{s,t,\rho }(u,v)\mathrm{\mid }$ is bounded by an integrable polynomial times $e^{-c(u^2+v^2)}$, uniformly on this interval; $g$ has at most linear growth and $g^{\mathrm{\prime }}$ is bounded. Thus [F8] justifies differentiating the expectation under the integral. On finite rectangles apply [F9] successively in $u$ and $v$; the boundary terms tend to zero as the rectangles expand because of Gaussian decay and the growth bound. The same integrable majorant and [F10] pass the rectangle limits, giving ${\mathrm{\partial }}_{\rho }E[g(U)g(V)]=\sqrt{st}E[g^{\mathrm{\prime }}(U)g^{\mathrm{\prime }}(V)]$. Its absolute value is at most $Q{L}^2$ for $g=\sigma$ and $Q{\beta }^2$ for $g={\sigma }^{\mathrm{\prime }}$, since $\mathrm{\mid }{\sigma }^{\mathrm{\prime }}\mathrm{\mid }\le L$ and $\mathrm{\mid }{\sigma }^{\mathrm{\prime }\mathrm{\prime }}\mathrm{\mid }\le \beta$. At $\rho =\pm 1$, the representation $U=\sqrt{s}Z$, $V=\sqrt{t}(\rho Z+\sqrt{1-{\rho }^2}{Z}^{\mathrm{\prime }})$ is dominated after multiplication by either pair of activations by $C(1+\mathrm{\mid }Z\mathrm{\mid }+\mathrm{\mid }{Z}^{\mathrm{\prime }}\mathrm{\mid }{)}^2$, which is integrable; [F10] gives continuity at both endpoints. Finally $\rho =Q_{ij}\mathrm{/}\sqrt{st}$ is Lipschitz in the covariance entries when $s,t\ge v$.

Write ${\widehat{S}}_{ij}^{(h)}=⟨x_i^{(h)}(0),x_j^{(h)}(0)⟩$ and, for residual layers, ${\widehat{b}}_i^{(h)}=m^{-1\mathrm{/}2}{\sum }_r{x}_{i,r}^{(h)}(0)$. Conditional on previous layers, the preactivation pairs in distinct rows are independent centered Gaussians with covariance given by ${\widehat{S}}^{(h-1)}$. In the fully connected case, step 1.1 bounds each difference of ${\widehat{S}}_{ij}^{(h)}$ from its conditional mean by $C{t}_0$, except with probability $\delta \mathrm{/}(32H{n}^2)$, where $t_0=\sqrt{\log (64H{n}^2\mathrm{/}\delta )\mathrm{/}m}$. Its conditional mean is exactly the fully connected population expectation evaluated at ${\widehat{S}}^{(h-1)}$. Since the population diagonal is one, step 2.1 gives, while the error is at most a fixed small constant, $e_h\le C{e}_{h-1}+C{t}_0$, where $e_h={\max }_{ij}\mathrm{\mid }{\widehat{S}}_{ij}^{(h)}-K_{ij}^{(h)}\mathrm{\mid }$ and $e_0=0$. If $B>1$ is a fixed base for this recurrence, then $e_h\le B^H{t}_0$ after enlarging its prefactor. Choose the statement constant $A\ge B^2$ and enlarge the width constant; its term $T{n}^2\log (16Hn\mathrm{/}\delta )\mathrm{/}{\lambda }^2$, with $T=A^H$, then makes this error smaller than both the fixed stopping threshold and $c\lambda \mathrm{/}n$. Conditional bounds and the finite union bound [F21] over $h,i,j$ therefore justify the estimates through layers $h<H$; the last hidden-feature norm is bounded by one further conditional application of the activation moment estimate, without identifying its covariance with the terminal derivative kernel. A first layer crossing the stopping threshold would contradict this recurrence. In particular all feature norms are bounded by a constant, hence by $T$.

In the residual case the population feature variances lie in a fixed compact interval $[v,Q]$ independent of $H$. Indeed the vector construction in [F1] gives $S^{(h)}-b^{(h)}(b^{(h)}{)}^T⪰S^{(1)}-b^{(1)}(b^{(1)}{)}^T$, whose diagonal is $v=c_{\sigma }\mathrm{Var}(\sigma (Z))>0$. Positivity of $v$ follows from nonconstancy, continuity and the positive Gaussian density. Also $\sqrt{S_{ii}^{(h)}}\le \sqrt{S_{ii}^{(h-1)}}+\alpha (B+L\sqrt{S_{ii}^{(h-1)}})$ with $\alpha =c_{\mathrm{r}\mathrm{e}\mathrm{s}}\mathrm{/}H$, so iteration gives a fixed upper bound. The empirical conditional mean satisfies exactly the same residual second-moment/mean recursion with $S,b$ replaced by $\widehat{S},\widehat{b}$. The nonlinear quadratic part has conditional fluctuation at most $C{\alpha }^2{t}_0$ by step 1.1. Each cross part is $\alpha m^{-1\mathrm{/}2}{\sum }_r{x}_{i,r}^{(h-1)}\sigma (U_{j,r})$; its centered value has absolute size at most $C\alpha t_0$ by [F2], since its Lipschitz constant in the independent row Gaussians is at most $C\alpha \mathrm{\parallel }{x}_i^{(h-1)}\mathrm{\parallel }\mathrm{/}\sqrt{m}$. The empirical mean has fluctuation at most $C\alpha t_0$ by [F2]. Step 2.1 now yields $e_h\le (1+C\mathrm{/}H)e_{h-1}+C{t}_0\mathrm{/}H$, taking the maximum of the covariance and mean errors. The first layer has $e_1\le C{t}_0$ by step 1.1 and [F2]. Hence $e_h\le C{t}_0$ uniformly in $H$. The width assumption and the same stopping argument, now in $[v\mathrm{/}2,2Q]$, justify the conditional estimates and give the feature bound through $h<H$; the finite union bound [F21] combines their layerwise failure probabilities. One further conditional step gives $\mathrm{\parallel }{x}_i^{(H)}\mathrm{\parallel }\le C$, using the same upper-norm recurrence, so ${\max }_{i,h}\mathrm{\parallel }{x}_i^{(h)}\mathrm{\parallel }\le C$.

For either architecture, conditional on layers before the last, the last-hidden-layer Jacobian Gram entries are $s_H^2{\widehat{S}}_{ij}^{(H-1)}{m}^{-1}{\sum }_r{a}_r^2{\sigma }^{\mathrm{\prime }}(z_{i,r}){\sigma }^{\mathrm{\prime }}(z_{j,r})$, where $s_H^2=c_{\sigma }$ for the fully connected case and $s_H^2=c_{\mathrm{r}\mathrm{e}\mathrm{s}}^2\mathrm{/}{H}^2$ for residual $H\ge 2$. This follows by differentiating the last layer. The summands are independent across $r$. Steps 1.1–2.1, the feature bounds and errors just proved give entry error at most $B_2^H{t}_0$ in the fully connected case, for a fixed base $B_2$ depending only on the fixed activation bounds, and at most $C{t}_0\mathrm{/}{H}^2$ in the residual case. Choose the statement constant $A$ also with $A\ge B_2^2$; the displayed logarithmic width term and a sufficiently large fixed multiplier then make both errors at most $\lambda \mathrm{/}(4n)$. A finite union bound [F21] covers the terminal-kernel entry events. If an $n\times n$ matrix $E$ has entries bounded by $\epsilon$, then for any $z$, $\mathrm{\parallel }Ez{\mathrm{\parallel }}_2\le \sqrt{n}\epsilon {\sum }_j\mathrm{\mid }{z}_j\mathrm{\mid }\le n\epsilon \mathrm{\parallel }z{\mathrm{\parallel }}_2$ by [F6], hence $\mathrm{\parallel }E{\mathrm{\parallel }}_{\mathrm{o}\mathrm{p}}\le n\epsilon$. Thus $\mathrm{\parallel }{J}_H{J}_H^T-K{\mathrm{\parallel }}_{\mathrm{o}\mathrm{p}}\le \lambda \mathrm{/}4$. Both matrices are symmetric, and for every $z$, $z^T{J}_H{J}_H^{T}z\ge z^{T}Kz-\mathrm{\mid }{z}^{T}Ez\mathrm{\mid }\ge (\lambda -\mathrm{\parallel }E{\mathrm{\parallel }}_{\mathrm{o}\mathrm{p}})\mathrm{\parallel }z{\mathrm{\parallel }}_2^2\ge 3\lambda \mathrm{\parallel }z{\mathrm{\parallel }}_2^2\mathrm{/}4$, where $E=J_H{J}_H^T-K$. The Rayleigh-quotient characterization of the smallest eigenvalue now gives ${\lambda }_{\min }(J_H{J}_H^T)\ge 3\lambda \mathrm{/}4$.

By [F16], choose a $1\mathrm{/}4$-net $\mathcal{N}$ of the unit sphere with at most $9^m$ points. The Euclidean norm duality $\mathrm{\parallel }z{\mathrm{\parallel }}_2={\sup }_{\mathrm{\parallel }x{\mathrm{\parallel }}_2=1}\mathrm{\mid }{x}^{T}z\mathrm{\mid }$ follows from [F6], so $\mathrm{\parallel }W{\mathrm{\parallel }}_{\mathrm{o}\mathrm{p}}={\sup }_{\mathrm{\parallel }x\mathrm{\parallel }=\mathrm{\parallel }y\mathrm{\parallel }=1}\mathrm{\mid }{x}^{T}Wy\mathrm{\mid }$. For unit $x,y$, choose $v,w\in \mathcal{N}$ within $1\mathrm{/}4$ of them. Then [F6] gives $\mathrm{\mid }{x}^{T}Wy-v^{T}Ww\mathrm{\mid }\le \mathrm{\parallel }x-v{\mathrm{\parallel }}_2\mathrm{\parallel }Wy{\mathrm{\parallel }}_2+\mathrm{\parallel }v{\mathrm{\parallel }}_2\mathrm{\parallel }W(y-w){\mathrm{\parallel }}_2\le \frac{1}{2}\mathrm{\parallel }W{\mathrm{\parallel }}_{\mathrm{o}\mathrm{p}}$. Taking suprema yields $\mathrm{\parallel }W{\mathrm{\parallel }}_{\mathrm{o}\mathrm{p}}\le 2{\max }_{v,w\in \mathcal{N}}\mathrm{\mid }{v}^{T}Ww\mathrm{\mid }$. Each $v^{T}Ww$ is standard normal, whose tail is at most $2e^{-t^2\mathrm{/}2}$ by [F2]. The finite union bound [F21] gives $\mathrm{\parallel }{W}^{(h)}\mathrm{\parallel }\le C\sqrt{m}$ for all $h\ge 2$ with failure at most $\delta \mathrm{/}16$ when $m\ge C\log (16H\mathrm{/}\delta )$, which follows from either displayed width because $\lambda \le c_{\sigma }{L}^2$ in the fully connected case and $\lambda \le C\mathrm{/}{H}^2$ in the residual case. The latter bound follows by bounding the diagonal of its terminal kernel with step 4.1. Gaussian second and eighth moments and [F12] give $\mathrm{\parallel }a{\mathrm{\parallel }}_2\le C\sqrt{m}$ and $\mathrm{\parallel }a{\mathrm{\parallel }}_4\le C{m}^{1\mathrm{/}4}$ with failure at most $\delta \mathrm{/}16$ when $m\ge C\mathrm{/}\delta$, also implied by the widths. These arguments never bound $\mathrm{\parallel }{W}^{(1)}\mathrm{\parallel }$ on the whole input space, so no input-dimension condition is used.

Conditional on hidden weights, $b_{i,h}(0)=P_{i,h}^{T}a$, where $P_{i,h}$ is the product of downstream feature Jacobians. Their operator norms have fixed envelopes $B^H$ in the fully connected case and $B$ in the residual case, using ${\prod }_{l>h}(1+C\mathrm{/}H)\le e^C$ for residual layers. Increase the conclusion constants so these envelopes are at most $T\mathrm{/}2$ and $C\mathrm{/}2$, respectively. Each coordinate is centered Gaussian of variance at most $(T\mathrm{/}2{)}^2$ or $(C\mathrm{/}2{)}^2$. Since $M^2=1+\log (16mnH\mathrm{/}\delta )$, its two-sided tail is at most $2e^{-2M^2}\le \delta \mathrm{/}(8mnH)$; the finite union bound [F21] over at most $mnH$ coordinates gives the asserted coordinate maxima with failure at most $\delta \mathrm{/}8$. Finally [F4] and $\mathrm{\parallel }\mathrm{\Phi }{\mathrm{\parallel }}_{\mathrm{o}\mathrm{p}}\le \mathrm{\parallel }\mathrm{\Phi }{\mathrm{\parallel }}_F\le \sqrt{n}{\max }_i\mathrm{\parallel }{x}_i^{(H)}\mathrm{\parallel }$ give $\mathrm{\parallel }u(0)\mathrm{\parallel }\le C\sqrt{Tnq}$ or $C\sqrt{nq}$ with failure at most $\delta \mathrm{/}16$.

By the finite union bound [F21], allocate at most $\delta \mathrm{/}(32H{n}^2)$ to each fully connected feature-covariance or terminal-feature-norm event in step 3.1; there are at most $2H{n}^2$ such events. Allocate at most $\delta \mathrm{/}(128H{n}^2)$ to each residual covariance, cross-term, mean, or terminal-feature-norm event in step 4.1; there are at most $4H{n}^2$ such events. Allocate at most $\delta \mathrm{/}(64n^2)$ to each terminal-kernel entry event. The weight-matrix and readout norm events in step 5.2 each use $\delta \mathrm{/}16$; backward coordinates use $\delta \mathrm{/}8$ and the output norm uses $\delta \mathrm{/}16$ in step 6.1. Even summing both architecture allocations, these budgets total at most $27\delta \mathrm{/}64<\delta$. The thresholds are constant multiples of $t_0$ or $M$; increasing the fixed constants in the width and conclusions makes each displayed concentration bound meet its allocation without changing their forms. Adding the deterministic bounded-label contribution proves the simultaneous event.