TheoremStatement: Literature-sourcedProof: AI-adaptedPipeline-generatedjudge pass (gpt-6-luna)

How statement and proof provenance work

The first chip identifies the source of the statement or construction; the second identifies the source of its local proof or verification.

Literature-sourced: the exact statement appears in a cited source; only wording and notation differ.
AI-adapted: a semantically identical restatement of literature-sourced material, modulo indexing, notation, and boundary cases adopted by the library.
AI-generated: a genuinely novel statement formulated by AI, with no source for the claim itself.

These labels describe origin, not correctness: citations and verification chips remain separate evidence.

Sequential infinite-width NTK limit and finite-time lazy training dynamics

Statement

Assume AC. Fix a finite list $x_{1}, \dots, x_{N} \in R^{n_{0}}$ with $N, n_{0} \geq 1$ , a depth $L \geq 2$ , a bias scale $β > 0$ , and a globally Lipschitz activation $σ : R \to R$ . Let $s$ be any bounded Borel representative of the almost-everywhere derivative of $σ$ , meaning that it agrees with the ordinary derivative outside a Lebesgue-null set; such a representative exists and is bounded by the Lipschitz constant. For each tuple of positive integer hidden widths $n = (n_{1}, \dots, n_{L - 1})$ , let $P_{n}$ be a probability law carrying the product standard-normal initialization marginal and any auxiliary randomness used by training; take all probabilities in the width limit under these laws. Use the width-normalized network with scalar linear output.

Put

$Σ_{i j}^{1} = \frac{x_{i} \cdot x_{j}}{n_{0}} + β^{2} .$

For $r = 2, \dots, L$ , let $(U_{i}, U_{j})$ have the centered Gaussian law with variances $Σ_{i i}^{r - 1}, Σ_{j j}^{r - 1}$ and covariance $Σ_{i j}^{r - 1}$ , and define

$Σ_{i j}^{r} = E [σ (U_{i}) σ (U_{j})] + β^{2}, {\dot{Σ}}_{i j}^{r} = E [s (U_{i}) s (U_{j})] .$

These Gaussian laws may be singular; their marginal variances are at least $β^{2}$ , and their laws and expectations depend only on the displayed covariance. Define $Θ_{i j}^{1} = Σ_{i j}^{1}$ and, for $r = 2, \dots, L$ ,

$Θ_{i j}^{r} = Θ_{i j}^{r - 1} {\dot{Σ}}_{i j}^{r} + Σ_{i j}^{r}, K_{\infty} = (Θ_{i j}^{L})_{i, j = 1}^{N} .$

At initialization, the finite-data neural tangent Gram matrix $K_{0}$ converges in probability to $K_{\infty}$ under the sequential hidden-width limit $n_{1} \to \infty$ first, then $n_{2} \to \infty$ , and so on through $n_{L - 1}$ .

For the training conclusion, fix a horizon $T \geq 0$ . In addition, assume that $σ$ is twice continuously differentiable with bounded second derivative. For each width tuple, let $d_{n}$ and $θ_{n}$ be jointly measurable in the probability outcome and time; for $P_{n}$ -almost every outcome, assume $d_{n}$ is integrable on $[0, T]$ and $θ_{n}$ is differentiable there and satisfies, with width and probability-outcome subscripts suppressed in the equation,

${\dot{θ}}_{p} (t) = - \frac{1}{N} \sum_{i = 1}^{N} \partial_{θ_{p}} f_{θ (t)} (x_{i}) d_{i} (t) .$

Write $f_{t} (x) = f_{θ (t)} (x)$ and $K_{t} (i, j) = K_{θ (t)} (x_{i}, x_{j})$ , where $K_{θ}$ is the finite-network neural tangent kernel; width and probability-outcome subscripts are suppressed. The sample paths of $K_{t}$ are continuous, so joint measurability makes the supremum events below measurable. Define $∥ v ∥_{N} = (N^{- 1} \sum_{i} ∣ v_{i} ∣^{2})^{1 / 2}$ . Suppose $\int_{0}^{T} ∥ d_{n} (t) ∥_{N} d t$ is stochastically bounded in the prescribed sequential limit, meaning that for every $ε > 0$ some $M < \infty$ satisfies ${lim sup}_{n_{L - 1} \to \infty} \dots {lim sup}_{n_{1} \to \infty} P_{n} (\int_{0}^{T} ∥ d_{n} (t) ∥_{N} d t > M) < ε$ . Then

$\sup_{0 \leq t \leq T} ∥ K_{t} - K_{\infty} ∥_{o p} ⟶ 0 in probability,$

where $∥ \cdot ∥_{o p}$ is the Euclidean operator norm. Along the path,

${\dot{f}}_{t} (x_{i}) = - \frac{1}{N} \sum_{j = 1}^{N} K_{t} (i, j) d_{j} (t),$

and

$\sup_{0 \leq t \leq T} {∥ f_{t} - f_{0} + \frac{1}{N} \int_{0}^{t} K_{\infty} d (s) d s ∥}_{N} ⟶ 0 in probability .$

Thus the finite-time output equation has the deterministic limiting kernel as its generator in the precise integrated-error sense displayed above.

Statement references

Jacot, Gabriel, and Hongler, Theorems 1–2, sequential infinite-width NTK and training limits

Jacot–Gabriel–Hongler width-normalized fully connected networks: Hidden layers use $1 / \sqrt{n_{ℓ}}$ scaling and independent Gaussian initialization

Gaussian feature expectations are continuous, including singular covariance limits

The finite-data NTK stays close to its initialization on fixed time intervals

Facts & Assumptions

Given: AC; finite data; $L \geq 2$ ; $β > 0$ ; a globally Lipschitz activation; the source-normalized network under the width-specific laws; and, for training, the additional $C^{2}$ bound, jointly measurable paths and directions with the stated pathwise properties, and an integrated-direction bound.

[A1]

AC supplies the finite Gaussian probability laws and the countable/dependent choice needed by the sequential construction (The Axiom of Choice, AC supplies countable selections and prescribed serial paths).

[F1]

The network definition fixes the layer dimensions, fan-in scaling, bias scale, scalar linear output, and independent standard-normal initialization (Jacot–Gabriel–Hongler width-normalized fully connected networks).

[F2]

A finite Gaussian law, including a singular one, is uniquely determined by its mean and covariance; the layer's independent Gaussian weights and biases have product laws (Multivariate normal law, including singular covariance, Characteristic function of a multivariate normal law, Standard normal and normal laws, Expectations factor over finite products of independent random variables).

[F3]

Empirical averages of iid square-integrable variables converge in probability to their means, and standard normals have finite fourth moment (IID finite-variance weak law, A normal variable has finite fourth moment).

[F4]

Bivariate Gaussian expectations of Lipschitz features and bounded Borel features vary continuously with covariance, including singular limits (Continuity of bivariate Gaussian feature expectations, including singular limits).

[F5]

The pointwise total-derivative chain rule applies to the finite network at differentiability points (The chain rule for total derivatives: $D (g \circ f) (a) = D g (f (a)) \circ D f (a)$ ).

[F6]

Under the additional $C^{2}$ bounded-second-derivative, path-existence, and integrated-direction hypotheses, the finite-data NTK remains uniformly close to its initialization (Finite-data fixed-time NTK stability under integrably bounded training directions).

[F7]

The probability limit uses the event-probability convention in Convergence in probability.

[F8]

A globally Lipschitz real function is absolutely continuous on every compact interval ( $C^{1}$ implies Lipschitz, Lipschitz implies absolutely continuous, and absolutely continuous implies continuous and bounded variation).

[F9]

Under countable choice, every absolutely continuous real function on a compact interval is differentiable almost everywhere (Absolutely continuous functions have integrable derivatives).

Proof

technique · establish almost-sure differentiability at initialization, induct on layer covariances and Jacobian blocks, then apply the separate fixed-time drift estimate

1.1A1F1F2F5F8F9givenconstruct

By [F8], $σ$ is absolutely continuous on every compact interval; applying [F9] on $[- m, m]$ for each positive integer $m$ shows that it is differentiable outside a Lebesgue-null set. The forward difference quotients $s_{0} (x) = {lim sup}_{k \to \infty} k (σ (x + 1 / k) - σ (x))$ define a bounded Borel function and equal $σ' (x)$ wherever the derivative exists. Since the given representative $s$ also equals $σ'$ almost everywhere, there is a Lebesgue-null set $E$ outside which $σ$ is differentiable and $s = s_{0} = σ'$ . For any fixed finite width tuple, condition on the preceding layers: every coordinate of every data-point preactivation is normal with variance $∥ z^{ℓ} (x_{i}) ∥_{2}^{2} / n_{ℓ} + β^{2} \geq β^{2}$ . Thus each avoids $E$ almost surely, and a finite union shows all preactivations at initialization avoid $E$ almost surely. The finite network is therefore parameter-differentiable at initialization for every data point, and [F5] applies with derivative values $s (U)$ . The first-layer covariance is $Σ^{1}$ ; its diagonal is positive, and its Jacobian Gram block is exactly $K_{(i, a), (j, b)}^{1} = δ_{a b} Σ_{i j}^{1}$ because different neurons use disjoint parameters.

2.1F1F2F3F4step 1.1given

Let $C_{n}$ be the conditional empirical covariance of the layer- $(r - 1)$ preactivation vectors across the $n_{r - 1}$ neurons. By induction $C_{n} \to Σ^{r - 1}$ in probability; conditional on the preceding layer, these vectors are iid centered Gaussian with covariance $C_{n}$ and diagonal entries at least $β^{2}$ . Choose $R > 1 + \max_{i} Σ_{i i}^{r - 1}$ , so $P (\max_{i} (C_{n})_{i i} > R) \to 0$ . Write $L_{σ}$ for a global Lipschitz constant. If $Y_{a} = σ (U_{a}^{r - 1} (x_{i})) σ (U_{a}^{r - 1} (x_{j}))$ , the Lipschitz bound $∣ σ (u) ∣ \leq ∣ σ (0) ∣ + L_{σ} ∣ u ∣$ , Gaussian fourth moments, and Cauchy--Schwarz give $E [Y_{a}^{2} ∣ C_{n}] \leq M_{R}$ whenever all diagonal entries of $C_{n}$ are at most $R$ , for a finite constant $M_{R}$ . The conditional variance bound in [F3] therefore gives $P (∣ n_{r - 1}^{- 1} \sum_{a} Y_{a} - E [Y_{a} ∣ C_{n}] ∣ > η) \leq P (\max_{i} (C_{n})_{i i} > R) + M_{R} / (n_{r - 1} η^{2}) \to 0$ . The conditional mean is a bivariate Gaussian feature expectation and converges by [F4] to $E [σ (U_{i}) σ (U_{j})]$ . For $V_{a} = s (U_{a}^{r - 1} (x_{i})) s (U_{a}^{r - 1} (x_{j}))$ , the same conditional variance estimate holds with $M_{R} = ∥ s ∥_{\infty}^{4}$ , and [F4] makes its conditional mean converge to ${\dot{Σ}}_{i j}^{r}$ . Adding the bias Gram gives $Σ^{r}$ . Each empirical feature Gram plus that bias Gram is positive semidefinite with diagonal at least $β^{2}$ ; the cone of positive-semidefinite matrices is closed, so its deterministic limit $Σ^{r}$ is positive semidefinite and defines the next Gaussian pairs. This proves the covariance and derivative recursions at every layer.

2.2F1F5step 1.1algebra

For a finite network, [F5] gives the exact tangent-kernel block recursion $K_{(i, c), (j, d)}^{r} = δ_{c d} (n_{r - 1}^{- 1} \sum_{a} Z_{a i}^{r - 1} Z_{a j}^{r - 1} + β^{2}) + n_{r - 1}^{- 1} \sum_{a, b} W_{c a}^{(r - 1)} W_{d b}^{(r - 1)} s (U_{a i}^{r - 1}) s (U_{b j}^{r - 1}) K_{(i, a), (j, b)}^{r - 1}$ . The first term is the current-layer weight-and-bias Gram, and the second is the contribution of all earlier shared parameters. At the first layer this block is the one in step 1.1.

3.1F2F3step 2.1step 2.2algebra

At this sequential stage the previous widths have already been taken to their limits while $n_{r - 1}$ and $n_{r}$ are fixed, so the inductive block has limit $K_{(i, a), (j, b)}^{r - 1} = δ_{a b} Θ_{i j}^{r - 1}$ for the finitely many $a, b$ . Substituting this block in the Jacobian term of step 2.2 leaves $Θ_{i j}^{r - 1} n_{r - 1}^{- 1} \sum_{a} W_{c a}^{(r - 1)} W_{d a}^{(r - 1)} q_{a}$ , where $q_{a} = s (U_{a}^{r - 1} (x_{i})) s (U_{a}^{r - 1} (x_{j}))$ and $∣ q_{a} ∣ \leq ∥ s ∥_{\infty}^{2}$ . The current weights are independent standard normals, independent of the preceding $U$ values. Conditional on those values, for $c = d$ the mean is $Θ_{i j}^{r - 1} n_{r - 1}^{- 1} \sum_{a} q_{a}$ and the variance is at most $2 (Θ_{i j}^{r - 1})^{2} ∥ s ∥_{\infty}^{4} / n_{r - 1}$ , using $Var (W^{2}) = 2$ ; for $c \neq d$ the mean is zero and the variance is at most $(Θ_{i j}^{r - 1})^{2} ∥ s ∥_{\infty}^{4} / n_{r - 1}$ , using independence of the two rows. The empirical average of $q_{a}$ converges to ${\dot{Σ}}_{i j}^{r}$ by step 2.1, so the Jacobian term converges to $δ_{c d} Θ_{i j}^{r - 1} {\dot{Σ}}_{i j}^{r}$ . The covariance term converges to $δ_{c d} Σ_{i j}^{r}$ by steps 2.1–2.2. Hence the limiting block is $δ_{c d} (Σ_{i j}^{r} + Θ_{i j}^{r - 1} {\dot{Σ}}_{i j}^{r}) = δ_{c d} Θ_{i j}^{r}$ . Since the final layer has one output, its scalar block is $K_{\infty}$ .

4.1F5F6F7step 3.1algebra∎

Under the additional $C^{2}$ training hypotheses, [F6] and the initialization convergence from step 3.1 give $\sup_{t \leq T} ∥ K_{t} - K_{\infty} ∥_{o p} \leq \sup_{t \leq T} ∥ K_{t} - K_{0} ∥_{o p} + ∥ K_{0} - K_{\infty} ∥_{o p} \to 0$ in probability. The parameter-flow equation and [F5] give ${\dot{f}}_{t} (x_{i}) = - N^{- 1} \sum_{j} K_{t} (i, j) d_{j} (t)$ . Integrating and subtracting the equation with $K_{\infty}$ yields $\sup_{t \leq T} ∥ f_{t} - f_{0} + N^{- 1} \int_{0}^{t} K_{\infty} d (s) d s ∥_{N} \leq N^{- 1} \sup_{t \leq T} ∥ K_{t} - K_{\infty} ∥_{o p} \int_{0}^{T} ∥ d (s) ∥_{N} d s$ . For any $M > 0$ and $δ > 0$ , the product exceeds $δ$ only if $\int_{0}^{T} ∥ d (s) ∥_{N} d s > M$ or $\sup_{t \leq T} ∥ K_{t} - K_{\infty} ∥_{o p} > δ N / M$ ; first use stochastic boundedness to choose $M$ , then use kernel convergence. Thus the right side tends to zero in probability.

Depends on

Used by

Dependency tree · two levels

82 results within two dependency steps of this one, each drawn at its shortest distance from it. An arrow runs from a result to what uses it, so the chart reads left to right and ends at this result, which carries a heavier outline. Every node is a link to that result. Click elsewhere on the chart to enlarge it.

Sources

Arthur Jacot, Franck Gabriel, and Clément Hongler, Neural Tangent Kernel: Convergence and Generalization in Neural Networks (standard reference, not scraped)