TheoremStatement: Literature-sourcedProof: AI-adaptedPipeline-generated

How statement and proof provenance work

The first chip identifies the source of the statement or construction; the second identifies the source of its local proof or verification.

Literature-sourced: the exact statement appears in a cited source; only wording and notation differ.
AI-adapted: a semantically identical restatement of literature-sourced material, modulo indexing, notation, and boundary cases adopted by the library.
AI-generated: a genuinely novel statement formulated by AI, with no source for the claim itself.

These labels describe origin, not correctness: citations and verification chips remain separate evidence.

Expected linear contraction of an SVRG epoch

Statement

Let $n, d \geq 1$ , and let $f_{i} : R^{d} \to R$ be convex, continuously differentiable, with $L$ -Lipschitz gradients for a common $L > 0$ . Put $F = n^{- 1} \sum_{i = 1}^{n} f_{i}$ , assume $F$ is $μ$ -strongly convex for $μ > 0$ , and let $x^{*}$ satisfy $\nabla F (x^{*}) = 0$ . In one SVRG epoch Finite-sum SVRG, take a deterministic snapshot $\tilde{x}$ , set $x_{0} = \tilde{x}$ , choose

$η = \frac{1}{10 L}, m = 2 n,$

and choose each uniform inner index independently of the past. Assume $n \geq 10 L / μ$ . Choose the output index $J$ independently and uniformly from ${0, \dots, m - 1}$ . Then

$E [F (x_{J}) - F (x^{*})] \leq q (F (\tilde{x}) - F (x^{*})), q = \frac{1}{μ η (1 - 2 L η) m} + \frac{2 L η}{1 - 2 L η} \leq \frac{7}{8} .$

The expectation is over the inner indices and $J$ ; this is an epoch expectation bound and asserts no almost-sure convergence.

Facts & Assumptions

Given: The functions, smoothness, strong convexity, stationary minimizer, snapshot, inner iteration, and independent uniform output index in the Statement. All finite-time moments exist because a finite horizon has only finitely many index histories and each update evaluates finite gradients at finite points.

[F1]

Finite-sum SVRG: $x_{t + 1} = x_{t} - η v_{t}$ , with $v_{t} = \nabla f_{I_{t}} (x_{t}) - \nabla f_{I_{t}} (\tilde{x}) + \nabla F (\tilde{x})$ and independent uniform $I_{t}$ .

[F2]

SVRG conditional unbiasedness and variance bound: conditionally on the history, $E [v_{t}] = \nabla F (x_{t})$ and $E ∥ v_{t} ∥_{2}^{2} \leq 4 L (F (x_{t}) - F (x^{*})) + 4 L (F (\tilde{x}) - F (x^{*}))$ .

[F3]

Descent inequality for a Lipschitz gradient: an $L$ -Lipschitz gradient gives $f_{i} (z) \leq f_{i} (w) + ⟨ \nabla f_{i} (w), z - w ⟩ + (L / 2) ∥ z - w ∥_{2}^{2}$ .

[F4]

Gradient of a finite sum is the finite sum of gradients: $\nabla F (w) = n^{- 1} \sum_{i} \nabla f_{i} (w)$ .

[F5]

Strong convexity makes a differentiable gradient strongly monotone: $F (y) \geq F (x) + ⟨ \nabla F (x), y - x ⟩ + (μ / 2) ∥ y - x ∥_{2}^{2}$ .

[F6]

Differentiable convex functions are characterized by the gradient inequality: convex differentiable $F$ satisfies $F (y) \geq F (x) + ⟨ \nabla F (x), y - x ⟩$ .

[F7]

The Euclidean inner product $⟨ x, y ⟩ = \sum_{k < n} x_{k} y_{k}$ on $R^{n}$ : $∥ z ∥_{2}^{2} = ⟨ z, z ⟩$ and the inner product is bilinear.

[F8]

Laws of finite sums and finite products: finite sums are additive, obey scaling, and telescope.

[A1]

The Axiom of Choice: AC is assumed for the conditional-expectation interfaces used in [F2].

Proof

Given: The objects and assumptions in the Statement.

1.1A1F1F2F3F4F6F7F8givenalgebra

Write $Δ_{t} = F (x_{t}) - F (x^{*})$ and $\tilde{Δ} = F (\tilde{x}) - F (x^{*})$ . Averaging [F3] and using [F4], [F8] gives the $L$ -smooth descent inequality for $F$ . By [F1], [F2], [F6], and [F7], expansion of the squared distance and the convex gradient inequality yield $E [∥ x_{t + 1} - x^{*} ∥_{2}^{2} ∣ F_{t}] \leq ∥ x_{t} - x^{*} ∥_{2}^{2} - 2 η Δ_{t} + η^{2} E [∥ v_{t} ∥_{2}^{2} ∣ F_{t}] \leq ∥ x_{t} - x^{*} ∥_{2}^{2} - 2 η (1 - 2 L η) Δ_{t} + 4 η^{2} L \tilde{Δ}$ .

2.1A1F5F8step 1.1givenalgebra

Average the conditional inequality in step 1.1 over the finite index histories and sum for $t = 0, \dots, m - 1$ . By [F8] the finite sum telescopes; the terminal squared distance is nonnegative and $x_{0} = \tilde{x}$ , hence $m^{- 1} \sum_{t = 0}^{m - 1} E [Δ_{t}] \leq ∥ \tilde{x} - x^{*} ∥_{2}^{2} / [2 η (1 - 2 L η) m] + [2 η L / (1 - 2 L η)] \tilde{Δ}$ . By [F5] and $\nabla F (x^{*}) = 0$ , $∥ \tilde{x} - x^{*} ∥_{2}^{2} \leq 2 \tilde{Δ} / μ$ ; the independent uniform $J$ makes its expected gap the average on the left, giving the stated $q$ .

3.1step 2.1givenalgebra∎

With $η = 1 / (10 L)$ and $m = 2 n$ , $1 - 2 L η = 4 / 5$ , so $q = 25 L / (4 μ n) + 1 / 4 \leq 5 / 8 + 1 / 4 = 7 / 8$ by $n \geq 10 L / μ$ . This proves the Statement.

Depends on

Used by

A regularized logistic finite sum with an exact minimizer Example

Dependency tree · two levels

56 results within two dependency steps of this one, each drawn at its shortest distance from it. An arrow runs from a result to what uses it, so the chart reads left to right and ends at this result, which carries a heavier outline. Every node is a link to that result. Click elsewhere on the chart to enlarge it.

Sources

Rie Johnson and Tong Zhang, Accelerating Stochastic Gradient Descent using Predictive Variance Reduction, NeurIPS 2013 (standard reference, not scraped)
Sébastien Bubeck, Convex Optimization: Algorithms and Complexity, complete monograph (standard reference, not scraped)