TheoremStatement: Literature-sourcedProof: AI-adaptedPipeline-generated

How statement and proof provenance work

The first chip identifies the source of the statement or construction; the second identifies the source of its local proof or verification.

Literature-sourced: the exact statement appears in a cited source; only wording and notation differ.
AI-adapted: a semantically identical restatement of literature-sourced material, modulo indexing, notation, and boundary cases adopted by the library.
AI-generated: a genuinely novel statement formulated by AI, with no source for the claim itself.

These labels describe origin, not correctness: citations and verification chips remain separate evidence.

Diagonal AdaGrad regret bound

Statement

Use one-based coordinate labels $z_{j} : = z (j - 1)$ for $1 \leq j \leq d$ , as in Diagonal AdaGrad for online convex losses. Let $C \subseteq R^{d}$ be nonempty, closed, and convex, and suppose its coordinate diameters $D_{j} = \sup {∣ x_{j} - u_{j} ∣ : x, u \in C}$ are finite. Run diagonal AdaGrad Diagonal AdaGrad for online convex losses for $T \geq 1$ rounds on convex losses $ℓ_{t} : C \to R$ , with $g_{t} \in \partial ℓ_{t} (x_{t})$ , step size $η > 0$ , and accumulator $s_{t, j} = \sqrt{ϵ^{2} + \sum_{k = 1}^{t} g_{k, j}^{2}}$ for $ϵ > 0$ . For every comparator $u \in C$ ,

$\sum_{t = 1}^{T} (ℓ_{t} (x_{t}) - ℓ_{t} (u)) \leq \sum_{j = 1}^{d} (\frac{D_{j}^{2}}{2 η} + η) s_{T, j} .$

Facts & Assumptions

Given: The set, losses, subgradients, finite horizon, positive parameters, and update in the Statement.

[F1]

Diagonal AdaGrad for online convex losses: $A_{t} = diag (s_{t, 1}, \dots, s_{t, d})$ and the update minimizes the stated diagonal quadratic objective.

[F2]

One-step diagonal AdaGrad regret inequality: for each $u \in C$ , $⟨ g_{t}, x_{t} - u ⟩ \leq (∥ x_{t} - u ∥_{A_{t}}^{2} - ∥ x_{t + 1} - u ∥_{A_{t}}^{2}) / (2 η) + (η / 2) ∥ g_{t} ∥_{A_{t}^{- 1}}^{2}$ .

[F3]

Subgradients and the subdifferential of a convex function: $g_{t} \in \partial ℓ_{t} (x_{t})$ means $ℓ_{t} (u) \geq ℓ_{t} (x_{t}) + ⟨ g_{t}, u - x_{t} ⟩$ for all $u \in C$ .

[F4]

Finite sums and finite products, by recursion: finite sums obey their recursive addition and finite-sum algebra.

[F5]

The Euclidean inner product $⟨ x, y ⟩ = \sum_{k < n} x_{k} y_{k}$ on $R^{n}$ : for diagonal $A_{t}$ , $∥ z ∥_{A_{t}}^{2} = \sum_{j} s_{t, j} z_{j}^{2}$ .

[F6]

The Cauchy-sequence reals have the least-upper-bound property establishes completeness of the real field in the sense of Complete ordered field (least-upper-bound property): every nonempty set of reals bounded above has a real supremum. Since $C$ is nonempty, each coordinate-difference set contains $0$ ; the finite-diameter hypothesis makes it bounded above, so each $D_{j}$ exists.

[F7]

Laws of finite sums and finite products: finite sums are additive, monotone termwise, and telescope.

[F8]

Squares of nonzero elements are positive makes every nonzero square positive; Multiplication by zero: $0 \cdot a = 0$ gives square zero for zero. Thus every square is nonnegative.

[F9]

Squaring is monotone on the nonnegatives: squaring is monotone on nonnegative reals.

[F10]

Complete ordered field (least-upper-bound property), Sign rules for products and monotonicity of multiplication, and Inverses of positives are positive, and reciprocation reverses order: order is total, and multiplication or division by a positive real preserves order.

[F11]

Square roots exist: a unique $\sqrt{a} \geq 0$ with $(\sqrt{a})^{2} = a$ ; the positives are ${x^{2} : x \neq 0}$ : each accumulator is the nonnegative root of its displayed radicand, so its square equals that radicand.

Proof

Given: The objects and hypotheses in the Statement.

1.1F2F4F5F7givenalgebra

Sum [F2] for $t = 1, \dots, T$ . Finite-sum algebra [F4], [F7] telescopes each coordinate to $\sum_{t} ⟨ g_{t}, x_{t} - u ⟩ \leq (∥ x_{1} - u ∥_{A_{1}}^{2} + \sum_{t = 2}^{T} ∥ x_{t} - u ∥_{A_{t} - A_{t - 1}}^{2} - ∥ x_{T + 1} - u ∥_{A_{T}}^{2}) / (2 η) + (η / 2) \sum_{t} ∥ g_{t} ∥_{A_{t}^{- 1}}^{2}$ . The last metric term is nonnegative and may be dropped.

2.1F1F4F5F6F7F8F9F11step 1.1givenalgebra

Set $s_{0, j} = ϵ$ . By [F1], [F11], and the finite-sum recursion [F4], $s_{t, j}^{2} - s_{t - 1, j}^{2} = g_{t, j}^{2} \geq 0$ by [F8]. Both roots are nonnegative by [F11], so [F9] gives $s_{t, j} \geq s_{t - 1, j}$ , and $s_{t - 1, j} > 0$ follows from the positive accumulator in [F1]. Thus every metric increment in step 1.1 is nonnegative. Since $x_{t}, u \in C$ , $∣ x_{t, j} - u_{j} ∣ \leq D_{j}$ . [F5] then bounds the initial and metric-increment terms in step 1.1 by $\sum_{j} D_{j}^{2} s_{T, j}$ : for each $j$ , $s_{1, j} + \sum_{t = 2}^{T} (s_{t, j} - s_{t - 1, j}) = s_{T, j}$ by [F7].

3.1F1F4F5F7F10step 1.1step 2.1givenalgebra

The accumulator identity and monotonicity from step 2.1 give $g_{t, j}^{2} / s_{t, j} = (s_{t, j} - s_{t - 1, j}) (s_{t, j} + s_{t - 1, j}) / s_{t, j} \leq 2 (s_{t, j} - s_{t - 1, j})$ by [F10]. Summing and telescoping by [F7] gives $\sum_{t} ∥ g_{t} ∥_{A_{t}^{- 1}}^{2} \leq 2 \sum_{j} (s_{T, j} - ϵ) \leq 2 \sum_{j} s_{T, j}$ . Substitute in step 1.1 and combine with step 2.1 to obtain the linearized-loss bound.

4.1F3F7step 3.1givenalgebra∎

By [F3], each convex-loss difference satisfies $ℓ_{t} (x_{t}) - ℓ_{t} (u) \leq ⟨ g_{t}, x_{t} - u ⟩$ . Summing these inequalities using [F7] and applying step 3.1 proves the stated regret bound.

Depends on

Used by

Nothing in the library uses this result yet.

Dependency tree · two levels

50 results within two dependency steps of this one, each drawn at its shortest distance from it. An arrow runs from a result to what uses it, so the chart reads left to right and ends at this result, which carries a heavier outline. Every node is a link to that result. Click elsewhere on the chart to enlarge it.

Sources

John Duchi, Elad Hazan, and Yoram Singer, Adaptive Subgradient Methods for Online Learning and Stochastic Optimization, JMLR 12 (2011) (standard reference, not scraped)
Sébastien Bubeck, Convex Optimization: Algorithms and Complexity, complete monograph (standard reference, not scraped)