LemmaStatement: AI-adaptedProof: AI-adaptedPipeline-generatedjudge pass (gpt-6-luna)

How statement and proof provenance work

The first chip identifies the source of the statement or construction; the second identifies the source of its local proof or verification.

Literature-sourced: the exact statement appears in a cited source; only wording and notation differ.
AI-adapted: a semantically identical restatement of literature-sourced material, modulo indexing, notation, and boundary cases adopted by the library.
AI-generated: a genuinely novel statement formulated by AI, with no source for the claim itself.

These labels describe origin, not correctness: citations and verification chips remain separate evidence.

RMSProp, Adam, and AMSGrad moment updates are well-defined

Statement

For every finite admissible run of the RMSProp, uncorrected Adam, or uncorrected AMSGrad procedures defined on this page, all moment coordinates are finite and all second-moment coordinates are nonnegative. Every denominator specified by the procedure is positive. For AMSGrad, a zero historical-maximum coordinate has zero current and previous first moments, and the raw second moment is positive whenever its historical maximum is positive; the zero-ratio convention is therefore consistent. Each weighted minimization over the compact feasible set attains a minimum.

Statement references

RMSProp with an exponential second-moment estimate: defines the exponential second-moment recurrence and positive stabilizer.

Uncorrected Adam for online convex losses: defines the uncorrected scalar moment recurrences, admissible denominators, and clipping update.

Uncorrected AMSGrad with a coordinatewise historical maximum: defines the raw-moment recurrence, historical maximum, zero-coordinate convention, and compact-set minimization.

Reddi, Kale, and Kumar, adaptive-moment procedures

Bottou, Curtis, and Nocedal, RMSprop recurrence

Facts & Assumptions

Given: Finite gradient inputs for the three procedures, their parameter ranges and initial zero moments, and the admissibility conditions in their Definitions.

[F1]

RMSProp with an exponential second-moment estimate: $r_{0} = 0$ and $r_{t, i} = ρ r_{t - 1, i} + (1 - ρ) g_{t, i}^{2}$ , with $0 \leq ρ < 1$ and $ϵ > 0$ .

[F2]

Uncorrected Adam for online convex losses: $m_{0} = v_{0} = 0$ , $v_{t} = β_{2} v_{t - 1} + (1 - β_{2}) g_{t}^{2}$ , and each used radicand $v_{t} + ϵ$ is positive.

[F3]

Uncorrected AMSGrad with a coordinatewise historical maximum: $m_{0} = v_{0} = {\hat{v}}_{0} = 0$ , $β_{2} \in (0, 1)$ , $v_{t, i} = β_{2} v_{t - 1, i} + (1 - β_{2}) g_{t, i}^{2}$ , and ${\hat{v}}_{t, i} = \max {{\hat{v}}_{t - 1, i}, v_{t, i}}$ .

[F4]

The principle of mathematical induction: a property true at the initial index and preserved at each finite successor holds at every finite round.

[F5]

Squares of nonzero elements are positive, Multiplication by zero: $0 \cdot a = 0$ , and Sign rules for products and monotonicity of multiplication: coordinate squares are nonnegative, and nonnegative coefficients preserve nonnegativity under sums and products.

[F6]

Square roots exist: a unique $\sqrt{a} \geq 0$ with $(\sqrt{a})^{2} = a$ ; the positives are ${x^{2} : x \neq 0}$ : each nonnegative real has a unique nonnegative square root.

[F7]

Maximum and minimum of a set and Every nonempty finite set of reals has a maximum and a minimum: every two-element set of real numbers has a maximum.

[F8]

For a nonempty subset of $R^{n}$ with $n \geq 1$ , compactness, closedness and boundedness, pseudocompactness, and attainment of extrema by every continuous real-valued function are equivalent: a compact subset of $R^{d}$ is bounded, and a continuous real-valued function on a nonempty compact subset attains its minimum.

[F9]

The Euclidean inner product $⟨ x, y ⟩ = \sum_{k < n} x_{k} y_{k}$ on $R^{n}$ , Laws of finite sums and finite products, Squaring is monotone on the nonnegatives, and Cauchy-Schwarz $∣ ⟨ x, y ⟩ ∣ \leq ∥ x ∥_{2} ∥ y ∥_{2}$ with its equality case, the triangle inequality for $∥ \cdot ∥_{2}$ , the parallelogram law and polarisation: $∣ x_{i} - z_{i} ∣ \leq ∥ x - z ∥_{2}$ , the Euclidean norm satisfies the triangle inequality, and finite sums of nonnegative terms remain nonnegative.

Proof

Given: The data and hypotheses in the Statement.

1.1F1F2F3F4F5F9giveninductionalgebra

For each procedure, induction over its finite horizon shows that every moment is a finite real and each second moment is nonnegative. The initial moments are zero. At a successor round, each recurrence is a finite sum of products of finite reals by [F1]–[F3] and [F9]; the coefficients $ρ, 1 - ρ, β_{2}, 1 - β_{2}$ are nonnegative, and each squared gradient is nonnegative by [F5], so the next second moment remains nonnegative.

2.1F1F2F6step 1.1givenalgebra

For RMSProp, [F6] supplies $\sqrt{r_{t, i}} \geq 0$ , so $\sqrt{r_{t, i}} + ϵ > 0$ by [F1]. For Adam, [F6] supplies $\sqrt{v_{t} + ϵ}$ because $v_{t} \geq 0$ and $ϵ \geq 0$ ; its positivity is the explicit admissibility condition in [F2]. Thus each stated quotient has a positive denominator.

2.2F3F5F7step 1.1giveninductionalgebra

In AMSGrad, both entries in each maximum are finite and nonnegative by [F3], [F5], and step 1.1. The maximum exists by [F7] and is nonnegative. If ${\hat{v}}_{t, i} = 0$ , then ${\hat{v}}_{t - 1, i} = v_{t, i} = 0$ . Since $β_{2} > 0$ , $1 - β_{2} > 0$ , and both summands defining $v_{t, i}$ are nonnegative, $v_{t - 1, i} = 0$ and $g_{t, i} = 0$ . Repeating this argument back to time zero gives every earlier $g_{s, i} = 0$ ; the first-moment recurrence then gives $m_{t, i} = m_{t - 1, i} = 0$ . Conversely, if ${\hat{v}}_{t, i} > 0$ , some $v_{s, i} > 0$ for $s \leq t$ , and then $v_{u + 1, i} \geq β_{2} v_{u, i} > 0$ for each $u \geq s$ , so $v_{t, i} > 0$ . The zero-ratio convention in [F3] is therefore consistent with the recurrence.

3.1F3F8F9step 2.2givenalgebra∎

Fix a round and its finite vector $y$ and nonnegative weights $q_{i} = \sqrt{{\hat{v}}_{t, i}}$ . Since $F$ is compact, [F8] bounds $∥ x ∥_{2}$ by some $B$ for $x \in F$ . For $x, z \in F$ , coordinatewise expansion and [F9] give $∣ \sum_{i} q_{i} (x_{i} - y_{i})^{2} - \sum_{i} q_{i} (z_{i} - y_{i})^{2} ∣ \leq ∥ x - z ∥_{2} \sum_{i} q_{i} (2 B + 2 ∥ y ∥_{2}) .$ Thus the objective is continuous on $F$ . Since $F$ is nonempty and compact, [F8] gives a minimizer. The objective can be constant along zero-weight coordinates, so allowing any minimizer is necessary and sufficient. This proves the remaining well-definedness claim.

Depends on

Used by

Cited to discharge well-definedness by RMSProp with an exponential second-moment estimate, Uncorrected Adam for online convex losses and Uncorrected AMSGrad with a coordinatewise historical maximum.

Dependency tree · two levels

57 results within two dependency steps of this one, each drawn at its shortest distance from it. An arrow runs from a result to what uses it, so the chart reads left to right and ends at this result, which carries a heavier outline. Every node is a link to that result. Click elsewhere on the chart to enlarge it.

Sources

Léon Bottou, Frank E. Curtis, and Jorge Nocedal, Optimization Methods for Large-Scale Machine Learning (standard reference, not scraped)
Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar, On the Convergence of Adam and Beyond (standard reference, not scraped)