Bias-corrected AMSGrad update

Fix a dimension d at least 1, an initial point x_1 in R^d, a constant step size eta>0, a denominator constant epsilon>0, and parameters beta_1,beta_2 in [0,1). Set m_0=v_0=\widehat v^{\max}_0=0. Given a gradient g_t in R^d at each integer time t>=1, calculate

$m_t={\beta }_1{m}_{t-1}+(1-{\beta }_1)g_t,v_t={\beta }_2{v}_{t-1}+(1-{\beta }_2)(g_t\odot g_t),$

${\widehat{m}}_t=\frac{m_t}{1-{\beta }_1^t},{\widehat{v}}_t=\frac{v_t}{1-{\beta }_2^t},{\widehat{v}}_t^{\max }=\max ({\widehat{v}}_{t-1}^{\max },{\widehat{v}}_t),$

where \odot and max are coordinatewise. Update

coordinatewise. Each coordinate of v_t and \widehat v^{\max}_t is nonnegative, and the denominator is at least epsilon>0, so every finite update is defined.

This carrier defines a bias-corrected AMSGrad variant under the assigned denominator convention. Reddi–Kale–Kumar's Algorithm 2 takes the running maximum of the uncorrected second-moment recurrence and uses that maximum directly in the denominator; it does not apply Adam's second-moment bias correction. The recurrences are not identified here. No AMSGrad convergence theorem is claimed.

For $0\le {\beta }_j<1$ and integer $t\ge 1$, Monotonicity of $x\mapsto x^n$ and of $n\mapsto a^n$ gives $0\le {\beta }_j^t<1$, so each bias-correction denominator is positive. Induction in the second-moment recurrence, Squares of nonzero elements are positive, Sign rules for products and monotonicity of multiplication and Multiplication by zero: $0\cdot a=0$ give $v_t\ge 0$. Division by the positive bias-correction factor preserves nonnegativity by Inverses of positives are positive, and reciprocation reverses order, so ${\widehat{v}}_t\ge 0$ and its square root exists by Square roots exist: a unique $\sqrt{a}\ge 0$ with $(\sqrt{a}{)}^2=a$; the positives are $\{x^2:x\mathrm{\ne }0\}$. The coordinatewise maximum is the maximum of a two-element real set, which exists by Maximum and minimum of a set and Every nonempty finite set of reals has a maximum and a minimum and is nonnegative because it is one of its two nonnegative entries. Thus every denominator is at least $ϵ>0$.