Bias-corrected Adam update

Fix a dimension d at least 1, an initial point x_1 in R^d, a constant step size eta>0, a denominator constant epsilon>0, and parameters beta_1,beta_2 in [0,1). Set m_0=v_0=0. Given a gradient g_t in R^d at each integer time t>=1, calculate

$m_t={\beta }_1{m}_{t-1}+(1-{\beta }_1)g_t,v_t={\beta }_2{v}_{t-1}+(1-{\beta }_2)(g_t\odot g_t),$

where \odot denotes coordinatewise multiplication. Define the bias-corrected moments

${\widehat{m}}_t=\frac{m_t}{1-{\beta }_1^t},{\widehat{v}}_t=\frac{v_t}{1-{\beta }_2^t},$

and update, coordinatewise,

$$x_{t+1}=x_t-\eta \frac{{\widehat{m}}_t}{\sqrt{{\widehat{v}}_t}+ϵ}\mathrm{.}$$

All operations involving vectors in the update are coordinatewise except the ordinary vector addition in the moment recurrences. Since 1-\beta_j^t>0 for t>=1 and v_t has nonnegative coordinates, every denominator is positive; the procedure is defined at every finite step for every finite gradient sequence.

This item defines the update only. It makes no convergence or regret claim.

The real field used here is complete by The Cauchy-sequence reals have the least-upper-bound property, so the complete-ordered-field hypothesis of Square roots exist: a unique $\sqrt{a}\ge 0$ with $(\sqrt{a}{)}^2=a$; the positives are $\{x^2:x\mathrm{\ne }0\}$ holds.

For $0\le {\beta }_j<1$ and integer $t\ge 1$, Monotonicity of $x\mapsto x^n$ and of $n\mapsto a^n$ gives $0\le {\beta }_j^t<1$, so each bias-correction denominator is positive. For each coordinate, a squared gradient is nonnegative by Squares of nonzero elements are positive (with the zero case equal to zero); induction in the recurrence, using the nonnegative coefficients ${\beta }_2$ and $1-{\beta }_2$ and the product sign rules Sign rules for products and monotonicity of multiplication, Multiplication by zero: $0\cdot a=0$, gives $v_t\ge 0$. Division by the positive bias-correction factor preserves nonnegativity by Inverses of positives are positive, and reciprocation reverses order, so ${\widehat{v}}_t\ge 0$; its square root exists by Square roots exist: a unique $\sqrt{a}\ge 0$ with $(\sqrt{a}{)}^2=a$; the positives are $\{x^2:x\mathrm{\ne }0\}$, and $\sqrt{{\widehat{v}}_t}+ϵ>0$. These facts justify every finite update.