RMSProp with an exponential second-moment estimate

Fix a dimension $d\ge 1$, an initial vector $x_1\in {\mathbb{R}}^d$, a finite or infinite sequence of finite gradient vectors $g_t\in {\mathbb{R}}^d$, positive step sizes ${\eta }_t>0$, a decay parameter $\rho \in [0,1)$, and a denominator constant $ϵ>0$. Use one-based coordinate notation $z_i:=z(i-1)$ for every vector $z\in {\mathbb{R}}^d$ and $1\le i\le d$. Set $r_0=0$. At each integer round $t\ge 1$, update the second-moment estimate coordinatewise by

$r_{t,i}=\rho r_{t-1,i}+(1-\rho )g_{t,i}^2(1\le i\le d),$

and then set

$x_{t+1,i}=x_{t,i}-{\eta }_t\frac{g_{t,i}}{\sqrt{r_{t,i}}+ϵ}(1\le i\le d)\mathrm{.}$

For a finite gradient sequence of length $T\ge 0$, perform rounds $1\le t\le T$ and return the trajectory $(x_1,\dots ,x_{T+1})$; when $T=0$, return $(x_1)$. For an infinite gradient sequence, perform every round $t\ge 1$, thereby defining and returning the iterate sequence $(x_t{)}_{t\ge 1}$. The root, addition, and division in the update are coordinatewise. The procedure makes no convergence or regret claim.

The Euclidean inner product $⟨x,y⟩={\sum }_{k<n}{x}_k{y}_k$ on ${\mathbb{R}}^n$: vectors in ${\mathbb{R}}^d$ have the stated real coordinates, and vector updates are interpreted coordinatewise.

Squares of nonzero elements are positive: every nonzero squared gradient coordinate is positive.

Multiplication by zero: $0\cdot a=0$: a zero gradient coordinate has square zero.

Square roots exist: a unique $\sqrt{a}\ge 0$ with $(\sqrt{a}{)}^2=a$; the positives are $\{x^2:x\mathrm{\ne }0\}$: each nonnegative real has a unique nonnegative square root.

Ordered field

Sign rules for products and monotonicity of multiplication: the decay coefficients and their complements are nonnegative, so the moment recurrence preserves nonnegativity.

Bottou, Curtis, and Nocedal, RMSprop update

Reddi, Kale, and Kumar, adaptive-method setup

This is the no-momentum RMSProp variant with a positive constant added outside the square root. It has no bias correction and defines only an update procedure.