Heavy-ball momentum

Let $d\ge 1$ be an integer and let $F:{\mathbb{R}}^d\to \mathbb{R}$ be continuously Fréchet differentiable: at every $w$, $DF(w)$ is a linear first-order approximation with normwise remainder tending to zero, and $w\mapsto DF(w)$ is continuous. Write its coordinate gradient as $\mathrm{\nabla }F(w)=({\mathrm{\partial }}_{j}F(w){)}_{0\le j<d}$. Given an initial vector $w_{\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{t}}\in {\mathbb{R}}^d$, a constant step size $\alpha >0$, a constant momentum parameter $\beta \ge 0$, and an integer horizon $T\ge 1$, initialize $w_0=w_1=w_{\mathrm{i}\mathrm{n}\mathrm{i}\mathrm{t}}$. For each integer $t=1,\dots ,T-1$, set

$w_{t+1}:=w_t-\alpha \mathrm{\nabla }F(w_t)+\beta (w_t-w_{t-1})\mathrm{.}$

Output $(w_0,w_1,\dots ,w_T)$. The gradient is evaluated at the current iterate $w_t$. When $\beta =0$, every update is the fixed-step gradient-descent update with step size $\alpha$.

Gradient descent

The total (Fréchet) derivative $Df(a)$ as the linear first-order approximation with $o(\mathrm{\parallel }h{\mathrm{\parallel }}_2)$ remainder

The Jacobian matrix of partial derivatives and the gradient in the scalar-valued case