Stochastic gradient descent

Given $d\ge 1$, a continuously differentiable objective $f:{\mathbb{R}}^d\to \mathbb{R}$, a discrete filtered probability space $(\mathrm{\Omega },\mathcal{F},({\mathcal{F}}_t{)}_{t\ge 0},P)$, an initial iterate $x_1$ measurable with respect to ${\mathcal{F}}_1$, an integer horizon $N\ge 1$, deterministic finite stepsizes ${\alpha }_1,\dots ,{\alpha }_N>0$, and a stochastic-gradient oracle whose output at $x_t$ is a random vector $G_t:\mathrm{\Omega }\to {\mathbb{R}}^d$ measurable with respect to ${\mathcal{F}}_{t+1}$, the procedure returns the trajectory $(x_t{)}_{t=1}^{N+1}$. For each iteration $t=1,\dots ,N$, query the oracle at $x_t$ and set $x_{t+1}=x_t-{\alpha }_t{G}_t\mathrm{.}$ No conditional mean, variance, or independence-across-iterations property is part of this procedure's definition.

Stochastic-gradient oracle and unconstrained-update definition: fixes the coordinatewise oracle, conditional-moment, and update conventions.

Filtered probability-space definition: fixes the increasing sigma-algebra sequence used to describe information at each iteration.

Adapted-process definition: fixes time-indexed measurability and integrability conventions.

Random-element definition: a vector-valued oracle output is a measurable map into its stated measurable space.

Population and empirical-risk definition: distinguishes the fixed finite-sample objective from population risk.

Supervised prediction-problem definition: fixes the sample, predictor, and loss context for the learning interpretation.

The oracle interface does not specify which risk is being optimized. For empirical risk, the objective is the fixed sample average and sampling is over its finite terms. For population risk, sampling is from the population law; identifying the expected sampled gradient with the population-risk gradient requires a separately justified differentiation-under-expectation step. This algorithm asserts neither mean identity.

Population and empirical-risk definition: gives the distinct finite-sample and population objectives.

Supervised prediction-problem definition: specifies the joint law and loss whose population risk is integrated.