Minibatch stochastic gradient descent

Given $d\ge 1$, a continuously differentiable objective $f:{\mathbb{R}}^d\to \mathbb{R}$, a discrete filtered probability space $(\mathrm{\Omega },\mathcal{F},({\mathcal{F}}_t{)}_{t\ge 0},P)$, an initial iterate $x_1$ measurable with respect to ${\mathcal{F}}_1$, an integer horizon $N\ge 1$, deterministic finite stepsizes ${\alpha }_1,\dots ,{\alpha }_N>0$, an integer batch size $b\ge 1$, and a batched oracle whose outputs at $x_t$ are random vectors $G_{t,1},\dots ,G_{t,b}:\mathrm{\Omega }\to {\mathbb{R}}^d$, each measurable with respect to ${\mathcal{F}}_{t+1}$. For each iteration $t=1,\dots ,N$, query the batched oracle at the current iterate $x_t$, receive the $b$ vectors, and form ${\bar{G}}_t=\frac{1}{b}{\sum }_{j=1}^b{G}_{t,j}$ and set $x_{t+1}=x_t-{\alpha }_t{\bar{G}}_t$. Return the trajectory $(x_t{)}_{t=1}^{N+1}$. The batch vectors are evaluated at the same current iterate; no independence, unbiasedness, or variance-reduction claim is part of this procedure's definition.

Single-direction stochastic-gradient-descent procedure: the averaged direction is supplied to its unconstrained update.

Stochastic-gradient oracle and update definition: fixes the vector-update convention.

Filtered probability-space definition: fixes the information sequence for the batch outputs.

Adapted-process definition: fixes time-indexed measurability.

Random-element definition: each vector output is a measurable map into ${\mathbb{R}}^d$.

Euclidean inner-product and norm definition: fixes the finite-dimensional vector operations and norm convention.

The procedure specifies how to average the returned vectors. The chosen batch law must separately justify any claim about the average's mean or variance; no such claim follows from the batch size alone.