Negative-gradient pseudo-residuals for a finite empirical loss

Let $N\ge 1$, let $(x_i,y_i{)}_{i=1}^N$ be a finite sample with $x_i\in \mathcal{X}$ and $y_i\in \mathcal{Y}$, let $f:\mathcal{X}\to \mathbb{R}$ be a score function, and let $\mathrm{\ell }:\mathcal{Y}\times \mathbb{R}\to \mathbb{R}$ be finite-valued. Assume that for each $i$, the one-variable function $t\mapsto \mathrm{\ell }(y_i,t)$ is differentiable at $z_i=f(x_i)$. For a score vector $z=(z_1,\dots ,z_N)$, define the empirical score loss $\mathrm{\Phi }(z)={\sum }_{i=1}^N\mathrm{\ell }(y_i,z_i)$. The pseudo-residual at observation $i$ for $f$ is

$r_i(f)=-{\frac{d}{dt}\mathrm{\ell }(y_i,t)\mid }_{t=f(x_i)}\mathrm{.}$

The sample-score gradient, when its coordinate partial derivatives exist, is the vector $\mathrm{\nabla }\mathrm{\Phi }(z)=({\mathrm{\partial }}_{z_1}\mathrm{\Phi }(z),\dots ,{\mathrm{\partial }}_{z_N}\mathrm{\Phi }(z))$. The sample pseudo-residual vector is $(r_1(f),\dots ,r_N(f))$; the word “pseudo” distinguishes these score derivatives from derivatives with respect to parameters of a restricted model.

Trevor Hastie, Robert Tibshirani, and Jerome Friedman, The Elements of Statistical Learning, 2nd ed., §10.10.3, printed p. 361

Alexey Natekin and Alois Knoll, “Gradient boosting machines, a tutorial,” §2.4