Convexity, Lipschitz bounds, and oscillation of the source contrastive losses

The positive-part function is nondecreasing and $1$-Lipschitz: if two inputs have the same sign, the positive-part difference is either their difference or zero; if they have opposite signs, the positive part of the nonnegative input is at most the distance between them. It is convex as the maximum of the affine functions $0$ and $1+t$. Therefore ${\mathrm{\ell }}_{\mathrm{h}}$ is nondecreasing, convex, and $1$-Lipschitz, and ${\mathrm{\ell }}_{\mathrm{h}}(0)=1$.

By [F2]–[F3] and the chain rule, ${\mathrm{\ell }}_{\mathrm{l}\mathrm{o}\mathrm{g}}^{\mathrm{\prime }}(t)=\frac{e^t}{(1+e^t)\ln 2},{\mathrm{\ell }}_{\mathrm{l}\mathrm{o}\mathrm{g}}^{\mathrm{\prime }\mathrm{\prime }}(t)=\frac{e^t}{(1+e^t{)}^2\ln 2}\mathrm{.}$

By [F1], $\ln 2>0$, and [F2] gives $e^t>0$. Step 1.2 therefore shows that $0<{\mathrm{\ell }}_{\mathrm{l}\mathrm{o}\mathrm{g}}^{\mathrm{\prime }}(t)\le 1\mathrm{/}\ln 2$ for every real $t$. The derivative is positive, so the logistic loss is strictly increasing; applying [F5] on each closed interval gives the global $1\mathrm{/}\ln 2$ Lipschitz bound.

The second derivative in step 1.2 is nonnegative because its numerator and denominator are positive. The convexity criterion [F4] gives convexity of ${\mathrm{\ell }}_{\mathrm{l}\mathrm{o}\mathrm{g}}$. Also ${\mathrm{\ell }}_{\mathrm{l}\mathrm{o}\mathrm{g}}(0)={\log }_2(2)=1$ from the definition of ${\log }_2$.

Since $L_0$ exceeds both Lipschitz constants, steps 1.1 and 2.1 give the common $L_0$-Lipschitz bound. For the hinge loss, if $0<R<1\mathrm{/}\sqrt{2}$, then $B_{{\mathrm{\ell }}_{\mathrm{h}},R}=(1+2R^2)-(1-2R^2)=4R^2>0$; if $R\ge 1\mathrm{/}\sqrt{2}$, then $B_{{\mathrm{\ell }}_{\mathrm{h}},R}=1+2R^2>0$. The logistic loss is strictly increasing by step 2.1, so $B_{{\mathrm{\ell }}_{\mathrm{l}\mathrm{o}\mathrm{g}},R}>0$ as well. In either case, the interval has length $4R^2$, so the Lipschitz bound gives $B_{\mathrm{\ell },R}\le 4L_0{R}^2$.