Automatic differentiation and computational graphs — Examples

For $x>0$, the active formula is $f(x)=x$, whose derivative is $1$; for $x<0$, the active formula is $f(x)=-x$, whose derivative is $-1$. These are the two derivatives used by the strict-branch rule in [F1].

At zero, the right difference quotient is $(\mathrm{\mid }h\mathrm{\mid }-0)\mathrm{/}h=1$ for $h>0$, while the left quotient is $(\mathrm{\mid }h\mathrm{\mid }-0)\mathrm{/}h=-1$ for $h<0$. Since the one-sided limits differ, the derivative does not exist at zero.

By [F1], the branch boundary therefore returns NONDIFFERENTIABLE and no derivative; selecting a generalized derivative would require a separately supplied rule.

The system gives $u_2=1$ and $u_1+\theta u_2=2$, hence $u(\theta )=(2-\theta ,1{)}^T$ and $J(\theta )=3-\theta$. Therefore $J^{\mathrm{\prime }}(0)=-1$.

Write $R(u,\theta )=A(\theta )u-b$. At $\theta =0$, $u=(2,1{)}^T$, $R_u=A(0)=I$, $R_{\theta }=A_{\theta }u=(1,0{)}^T$, and $J_u=(1,1)$. The adjoint equation $R_u^T\lambda =J_u^T$ gives $\lambda =(1,1{)}^T$.

The objective has no explicit $\theta$ dependence, so $J_{\theta }=0$; the adjoint formula gives $J_{\theta }-{\lambda }^T{R}_{\theta }=-(1,1)\cdot (1,0)=-1$.

The adjoint result agrees with the direct derivative in step 1.1, verifying this nonsingular linear-solve instance.

In any arithmetic satisfying the operationwise model and representing $0,1,2,3$, with these small operations in range, at $\theta =0$ the primal and basis solves give $\widehat{x}=(2,1{)}^T$ and $B=I$ exactly. Their inverse defect and primal residual are zero. The certified solve-Jacobian result Certified inverse defects and primal residuals bound linear-solve Jacobian and pullback errors therefore gives zero fixed-map discrepancy; the actual dot and product operations here are also exact, and return derivative $-1$.

Substitution into the two Euler updates gives $x_1=1+(1\mathrm{/}2)(1)(1)=3\mathrm{/}2$ and $x_2=(3\mathrm{/}2)+(1\mathrm{/}2)(1)(3\mathrm{/}2)=9\mathrm{/}4$. As a function of $\alpha$, $J(\alpha )=(1+h\alpha {)}^2{x}_0$, so $J^{\mathrm{\prime }}(1)=2h(1+h)x_0=3\mathrm{/}2$.

Seed ${\bar{x}}_2=1$. At step $k=1$, $\mathrm{\partial }{x}_2\mathrm{/}\mathrm{\partial }{x}_1=1+h\alpha =3\mathrm{/}2$ and $\mathrm{\partial }{x}_2\mathrm{/}\mathrm{\partial }\alpha =h{x}_1=3\mathrm{/}4$, so ${\bar{x}}_1=3\mathrm{/}2$ and the accumulated $\bar{\alpha }=3\mathrm{/}4$.

At step $k=0$, $\mathrm{\partial }{x}_1\mathrm{/}\mathrm{\partial }{x}_0=3\mathrm{/}2$ gives ${\bar{x}}_0=(3\mathrm{/}2)(3\mathrm{/}2)=9\mathrm{/}4$, while $\mathrm{\partial }{x}_1\mathrm{/}\mathrm{\partial }\alpha =h{x}_0=1\mathrm{/}2$ adds $(1\mathrm{/}2)(3\mathrm{/}2)=3\mathrm{/}4$ to $\bar{\alpha }$. Thus the total parameter cotangent is $3\mathrm{/}2$.

The reverse result equals the direct derivative in step 1.1, and [F2] identifies it as the derivative of the finite trajectory map.

The primal values are $z_1=1+2=3$, $z_2=3\cdot 3=9$, $z_3=1\cdot 2=2$, and $f=z_2+z_3=11$.

For direction $(u,v)$, forward mode gives ${\dot{z}}_1=u+v$, ${\dot{z}}_2=z_1{\dot{z}}_1+{\dot{z}}_1{z}_1=6(u+v)$, ${\dot{z}}_3=yu+xv=2u+v$, hence $\dot{f}=8u+7v$. By [F2], the gradient is $(8,7)$.

In reverse mode, seed $\bar{f}=1$, so ${\bar{z}}_2={\bar{z}}_3=1$. The product $z_3=xy$ adds $2$ to $\bar{x}$ and $1$ to $\bar{y}$. The repeated slots in $z_2=z_1{z}_1$ add $z_1+z_1=6$ to ${\bar{z}}_1$; then $z_1=x+y$ adds $6$ to each input. The total input cotangents are $(8,7)$, consistent with [F3].

The forward value $11$ and both derivative calculations agree with the stated output and gradient.