Hypergradient of a smooth local stationary branch

Let $p,d\ge 1$, let $\mathrm{\Theta }\subseteq {\mathbb{R}}^p$ and $W\subseteq {\mathbb{R}}^d$ be open, and let $f:\mathrm{\Theta }\times W\to \mathbb{R}$ be $C^2$ and $g:W\times \mathrm{\Theta }\to \mathbb{R}$ be $C^1$. Here $C^1$ means total differentiability with continuous derivative, and $C^2$ means the derivative map is $C^1$. Define $F(\theta ,w)={\mathrm{\nabla }}_{w}f(\theta ,w)$, $H(\theta ,w)=D_{w}F(\theta ,w)$, and $B(\theta ,w)=D_{\theta }F(\theta ,w)$. Suppose $F({\theta }_0,w_0)=0$ and $H({\theta }_0,w_0)$ is invertible. Then there are open neighborhoods $P\subseteq \mathrm{\Theta }$ and $Q\subseteq W$ of ${\theta }_0$ and $w_0$ and a unique $C^1$ local stationary branch $w^{\ast }:P\to Q$ through $({\theta }_0,w_0)$, meaning $w^{\ast }({\theta }_0)=w_0$ and $F(\theta ,w^{\ast }(\theta ))=0$ for every $\theta \in P$. After shrinking $P$ if necessary, $H(\theta ,w^{\ast }(\theta ))$ is invertible and

$D{w}^{\ast }(\theta )=-H(\theta ,w^{\ast }(\theta ){)}^{-1}B(\theta ,w^{\ast }(\theta ))\mathrm{.}$

Write $H^{-T}:=(H^{-1}{)}^T$. For $J(\theta )=g(w^{\ast }(\theta ),\theta )$, where ${\mathrm{\partial }}_{\theta }g$ holds $w$ fixed and gradients are column vectors, the hypergradient is

${\mathrm{\nabla }}_{\theta }J(\theta )={\mathrm{\partial }}_{\theta }g(w^{\ast }(\theta ),\theta )-B(\theta ,w^{\ast }(\theta ){)}^{T}H(\theta ,w^{\ast }(\theta ){)}^{-T}{\mathrm{\nabla }}_{w}g(w^{\ast }(\theta ),\theta )\mathrm{.}$

The local branch need not be a global inner optimizer. In the separate global regime where, for every $\theta \in P$, $f(\theta ,\cdot )$ is defined on all of ${\mathbb{R}}^d$ and, for some $\mu >0$, satisfies $f(\theta ,v)\ge f(\theta ,u)+{\mathrm{\nabla }}_{w}f(\theta ,u{)}^T(v-u)+\frac{\mu }{2}\mathrm{\parallel }v-u{\mathrm{\parallel }}_2^2$ for every $u,v\in {\mathbb{R}}^d$, each stationary point is the unique global minimizer.

Bilevel stationary-problem notation and strong-convexity convention

Local branch existence, uniqueness, and derivative formula

Finite-dimensional chain rule