Integral probability metrics and Wasserstein GANs

By [F4] and [F6], every $P_{\theta }$ is a probability measure. By [F5], $P_{\theta }(A)=\rho (G_{\theta }^{-1}(A))$ for every Borel $A\subseteq X$. If $s={\sum }_{j=1}^m{c}_j{\mathbf{1}}_{A_j}$ is nonnegative simple with disjoint Borel sets, [F7]–[F8] give $\int sd{P}_{\theta }={\sum }_j{c}_j{P}_{\theta }(A_j)={\sum }_j{c}_j\rho (G_{\theta }^{-1}(A_j))=\int s\circ G_{\theta }d\rho$. For any bounded Borel real $f$, apply [F9] to $f^{+}$ and $f^{-}$; their increasing simple approximants are bounded by $\mathrm{\parallel }f{\mathrm{\parallel }}_{\mathrm{\infty }}$, whose integral is finite under both probability measures. Applying [F10] to the approximants and their compositions gives equality of the positive-part integrals and of the negative-part integrals; [F11] then yields $\int fd{P}_{\theta }=\int f(G_{\theta }(z))d\rho (z)$.

The zero function belongs to ${\mathcal{F}}_1(X,x_0)$, so this class is nonempty; it is also symmetric under $f\mapsto -f$, so its supremum of absolute integral differences equals its signed supremum. It is exactly the bounded Borel version of the compact KR test class in [F2]. Hence $V(\theta )=W_1^X(\mu ,P_{\theta })$ for every $\theta$, with the coupling-cost convention in [F3].

Fix ${\theta }_0$ and its attaining critic, and choose $r>0$ such that the ball $B({\theta }_0,r)$ lies in the envelope neighborhood. Let $E$ be the intersection of its measurable full-measure set with a measurable full-measure set where the composite is Fréchet differentiable. Extend the measurable derivative by zero off $E$. On $E$ its norm is at most $K_{{\theta }_0}$, by the envelope and the critic's $1$-Lipschitz inequality, so all coordinates are integrable. Write $R(h,z)=H_{{\theta }_0}({\theta }_0+h,z)-H_{{\theta }_0}({\theta }_0,z)-D_{\theta }{H}_{{\theta }_0}({\theta }_0,z)[h]$. For $j\ge 1$ put $s_j=\min \{r\mathrm{/}2,1\mathrm{/}j\}$ and let $Q_j(z)$ be zero off $E$ and otherwise the supremum of $\mathrm{\mid }R(q,z)\mathrm{\mid }\mathrm{/}\mathrm{\parallel }q\mathrm{\parallel }$ over rational $p$-tuples with $0<\mathrm{\parallel }q\mathrm{\parallel }<s_j$, including zero among the values. The explicit enumeration and [F13] give measurability; the envelope gives $0\le Q_j\le 2K_{{\theta }_0}$, and pointwise Fréchet differentiability gives $Q_j\to 0$ on $E$. By [F10], $\int Q_{j}d\rho \to 0$. On $E$ the remainder ratio is continuous in nonzero $h$ by the envelope, so rational density [F12] bounds it by $Q_j$ for every real $0<\mathrm{\parallel }h\mathrm{\parallel }<s_j$. Therefore $\mathrm{\mid }\int R(h,z)d\rho \mathrm{\mid }\mathrm{/}\mathrm{\parallel }h\mathrm{\parallel }\le \int Q_{j}d\rho$ for all such $h$, proving the full Fréchet limit without a sequential-criterion choice step. Thus $\mathrm{\Phi }(\theta )=\int f_{{\theta }_0}d\mu -\int H_{{\theta }_0}(\theta ,z)d\rho (z)$ has derivative $-\int D_{\theta }{H}_{{\theta }_0}({\theta }_0,z)d\rho (z)$.

Fix ${\theta }_0$ and a ball $B({\theta }_0,r)\subseteq \mathrm{\Theta }$. Put $s_j=\min \{r\mathrm{/}2,1\mathrm{/}j\}$. On $Z_0$, let $S_j(z)$ be the supremum of $d(G_{{\theta }_0+q}(z),G_{{\theta }_0}(z))$ over rational $p$-tuples with $\mathrm{\parallel }q\mathrm{\parallel }<s_j$, and set it to zero off $Z_0$. It is measurable by the assumed distance measurability, [F12] and [F13], is bounded by $D$, and tends pointwise to zero by parameter continuity. Thus [F10] gives $\int S_{j}d\rho \to 0$. Parameter continuity and rational density make $S_j$ dominate the same distance for all real increments $\mathrm{\parallel }h\mathrm{\parallel }<s_j$ on $Z_0$. For every critic $f$, step 1.1 and its $1$-Lipschitz property give $\mathrm{\mid }{\mathrm{\Delta }}_{{\theta }_0+h}(f)-{\mathrm{\Delta }}_{{\theta }_0}(f)\mathrm{\mid }\le \int S_{j}d\rho$, where ${\mathrm{\Delta }}_{\theta }(f)=\int fd\mu -\int fd{P}_{\theta }$. The same bound holds for the difference of their absolute values and hence of the two suprema. It follows that $\mathrm{\mid }V({\theta }_0+h)-V({\theta }_0)\mathrm{\mid }\le \int S_{j}d\rho$ for all $\mathrm{\parallel }h\mathrm{\parallel }<s_j$, proving continuity.

Under the common-neighborhood envelope, the same estimate gives $\mathrm{\mid }{\mathrm{\Delta }}_{\theta }(f)-{\mathrm{\Delta }}_{{\theta }^{\mathrm{\prime }}}(f)\mathrm{\mid }\le \mathrm{\parallel }\theta -{\theta }^{\mathrm{\prime }}\mathrm{\parallel }\int K_{{\theta }_0}d\rho$ for every $f\in {\mathcal{F}}_1(X,x_0)$ and $\theta ,{\theta }^{\mathrm{\prime }}\in U$. Taking absolute values and then suprema proves the stated local-Lipschitz bound for $V$.

By optimality, $\mathrm{\Phi }({\theta }_0)=V({\theta }_0)$ and $\mathrm{\Phi }(\theta )\le V(\theta )$ for every $\theta$. Thus $V-\mathrm{\Phi }$ has a local minimum of $0$ at ${\theta }_0$. Along every direction $v$, differentiability and the two one-sided difference quotients at this minimum give $D(V-\mathrm{\Phi })({\theta }_0)[v]=0$; since this holds for every $v$, the derivative of the difference is zero, giving the claimed gradient formula.

By [F1], $V$ is a locally Lipschitz real-valued function on the open set $\mathrm{\Theta }\subseteq {\mathbb{R}}^p$.

Under [A1], apply [F2] to $V$ on $\mathrm{\Theta }$; it is Fréchet differentiable outside a Lebesgue-null subset of $\mathrm{\Theta }$.