Integral probability metrics and Wasserstein GANs — Examples

For $t>0$, the second row sum $0$ and nonnegative entries force ${\pi }_{21}={\pi }_{22}=0$; the first column sum $0$ then forces ${\pi }_{11}=0$, and the first row sum $1$ forces ${\pi }_{12}=1$, which also gives the required second column sum $1$. This is the unique feasible matrix, and [F3] gives finite-space cost $d(0,t)=t$. For $t=0$, the singleton probability vector has unique coupling $(1)$ and cost $0$.

By [F1], $P$ and $Q_t$ are Borel probability measures. If $\gamma$ is any coupling of $P$ and $Q_t$, then $\gamma (\{0\}\times X)=1$ and $\gamma (X\times \{t\})=1$. The complement of their intersection is contained in the union of their two null complements, so finite subadditivity from [F6] gives $\gamma (\{(0,t)\})=1$. A probability measure concentrated on the singleton $(0,t)$ equals its Dirac measure there: sets containing $(0,t)$ have mass $1$, and sets omitting it have mass $0$. The Dirac measure at $(0,t)$ has the required marginals, so it is the unique coupling. By [F2] and [F4], its cost is $d(0,t)=t$, so $W_1(P,Q_t)=t$, including $t=0$.

Every function in the specified $1$-Lipschitz class satisfies $\mathrm{\mid }f(0)-f(t)\mathrm{\mid }\le t$, and $f(x)=x$ belongs to the class and has $\mathrm{\mid }f(0)-f(t)\mathrm{\mid }=t$. By [F5], the IPM is exactly $t$.

If $t>0$, the mixture $(P+Q_t)\mathrm{/}2$ assigns mass $1\mathrm{/}2$ to each of $0$ and $t$. The finite sum definition gives $\mathrm{KL}(P\mathrm{\parallel }(P+Q_t)\mathrm{/}2)=\log 2$ and $\mathrm{KL}(Q_t\mathrm{\parallel }(P+Q_t)\mathrm{/}2)=\log 2$, so $\mathrm{JS}(P,Q_t)=\log 2$. If $t=0$, the two laws agree, each KL term is $0$, and the JS value is $0$.

Under the weight bounds, the slope of $f_{a,b}(x)=abx$ has absolute value at most $c^2$. If $c>1$, the allowed choice $a=b=c$ gives slope $c^2>1$. If $0<c<1$, every allowed slope has magnitude less than $1$, so the function $x$ cannot occur. If $c=1$, every member is linear, while $x^2\mathrm{/}2$ is $1$-Lipschitz because $\mathrm{\mid }{x}^2-y^2\mathrm{\mid }\mathrm{/}2=\mathrm{\mid }x-y\mathrm{\mid }(x+y)\mathrm{/}2\le \mathrm{\mid }x-y\mathrm{\mid }$ on $[0,1]$; it is not linear, since its value at $1$ is $1\mathrm{/}2$ and at $1\mathrm{/}2$ is $1\mathrm{/}8$. These three exhaustive cases prove the stated failure of exactness for every $c>0$.

Set $a_n=4^{-n}$, $b_n={64}^{-n}$, $p_n=2^{-n}$, $t_n(\theta )=\max \{1-\mathrm{\mid }\theta -a_n\mathrm{\mid }\mathrm{/}{b}_n,0\}$ and $u_n(z)=\max \{1-4\mathrm{\mid }z-n\mathrm{\mid },0\}$. Put $g_{\theta }(z)={\sum }_{n\ge 1}{u}_n(z)t_n(\theta )$ and $\rho (A)={\sum }_{n\ge 1}{p}_n{\mathbf{1}}_A(n)$ for Borel $A$. The nonnegative-series formula is countably additive: for disjoint sets, equality holds first for finite partial sums and then by taking suprema of finite sums of nonnegative terms. By [F1] its total mass is one. Each $g_{\theta }$ is Borel because the sum is pointwise a limit of continuous partial sums.

On every bounded latent neighborhood only finitely many $u_n$ are nonzero, and the supports $[n-1\mathrm{/}4,n+1\mathrm{/}4]$ are disjoint. Each factor is bounded by one, $u_n$ is $4$-Lipschitz and $t_n$ is $b_n^{-1}$-Lipschitz. Thus each product is jointly Lipschitz with respect to $\mathrm{\mid }\theta -{\theta }^{\mathrm{\prime }}\mathrm{\mid }+\mathrm{\mid }z-z^{\mathrm{\prime }}\mathrm{\mid }$, and the locally finite sum is jointly locally Lipschitz and takes values in $[0,1]$.

Write $I_n=[a_n-b_n,a_n+b_n]$. These intervals are pairwise disjoint: $a_n-a_{n+1}=3a_n\mathrm{/}4$, while $b_n+b_{n+1}=(65\mathrm{/}64)b_n\le (65\mathrm{/}1024)a_n<3a_n\mathrm{/}4$. The measurable function $L(\theta ,z)={\sum }_{n\ge 1}(4+b_n^{-1}){\mathbf{1}}_{I_n}(\theta ){\mathbf{1}}_{(n-1\mathrm{/}2,n+1\mathrm{/}2)}(z)$ supplies a local joint Lipschitz constant at every $(\theta ,z)$. If $\theta \mathrm{\notin }{I}_n$, a small parameter neighborhood misses $I_n$; if $z$ is outside all active latent supports, a small latent neighborhood makes the function zero. At support boundaries the displayed positive constant bounds the piecewise-linear product. At a fixed $\theta$ at most one $I_n$ is active, so $\int L(\theta ,z)d\rho (z)$ is either zero or the finite number $p_n(4+b_n^{-1})$. In particular it is zero at $\theta =0$.

Let $\iota (y)=(0,y)$ and let ${\gamma }_{\theta }={\iota }_{\mathrm{\#}}{P}_{\theta }$. The map $\iota$ is continuous in the product topology, so [F6] makes ${\gamma }_{\theta }$ a Borel probability; for every Borel $A\subseteq X$ its marginals satisfy ${\gamma }_{\theta }(A\times X)={\mathbf{1}}_{\{0\in A\}}={\delta }_0(A)$ and ${\gamma }_{\theta }(X\times A)=P_{\theta }(A)$, so it is a coupling. If $\gamma$ is any coupling and $C\subseteq X\times X$ is Borel, set $B_C={\iota }^{-1}(C)$, which is Borel by continuity. The first marginal gives $\gamma ((X\setminus \{0\})\times X)=0$, so $\gamma (C)=\gamma (C\cap (\{0\}\times X))=\gamma (\{0\}\times B_C)$. It also gives $\gamma ((X\setminus \{0\})\times B_C)=0$, and the second marginal gives $\gamma (X\times B_C)=P_{\theta }(B_C)$; hence $\gamma (\{0\}\times B_C)=P_{\theta }(B_C)={\gamma }_{\theta }(C)$. Thus this coupling is unique, and [F6] gives its cost as ${\int }_{X}yd{P}_{\theta }(y)=\int g_{\theta }d\rho$. For any nonnegative simple $s={\sum }_j{c}_j{\mathbf{1}}_{A_j}$, the definition of $\rho$ and the simple-integral formula give $\int sd\rho ={\sum }_n{p}_{n}s(n)$, since the finite sum over $j$ commutes with the nonnegative series. Choose simple $s_k\uparrow g_{\theta }$; [F2] and [F7] give $\int g_{\theta }d\rho ={\lim }_k{\sum }_n{p}_n{s}_k(n)={\sum }_n{p}_n{g}_{\theta }(n)$, where the last equality follows by taking suprema over finite partial sums and over $k$. Since $g_{\theta }(n)=t_n(\theta )$ and the parameter intervals in step 3.1 are disjoint, at most one term is nonzero. Therefore $V(\theta )={\sum }_{n\ge 1}{p}_n{t}_n(\theta )$. Consequently $V(0)=0$ and $V(a_n)=p_n$, so $V(a_n)\mathrm{/}{a}_n=2^n\to \mathrm{\infty }$. No finite local Lipschitz constant at zero is possible.

A common-neighborhood generator envelope at zero would provide a measurable full-measure set $E$ and an integrable $K$ such that the envelope inequality holds for every $z\in E$ and all parameters in one neighborhood $U$. Since $\rho (\{n\})=p_n>0$ and $\rho (E^c)=0$, every integer $n$ lies in $E$. For all sufficiently large $n$, $a_n\in U$, and comparison of parameters $0,a_n$ at $z=n$ gives $K(n)\ge \mathrm{\mid }{g}_{a_n}(n)-g_0(n)\mathrm{\mid }\mathrm{/}{a}_n=1\mathrm{/}{a}_n=4^n$. By [F7], $\int Kd\rho ={\sum }_n{p}_{n}K(n)\ge {\sum }_{n\ge N}{2}^n=+\mathrm{\infty }$, contradicting integrability. Thus the missing common-neighborhood premise is real. This witness is smooth off the countable set of triangular corners and zero, so it makes no claim to refute almost-everywhere differentiability.

Set $a_n=4^{-n}$ for $n\ge 0$ and $p_n=2^{-n}$ for $n\ge 1$. Define a nonnegative piecewise-linear hat ${\alpha }_n$ on $[0,+\mathrm{\infty })$: it is zero outside $[a_{n+1},a_{n-1}]$, rises linearly from $0$ at $a_{n+1}$ to $1$ at $a_n$, then falls linearly to $0$ at $a_{n-1}$. On $[a_{n+1},a_n]$ only ${\alpha }_n,{\alpha }_{n+1}$ are nonzero and their sum is one. Thus ${\sum }_{n\ge 1}{\alpha }_n(r)=1$ for $0<r\le 1\mathrm{/}4$. At each fixed positive $r$ only finitely many hat supports meet it.

Put $B_n=[n-1\mathrm{/}8,n+1\mathrm{/}8]$ and $q(z)={\sum }_{n\ge 1}4p_n{\mathbf{1}}_{B_n}(z)$. Define $\rho$ on Borel subsets of $\mathbb{R}$ by $\rho (A)={\int }_{A}qd\lambda$, the restriction to Borel sets of the measure with density $q$. The density is Borel, and [F1]–[F4] give $\rho (\mathbb{R})={\sum }_{n}4p_n\lambda (B_n)={\sum }_n{p}_n=1$. Let $u_n(z)$ be one when $\mathrm{\mid }z-n\mathrm{\mid }\le 1\mathrm{/}8$, decrease linearly to zero when $1\mathrm{/}8<\mathrm{\mid }z-n\mathrm{\mid }<1\mathrm{/}4$, and be zero for $\mathrm{\mid }z-n\mathrm{\mid }\ge 1\mathrm{/}4$. It is $8$-Lipschitz; its supports are disjoint, and $u_n=1$ on $B_n$.

Set $b_n(\theta )=\theta {\alpha }_n(\mathrm{\mid }\theta \mathrm{\mid })\mathrm{/}{p}_n$ and $g_{\theta }(z)=1\mathrm{/}2+(1\mathrm{/}16){\sum }_n{u}_n(z)b_n(\theta )$. Since $\mathrm{\mid }{b}_n\mathrm{\mid }\le a_{n-1}\mathrm{/}{p}_n=2^{2-n}\le 2$ and at most one latent support is active, $3\mathrm{/}8\le g\le 5\mathrm{/}8$. Each $b_n$ is continuous piecewise quadratic with compact support and bounded one-sided slopes, hence globally Lipschitz. More explicitly $\mathrm{\mid }{\alpha }_n^{\mathrm{\prime }}\mathrm{\mid }\le 4\mathrm{/}(3a_n)$ and $\mathrm{\mid }\theta \mathrm{\mid }\le 4a_n$ on its support, so $\mathrm{\mid }{b}_n^{\mathrm{\prime }}\mathrm{\mid }\le (1+16\mathrm{/}3)\mathrm{/}{p}_n=19\mathrm{/}(3p_n)$ wherever the derivative exists; subdivision at finitely many corners gives this bound for all differences. Local finiteness in $z$ then proves joint local Lipschitzness of $g$ and Borel measurability of every section.

A valid measurable local joint constant is $L(\theta ,z)={\sum }_n{M}_n{\mathbf{1}}_{[a_{n+1},a_{n-1}]}(\mathrm{\mid }\theta \mathrm{\mid }){\mathbf{1}}_{(n-1\mathrm{/}2,n+1\mathrm{/}2)}(z)$, with $M_n=(16+19\mathrm{/}(3p_n))\mathrm{/}16$: the two product terms are bounded using $\mathrm{\mid }{b}_n\mathrm{\mid }\le 2$, the $8$-Lipschitz $u_n$, and the bound in step 2.1. At any fixed nonzero $\theta$ only finitely many parameter supports are active, so $\int L(\theta ,z)d\rho <\mathrm{\infty }$. At $\theta =0$ every $b_n$ vanishes on a neighborhood of zero, so for each fixed $z$ the locally finite sum is constant in a neighborhood of $(0,z)$; take $L(0,z)=0$. At latent or parameter support boundaries the displayed positive constants apply whenever needed, and outside the closed active supports a neighborhood has zero variation. Thus the original pointwise-local condition holds at every parameter.

Let $c_n(\theta )=1\mathrm{/}2+b_n(\theta )\mathrm{/}16$. The density $q$ vanishes off the disjoint union of the intervals $B_n$, and on each $B_n$ the function $g_{\theta }$ is constant with value $c_n(\theta )$. For $N\ge 1$, $q_N={\sum }_{n\le N}4p_n{\mathbf{1}}_{B_n}$ increases pointwise to $q$, and $g_{\theta }{q}_N$ is simple with integral ${\sum }_{n\le N}4p_n{c}_n(\theta )\lambda (B_n)={\sum }_{n\le N}{p}_n{c}_n(\theta )$ by [F4, F10]. By [F2], $\int g_{\theta }qd\lambda ={\sum }_{n\ge 1}{p}_n{c}_n(\theta )$; by [F9], this equals $\int g_{\theta }d\rho$. For each fixed $\theta$ only finitely many $b_n(\theta )$ are nonzero, while [F1] gives ${\sum }_n{p}_n=1$, so $\int g_{\theta }d\rho =1\mathrm{/}2+(1\mathrm{/}16){\sum }_n{p}_n{b}_n(\theta )=1\mathrm{/}2+(\theta \mathrm{/}16){\sum }_n{\alpha }_n(\mathrm{\mid }\theta \mathrm{\mid })$. Therefore this integral is $1\mathrm{/}2+\theta \mathrm{/}16$ for $0<\mathrm{\mid }\theta \mathrm{\mid }\le 1\mathrm{/}4$ and is $1\mathrm{/}2$ at $\theta =0$.

Put $\iota (y)=(0,y)$ and let ${\gamma }_{\theta }={\iota }_{\mathrm{\#}}{P}_{\theta }$. The map $\iota$ is continuous in the product topology, so [F8] makes ${\gamma }_{\theta }$ a Borel probability; for every Borel $A\subseteq X$ its marginals satisfy ${\gamma }_{\theta }(A\times X)={\mathbf{1}}_{\{0\in A\}}={\delta }_0(A)$ and ${\gamma }_{\theta }(X\times A)=P_{\theta }(A)$, so it is a coupling. Any coupling $\gamma$ is concentrated on $\{0\}\times X$ by its first marginal. For Borel $C\subseteq X\times X$, let $B_C={\iota }^{-1}(C)$, which is Borel by continuity; concentration gives $\gamma (C)=\gamma (C\cap (\{0\}\times X))=\gamma (\{0\}\times B_C)$. The first marginal also gives $\gamma ((X\setminus \{0\})\times B_C)=0$, so the second marginal gives $\gamma (\{0\}\times B_C)=\gamma (X\times B_C)=P_{\theta }(B_C)$. Hence $\gamma (C)={\gamma }_{\theta }(C)$ and this coupling is unique. Applying [F8] to $\iota$ and to $g_{\theta }$ gives its cost ${\int }_{X}yd{P}_{\theta }(y)=\int g_{\theta }d\rho$, so $V(\theta )=W_1^X({\delta }_0,P_{\theta })=\int g_{\theta }d\rho$.

For any normalized $1$-Lipschitz $h:[0,1]\to \mathbb{R}$, [F8] applied to the constant-zero map and to $g_{\theta }$ gives $\int hd{\delta }_0=h(0)=0$ and $\int hd{P}_{\theta }=\int h(g_{\theta }(z))d\rho (z)$. The function $f(x)=-x$ is normalized and $1$-Lipschitz, and its signed dual objective is $\int fd{\delta }_0-\int fd{P}_{\theta }=\int g_{\theta }d\rho =V(\theta )$ by step 4.1. The compact KR duality [F7] therefore makes $f$ an attaining critic, in particular at $\theta =0$. By step 3.1, each fixed latent section is constant near zero, so ${\mathrm{\partial }}_{\theta }f(g_{\theta }(z)){\mathrm{\mid }}_0=0$ for every $z$; this derivative is measurable and integrable with expectation zero. Step 3.2 gives $V^{\mathrm{\prime }}(0)=1\mathrm{/}16$, so the two defined terms in the proposed identity are unequal.

Suppose there were a common-neighborhood generator envelope at zero: an open neighborhood $U$ of zero, a nonnegative measurable $K$ with $\int Kd\rho <\mathrm{\infty }$, and a measurable set $E$ with $\rho (E)=1$ such that $\mathrm{\mid }{g}_{\theta }(z)-g_{{\theta }^{\mathrm{\prime }}}(z)\mathrm{\mid }\le K(z)\mathrm{\mid }\theta -{\theta }^{\mathrm{\prime }}\mathrm{\mid }$ for every $z\in E$ and every $\theta ,{\theta }^{\mathrm{\prime }}\in U$. Since $a_n\to 0$, there is $n_0$ such that $a_n\in U$ for every $n\ge n_0$. For every such $n$ and $z\in B_n\cap E$, the envelope then gives $K(z)\ge \mathrm{\mid }{g}_{a_n}(z)-g_0(z)\mathrm{\mid }\mathrm{/}{a}_n=1\mathrm{/}(16p_n)$, because $u_n=1$ on $B_n$ and ${\alpha }_n(a_n)=1$. By [F6], $\rho (B_n\setminus E)\le \rho (E^c)=0$, so $\rho (B_n\cap E)=\rho (B_n)=p_n$ by step 1.2. The sets $B_n\cap E$ are disjoint and each contributes at least $1\mathrm{/}16$ to $\int Kd\rho$. Summing these nonnegative contributions, using [F2] to pass from finite partial sums to their limit, forces $\int Kd\rho =\mathrm{\infty }$, a contradiction. The counterexample therefore violates precisely the stronger corrected envelope premise while satisfying the original one.