Inverse Gibbs models, identifiability and correlated uncertainty

For every nonempty compact $C\subseteq {\mathbb{R}}^k$ and every $x\in X$, [F3, F7] give $\mathrm{\mid }\eta \cdot T(x)-{\eta }^{\mathrm{\prime }}\cdot T(x)\mathrm{\mid }\le \mathrm{\parallel }\eta -{\eta }^{\mathrm{\prime }}{\mathrm{\parallel }}_2\mathrm{\parallel }T(x){\mathrm{\parallel }}_2$; hence $\eta \mapsto \eta \cdot T(x)$ is continuous. It attains a finite maximum $M_C(x)$ on $C$ by [F4]. Define $G_C(x):=w(x)\exp (M_C(x))(1+{\sum }_{j=1}^k\mathrm{\mid }{T}_j(x){\mathrm{\mid }}^2)\mathrm{.}$ The exponential is finite and positive at this real input by [F5, F8], and the sum is finite, so $G_C$ is finite and nonnegative at every $x$. It is integrable under counting measure by [F2]. Since $\eta \cdot T(x)\le M_C(x)$ for every $\eta \in C$, monotonicity of $\exp$ [F9] shows that $G_C$ dominates $w(x)e^{\theta \cdot T(x)}(1+{\sum }_j\mathrm{\mid }{T}_j(x){\mathrm{\mid }}^2)$ for every $\theta \in C$. Thus the counting-measure model satisfies the envelope hypotheses of [F1]. Applying [F1] gives $0<Z(\theta )<\mathrm{\infty }$, $A\in C^2({\mathbb{R}}^k)$, and the normalized probability mass function ${\pi }_{\theta }$.

Since every summand in $Z(\theta )$ is positive and finite [F5, F8], each ${\pi }_{\theta }(x)>0$. Also $\exp (A(\theta ))=Z(\theta )$ by [F6], so $e^{-A(\theta )}=1\mathrm{/}Z(\theta )$ by [F8]. By [F10], ${\pi }_{\theta }(x)=w(x)e^{\theta \cdot T(x)}{e}^{-A(\theta )}$, which is exactly the normalized density supplied by [F1]. Summing over finite $X$ gives ${\sum }_{x\in X}{\pi }_{\theta }(x)=e^{-A(\theta )}Z(\theta )=1$. Hence the Definition gives a probability vector with full support for every finite $\theta$.

The envelope verified in [F1] lets [F2] apply with counting measure, so $\mathrm{\nabla }A(\theta )={\mathbb{E}}_{{\pi }_{\theta }}T$ and ${\mathrm{\nabla }}^{2}A(\theta )={\mathrm{Cov}}_{{\pi }_{\theta }}(T)$.

For any $a\in {\mathbb{R}}^k$, the quadratic form is $a^{\mathsf{T}}{\mathrm{\nabla }}^{2}A(\theta )a={\sum }_{x\in X}{\pi }_{\theta }(x)(a\cdot T(x)-{\mathbb{E}}_{{\pi }_{\theta }}[a\cdot T]{)}^2$. Every weight is positive by [F1], so this sum is zero exactly when $a\cdot T(x)$ has the same value at every $x\in X$. Thus the Hessian is positive definite for every $\theta$ exactly when no nonzero $a$ makes $a\cdot T$ constant, which is minimality by [F1].

If ${\pi }_{\theta }={\pi }_{\phi }$ as functions on $X$, positivity allows taking logarithms of their equal positive values. The product law and inverse identity [F5, F6] then give $(\theta -\phi )\cdot T(x)=A(\theta )-A(\phi )$ for every $x$. Under minimality this forces $\theta -\phi =0$. Conversely, if a nonzero $a$ makes $a\cdot T(x)=c$ constant, the exponential addition formula [F7] gives $Z(\theta +ta)=\exp (tc)Z(\theta )$. Since $Z(\theta )>0$ and $\exp (tc)>0$ by [F1, F8], the logarithm laws [F5, F6] give $A(\theta +ta)=A(\theta )+tc$. Substitution in the family formula and another use of [F7] show ${\pi }_{\theta +ta}={\pi }_{\theta }$ for every real $t$; the parametrization is not injective.

Let $D=\mathrm{span}\{T(x)-T(x_0):x\in X\}$ for a fixed $x_0\in X$. If $D$ is proper, choose a basis $v_1,\dots ,v_r$ of $D$, with $r<k$, and define $L:{\mathbb{R}}^k\to {\mathbb{R}}^r$ by $L(a)=(a\cdot v_1,\dots ,a\cdot v_r)$. Rank-nullity [F3] gives $\dim \ker L\ge k-r>0$, so some nonzero $a$ is orthogonal to every difference and $a\cdot T$ is constant. Conversely, if such a nonzero $a$ exists, every difference lies in the proper hyperplane $a^{\perp }$, so $D$ is proper. By [F4], $D={\mathbb{R}}^k$ exactly when the affine hull is ${\mathbb{R}}^k$. This proves the stated equivalence with full affine dimension and completes the identifiability conditions.

Enumerate $X$ as $x_1,\dots ,x_m$ and, for $\lambda \in {\mathbb{R}}^m$, write ${\lambda }_j:=\lambda (j-1)$ for $1\le j\le m$. Let $\mathrm{\Delta }=\{\lambda \in [0,1{]}^m:{\sum }_{j=1}^m{\lambda }_j=1\}$. The set $\mathrm{\Delta }$ is nonempty (take ${\lambda }_1=1$ and all other coordinates zero) and bounded. Its coordinate maps and sum map are continuous, since each coordinate inequality $\mathrm{\mid }{\lambda }_j-{\lambda }_j^{\mathrm{\prime }}\mathrm{\mid }\le \mathrm{\parallel }\lambda -{\lambda }^{\mathrm{\prime }}{\mathrm{\parallel }}_2$ follows from the norm formula [F9], while $\mathrm{\mid }{\sum }_j({\lambda }_j-{\lambda }_j^{\mathrm{\prime }})\mathrm{\mid }\le m\mathrm{\parallel }\lambda -{\lambda }^{\mathrm{\prime }}{\mathrm{\parallel }}_2$ follows from the real triangle inequality; hence $\mathrm{\Delta }$, an intersection of their closed inverse images, is closed. By [F1], $\mathrm{\Delta }$ is compact. For $\lambda ,{\lambda }^{\mathrm{\prime }}\in \mathrm{\Delta }$, the triangle inequality and coordinate bound give ${\parallel {\sum }_j{\lambda }_{j}T(x_j)-{\sum }_j{\lambda }_j^{\mathrm{\prime }}T(x_j)\parallel }_2\le {\sum }_j\mathrm{\mid }{\lambda }_j-{\lambda }_j^{\mathrm{\prime }}\mathrm{\mid }\mathrm{\parallel }T(x_j){\mathrm{\parallel }}_2\le \mathrm{\parallel }\lambda -{\lambda }^{\mathrm{\prime }}{\mathrm{\parallel }}_2{\sum }_j\mathrm{\parallel }T(x_j){\mathrm{\parallel }}_2,$ so the map $\lambda \mapsto {\sum }_j{\lambda }_{j}T(x_j)$ is continuous. Thus [F2] makes its image $K$ compact; [F3] makes $K$ closed.

For each natural $r\le k$, define $P_r:=\{f:r\to \mathbb{Q}\}$. The set $P_0=\{\mathrm{\varnothing }\}$ is finite. Restriction and the last coordinate give a bijection $P_{r+1}\to P_r\times \mathbb{Q}$, namely $f\mapsto (f{\mathrm{\mid }}_r,f(r))$; its inverse extends $g:r\to \mathbb{Q}$ by the specified last value. By induction on $r$, the binary-product result [F6] makes every $P_r$ at most countable. Since $P_k$ contains the zero function, [F6] supplies one surjection $q:\mathbb{N}\to P_k={\mathbb{Q}}^k$. Embed its values coordinatewise in ${\mathbb{R}}^k$ using [F7] and write them as $q_j$. These vectors are dense: for $p\in {\mathbb{R}}^k$ and $\epsilon >0$, each interval $(p(i)-\epsilon \mathrm{/}(2k),p(i)+\epsilon \mathrm{/}(2k))$, $i<k$, contains a rational by [F7]. Select these finitely many coordinates by induction on $i<k$, obtaining $a\in P_k$. Then $\mathrm{\parallel }a-p{\mathrm{\parallel }}_2^2<k(\epsilon \mathrm{/}(2k){)}^2<{\epsilon }^2$ by [F9], so some $q_j=a$ lies in $B(p,\epsilon )$. Only finite induction and a single enumeration are used; no countable family of selections is made.

Fix an integer $n\ge 1$. Noninteriority of $\mu$ gives a point $p\mathrm{\notin }K$ with $\mathrm{\parallel }p-\mu {\mathrm{\parallel }}_2<1\mathrm{/}(2n)$. Since $K$ is closed, its complement is open, so some ball $B(p,\rho )$ of radius $\rho >0$ lies outside $K$. Set $r=\frac{1}{2}\min \{\rho ,1\mathrm{/}n-\mathrm{\parallel }p-\mu {\mathrm{\parallel }}_2\}>0$. Step 1.2 gives a rational vector $q_j\in B(p,r)$; then $q_j\mathrm{\notin }K$ and $\mathrm{\parallel }{q}_j-\mu {\mathrm{\parallel }}_2<1\mathrm{/}n$. Thus $J_n=\{j\in \mathbb{N}:q_j\mathrm{\notin }K,\mathrm{\parallel }{q}_j-\mu {\mathrm{\parallel }}_2<1\mathrm{/}n\}$ is nonempty. This argument proves nonemptiness separately for each $n$ without choosing a sequence of $p,\rho ,r$. Let $j_n$ be the least member of $J_n$ and set $y_n=q_{j_n}$. This least-index construction uses no choice and $y_n\to \mu$.

For $z,z^{\mathrm{\prime }}\in K$, the triangle inequality gives $\mathrm{\parallel }{y}_n-z{\mathrm{\parallel }}_2\le \mathrm{\parallel }{y}_n-z^{\mathrm{\prime }}{\mathrm{\parallel }}_2+\mathrm{\parallel }z-z^{\mathrm{\prime }}{\mathrm{\parallel }}_2$, and exchanging $z,z^{\mathrm{\prime }}$ gives $\mid \mathrm{\parallel }{y}_n-z{\mathrm{\parallel }}_2-\mathrm{\parallel }{y}_n-z^{\mathrm{\prime }}{\mathrm{\parallel }}_2\mid \le \mathrm{\parallel }z-z^{\mathrm{\prime }}{\mathrm{\parallel }}_2$. Hence $z\mapsto \mathrm{\parallel }{y}_n-z{\mathrm{\parallel }}_2$ is continuous on $K$ in its Euclidean metric. The nonempty compact set $K$ therefore has a minimizer $z_n\in K$ by [F5]. It is unique: if distinct $z,z^{\mathrm{\prime }}\in K$ both minimize, convexity puts $(z+z^{\mathrm{\prime }})\mathrm{/}2$ in $K$ and $\mathrm{\parallel }{y}_n-(z+z^{\mathrm{\prime }})\mathrm{/}2{\mathrm{\parallel }}_2^2=(\mathrm{\parallel }{y}_n-z{\mathrm{\parallel }}_2^2+\mathrm{\parallel }{y}_n-z^{\mathrm{\prime }}{\mathrm{\parallel }}_2^2)\mathrm{/}2-\mathrm{\parallel }z-z^{\mathrm{\prime }}{\mathrm{\parallel }}_2^2\mathrm{/}4$ is strictly smaller. For any $v\in K$ and $0<s\le 1$, convexity gives $z_n+s(v-z_n)\in K$. Minimality then implies $\mathrm{\parallel }{y}_n-z_n{\mathrm{\parallel }}_2^2\le \mathrm{\parallel }{y}_n-z_n-s(v-z_n){\mathrm{\parallel }}_2^2$; expanding the inner products, dividing by $s$, and letting $s\downarrow 0$ gives $(y_n-z_n)\cdot (v-z_n)\le 0$.

Since $\mu \in K$, minimality gives $\mathrm{\parallel }{y}_n-z_n{\mathrm{\parallel }}_2\le \mathrm{\parallel }{y}_n-\mu {\mathrm{\parallel }}_2$, so the triangle inequality gives $\mathrm{\parallel }{z}_n-\mu {\mathrm{\parallel }}_2\le 2\mathrm{\parallel }{y}_n-\mu {\mathrm{\parallel }}_2\to 0$. Put $u_n=(y_n-z_n)\mathrm{/}\mathrm{\parallel }{y}_n-z_n{\mathrm{\parallel }}_2$; the denominator is positive because $y_n\mathrm{\notin }K$, and homogeneity of the norm gives $\mathrm{\parallel }{u}_n{\mathrm{\parallel }}_2=1$. Thus $(u_n)$ is bounded in the Euclidean metric; [F8] gives a subsequence $u_{n_j}$ converging to $u$. The reverse triangle inequality, obtained by applying the triangle inequality in both orders, gives $\mid \mathrm{\parallel }{u}_{n_j}{\mathrm{\parallel }}_2-\mathrm{\parallel }u{\mathrm{\parallel }}_2\mid \le \mathrm{\parallel }{u}_{n_j}-u{\mathrm{\parallel }}_2$, hence $\mathrm{\parallel }u{\mathrm{\parallel }}_2=1$. For every $v\in K$, step 3.1 gives $u_{n_j}\cdot (v-z_{n_j})\le 0$. By [F10], $\mathrm{\mid }{u}_{n_j}\cdot (v-z_{n_j})-u\cdot (v-\mu )\mathrm{\mid }\le \mathrm{\parallel }{u}_{n_j}-u{\mathrm{\parallel }}_2\mathrm{\parallel }v-z_{n_j}{\mathrm{\parallel }}_2+\mathrm{\parallel }u{\mathrm{\parallel }}_2\mathrm{\parallel }\mu -z_{n_j}{\mathrm{\parallel }}_2⟶0,$ since $z_{n_j}\to \mu$ and hence $(v-z_{n_j})$ is bounded. Passing to the limit gives $u\cdot (v-\mu )\le 0$. Taking $v=T(x)\in K$ proves the claimed support inequality.

Since every ${\pi }_{\theta }(x_i)>0$, the likelihood is positive. Repeated application of the logarithm product law [F8], followed by $\log (\exp t)=t$ [F7], gives the log likelihood $\mathrm{\ell }(\theta )=\log {\prod }_{i=1}^n{\pi }_{\theta }(x_i)={\sum }_{i=1}^n\log w(x_i)+n(\theta \cdot \bar{T}-A(\theta ))\mathrm{.}$ Because $\log$ is strictly increasing [F8], maximizing this log likelihood is equivalent to maximizing the likelihood. Also $\bar{T}$ is the convex combination with weights $1\mathrm{/}n$ of $T(x_i)$, so $\bar{T}\in K$.

Suppose $\bar{T}\in \mathrm{ri}K$. Minimality makes $K$ full-dimensional, so there is $ϵ>0$ such that the open ball of radius $ϵ$ about $\bar{T}$ lies in $K$; set $\delta =ϵ\mathrm{/}2$, so the closed ball of radius $\delta$ lies in that open ball and hence in $K$. For every unit $u$, $\bar{T}+\delta u$ is a convex combination of the vectors $T(x)$; since $u\cdot u=1$ by [F11], at least one $x$ satisfies $u\cdot (T(x)-\bar{T})\ge \delta$. Put $w_{\min }={\min }_{x\in X}w(x)>0$. If $\theta \mathrm{\ne }0$, write $r=\mathrm{\parallel }\theta {\mathrm{\parallel }}_2$ and $u=\theta \mathrm{/}r$; [F10] gives $r>0$ and $\mathrm{\parallel }u{\mathrm{\parallel }}_2=1$. For some $x$ as above, $\theta \cdot (T(x)-\bar{T})\ge r\delta$. Since the $x$th summand is part of $Z(\theta )$ and is positive by [F13], monotonicity and the product law of $\log$ [F8], together with [F7], give $A(\theta )=\log Z(\theta )\ge \log \left(w_{\min }{e}^{\theta \cdot T(x)}\right)=\log w_{\min }+\theta \cdot T(x),$ and hence $A(\theta )-\theta \cdot \bar{T}\ge r\delta +\log w_{\min }$. Thus $\theta \mapsto A(\theta )-\theta \cdot \bar{T}$ tends to $+\mathrm{\infty }$ as $\mathrm{\parallel }\theta {\mathrm{\parallel }}_2\to \mathrm{\infty }$.

Let $\theta \mathrm{\ne }\phi$ and put $v=\phi -\theta$ and $g(s)=\mathrm{\ell }(\theta +sv)$ for $s\in [0,1]$. By $A\in C^2$ [F3], $g$ is twice continuously differentiable; by [F2] its second derivative is $g^{\mathrm{\prime }\mathrm{\prime }}(s)=-n{v}^{\mathsf{T}}{\mathrm{\nabla }}^{2}A(\theta +sv)v<0$. Thus $g^{\mathrm{\prime }}$ is strictly decreasing: for $0\le a<b\le 1$, the mean value theorem [F9] applied to $g^{\mathrm{\prime }}$ gives $g^{\mathrm{\prime }}(b)-g^{\mathrm{\prime }}(a)=g^{\mathrm{\prime }\mathrm{\prime }}(c)(b-a)<0$ for some $c\in (a,b)$. Fix $t\in (0,1)$. Applying [F9] to $g$ on $[0,t]$ and $[t,1]$ gives $\alpha \in (0,t)$ and $\beta \in (t,1)$ such that $\frac{g(t)-g(0)}{t}=g^{\mathrm{\prime }}(\alpha )>g^{\mathrm{\prime }}(\beta )=\frac{g(1)-g(t)}{1-t}\mathrm{.}$ Rearranging yields $g(t)>(1-t)g(0)+tg(1)$. By [F12], $\mathrm{\ell }$ is strictly concave on ${\mathbb{R}}^k$.

The function $\mathrm{\Phi }(\theta ):=A(\theta )-\theta \cdot \bar{T}$ is continuous: $A$ is continuous by [F3], and the linear form is continuous by the Cauchy–Schwarz bound [F10]. It tends to $+\mathrm{\infty }$ by step 1.2. Choose $R>0$ large enough that $\mathrm{\Phi }(\theta )>\mathrm{\Phi }(0)$ whenever $\mathrm{\parallel }\theta {\mathrm{\parallel }}_2>R$. The closed ball ${\bar{B}}_2(0,R)$ is nonempty and compact by [F6], so [F5] gives a minimizer $\widehat{\theta }$ on it. Since $0$ lies in the ball and every point outside has value greater than $\mathrm{\Phi }(0)$, $\widehat{\theta }$ is a global minimizer on ${\mathbb{R}}^k$. The log-likelihood formula in step 1.1 and $n\ge 1$ show that it is a finite MLE. If two distinct finite MLEs existed, their midpoint would have a strictly larger log likelihood by step 2.1, a contradiction; hence it is unique.

Suppose $\bar{T}\mathrm{\notin }\mathrm{ri}K$. Since $\bar{T}\in K$ by step 1.1 and $K$ is full-dimensional, $\bar{T}$ is a boundary point. By [F4] choose a unit $u$ with $u\cdot (T(x)-\bar{T})\le 0$ for every $x$. Full affine span implies at least one of these inequalities is strict; otherwise $u\cdot T$ would be constant on $X$. Every ${\pi }_{\theta }(x)$ is positive, so $u\cdot (\mathrm{\nabla }A(\theta )-\bar{T})={\sum }_x{\pi }_{\theta }(x)u\cdot (T(x)-\bar{T})<0$ for every finite $\theta$. Therefore the directional derivative of $\mathrm{\ell }$ along $u$ is $nu\cdot (\bar{T}-\mathrm{\nabla }A(\theta ))>0$ at every finite $\theta$. By the definition of a positive derivative, a sufficiently small positive step along $u$ increases $\mathrm{\ell }$, and therefore increases the likelihood by [F8]. No finite MLE exists. This proves both directions.

Fix $x\in X$ and define $f_x(u)=u\cdot T(x)$ and $g_x(u)=f_x(u)-A(u)$. By [F2], $\mathrm{\mid }{f}_x(u)-f_x(v)\mathrm{\mid }\le \mathrm{\parallel }u-v{\mathrm{\parallel }}_2\mathrm{\parallel }T(x){\mathrm{\parallel }}_2$, so $f_x$ is continuous. The function $A$ is continuous by [F1], hence $g_x$ is continuous: at any $u_0$ and for any $\epsilon >0$, intersect neighborhoods on which $\mathrm{\mid }{f}_x(u)-f_x(u_0)\mathrm{\mid }<\epsilon \mathrm{/}2$ and $\mathrm{\mid }A(u)-A(u_0)\mathrm{\mid }<\epsilon \mathrm{/}2$. By continuity of $\exp$ [F3], its composition with $g_x$ is continuous (choose the continuity neighborhood of $g_x(u_0)$ first, then use continuity of $g_x$); multiplying by the fixed scalar $w(x)$ preserves continuity by rescaling $\epsilon$ by $w(x)>0$. Thus the displayed formula makes $u\mapsto {\pi }_u(x)$ continuous. A continuous real-valued function is Borel measurable because preimages of open sets are open and open sets generate the Borel sigma-algebra. Therefore composing $u\mapsto {\pi }_u(x)$ with the measurable $\vartheta$ makes $\theta \mapsto {\pi }_{\vartheta (\theta )}(x)$ measurable.

For each fixed $\theta$, every summand in $Q(\theta ,B)$ is nonnegative. For pairwise disjoint events $B_j$, countable additivity of each $K((\theta ,x),\cdot )$ and the finiteness of $X$ let the finite sum over $x$ commute with the nonnegative series over $j$, so $Q(\theta ,\cdot )$ is countably additive. Also $Q(\theta ,\mathsf{Y})={\sum }_x{\pi }_{\vartheta (\theta )}(x)=1$.