Identifiability and approximation bias in computational inference — Examples

On $\mathrm{\Theta }$, $0\le ({\theta }^2-1{)}^2\le 9$, so $0<p_{n,\theta }(0)\le 1\mathrm{/}2$. Also $3\mathrm{/}16\le p_{n,\theta }(1)\le 5\mathrm{/}16$, hence $p_{n,\theta }(2)\ge 1-1\mathrm{/}2-5\mathrm{/}16=3\mathrm{/}16$; the three entries are nonnegative and sum to one. Thus each $P_{n,\theta }$ is a probability measure on $\mathcal{X}$. Since $p_{n,\theta }(1)=1\mathrm{/}4+\theta \mathrm{/}32$ is injective, $P_{n,\theta }=P_{n,\eta }$ implies $\theta =\eta$, so the identity estimand is identifiable by [F2].

The Borel sets of $\mathrm{\Theta }$ are Lebesgue measurable, $\lambda ([-2,2])=4$, and ${\mathrm{\ell }}_{n,0}(\theta )=p_{n,\theta }(0)$ is continuous and satisfies $\frac{1}{2}{e}^{-9n}\le {\mathrm{\ell }}_{n,0}(\theta )\le \frac{1}{2}$. Therefore ${\mathrm{\Pi }}_0$ is a probability measure, ${\mathrm{\ell }}_{n,0}$ is measurable, and $0<\frac{1}{2}{e}^{-9n}\le Z_n=\int {\mathrm{\ell }}_{n,0}d{\mathrm{\Pi }}_0\le \frac{1}{2}<\mathrm{\infty }$. By [F1], ${\mathrm{\Pi }}_n$ is the exact posterior target with density proportional to $e^{-n({\theta }^2-1{)}^2}$ relative to ${\mathrm{\Pi }}_0$.

The phase $h$ has exactly two global maxima, at $\theta =\pm 1$, because $({\theta }^2-1{)}^2=0$ exactly there and is positive elsewhere. Direct differentiation gives $h^{\mathrm{\prime }\mathrm{\prime }}(\theta )=4-12{\theta }^2$, so $h^{\mathrm{\prime }\mathrm{\prime }}(\pm 1)=-8$; each local negative Hessian is full rank, but [F3]'s unique-maximum hypothesis fails. Writing $t=1+u$ gives $h(1+u)=-(2u+u^2{)}^2=-4u^2-4u^3-u^4$, so its quadratic term at $+1$ is $-4u^2$. The one-mode Laplace density is therefore proportional to $e^{-4n{u}^2}$, exactly the stated $q_n$, and integrates to one by [A1, F4].

Under the affine change $z=\sqrt{8n}(t-1)$, [F5] sends $Q_n((0,\mathrm{\infty }))$ to the standard normal probability of $(-\sqrt{8n},\mathrm{\infty })$. These sets are increasing. Given any $z\in \mathbb{R}$, choose $n>z^2\mathrm{/}8$ by [F9]; positivity and monotonicity of the square root then give $\sqrt{8n}>\mathrm{\mid }z\mathrm{\mid }$, so their union is all of $\mathbb{R}$. Continuity from below [F6] therefore gives $Q_n((0,\mathrm{\infty }))\to 1$.

Define $w_n(t)=e^{-n(t^2-1{)}^2}{\mathbf{1}}_{[-2,2]}(t)$ on $\mathbb{R}$; it is a nonnegative even Borel function. Apply [F5] to the reflection diffeomorphism $T(t)=-t$ on $\mathbb{R}$, using the function $w_n{\mathbf{1}}_{(0,\mathrm{\infty })}$: the unnormalized integrals of $w_n$ over $(0,\mathrm{\infty })$ and $(-\mathrm{\infty },0)$ are equal. The singleton $\{0\}$ has Lebesgue measure zero by [F8], so these two integrals split the prior-weighted posterior normalizer equally. Thus ${\bar{\mathrm{\Pi }}}_n((0,\mathrm{\infty }))={\mathrm{\Pi }}_n((0,\mathrm{\infty }))=1\mathrm{/}2$.

Using event $(0,\mathrm{\infty })$ in [F7], $\parallel Q_n-{\bar{\mathrm{\Pi }}}_n{\parallel }_{\mathrm{T}\mathrm{V}}\ge \mathrm{\mid }{Q}_n((0,\mathrm{\infty }))-{\bar{\mathrm{\Pi }}}_n((0,\mathrm{\infty }))\mathrm{\mid }$, whose right side tends to $1\mathrm{/}2$ by steps 2.1 and 2.2. Thus ${\mathrm{lim\hspace{0.17em}inf}}_{n\to \mathrm{\infty }}\parallel Q_n-{\bar{\mathrm{\Pi }}}_n{\parallel }_{\mathrm{T}\mathrm{V}}\ge 1\mathrm{/}2$, proving that the local curvature at one mode misses persistent posterior mass at the other.

Let $\rho =1\mathrm{/}2$, so $\beta =1-{\rho }^2=3\mathrm{/}4$, and write $\varphi (t)=(2\pi {)}^{-1\mathrm{/}2}{e}^{-t^2\mathrm{/}2}$. Countable additivity of the supplied area measure makes $A\mapsto {\lambda }_2(A\times (0,1])$ a Borel measure, with ${\lambda }_1((a,b])=b-a$ by rectangle volume. The explicit intervals $(-n,n]$ exhaust $\mathbb{R}$ and have supplied measure $2n$, so the factor measure is sigma-finite. Its product exists by [F7]. It and the supplied ${\lambda }_2$ agree on half-open boxes by the volume formula and on the finite-mass exhaustion $(-n,n{]}^2$; [F7] and [F8] identify their Borel measures. All displayed densities and polynomial moment integrands are continuous and hence Borel: inverse images of open sets are open, and inverse-image preservation of sigma-algebra operations gives the Borel assertion. Thus [F8] makes them product-measurable. The compact comparison and Gaussian integral statements in [F3] now apply to these supplied measures. Direct algebra gives $\gamma (x,y)=\varphi (x)g_{\rho x,\beta }(y)=\varphi (y)g_{\rho y,\beta }(x)$. Each inner Gaussian density has mass one by [F3], so Tonelli [F4] proves that $\gamma$ has mass one and both marginals $\varphi$. By [F3] and density integration [F6], the marginal means are zero and second moments are one. Since $\mathrm{\mid }xy\mathrm{\mid }\le (x^2+y^2)\mathrm{/}2$, the cross moment is absolutely integrable. Fubini [F5], [F6] and the mean of $g_{\rho x,\beta }$ in [F3] give $E[XY]=\int x\varphi (x)\rho xdx=\rho$. Thus the target has marginal variances $1$ and covariance $1\mathrm{/}2$. With likelihood one the evidence is $1$, so [F1] identifies $\gamma$ as the exact target.

For $m_j\in \mathbb{R}$ and $v_j>0$, [F3] supplies the normalized Lebesgue densities $g_{m_j,v_j}$ and their means $m_j$ and variances $v_j$. Tonelli [F4] gives total mass one for $q$. The logarithms of $\gamma$ and $q$ are quadratic; the finite second moments and $\mathrm{\mid }xy\mathrm{\mid }\le (x^2+y^2)\mathrm{/}2$ make their absolute values integrable under $q$. Both densities are everywhere positive, so $q$ is admissible by [F2]. Fubini [F5] and [F6] give $E_q[X^2]=m_1^2+v_1$, $E_q[Y^2]=m_2^2+v_2$, and $E_q[XY]=m_1{m}_2$.

One has $\log \gamma =-\log (2\pi \sqrt{\beta })-(x^2-xy+y^2)\mathrm{/}(2\beta )$ and $-E_{g_{m_j,v_j}}\log g_{m_j,v_j}=\frac{1}{2}\log (2\pi v_j)+\frac{1}{2}$, using the mean and variance in step 2.1. Substituting these expressions and the second moments from step 2.1 into [F2] yields $\mathrm{ELBO}(q)=C-\frac{m_1^2-m_1{m}_2+m_2^2+v_1+v_2}{2\beta }+\frac{1}{2}\log v_1+\frac{1}{2}\log v_2$, where $C$ is independent of $(m_1,m_2,v_1,v_2)$. The mean quadratic equals $(m_1-\rho m_2{)}^2+\beta m_2^2$, so its unique minimum and the ELBO-maximizing means are $m_1=m_2=0$.

For either variance the variable part is $r(v)=-v\mathrm{/}(2\beta )+\frac{1}{2}\log v$. The identity has derivative $1$ and constants have derivative $0$ by their difference quotients; [F9] then gives that $r$ is differentiable on $(0,\mathrm{\infty })$ with $r^{\mathrm{\prime }}(v)=(\beta -v)\mathrm{/}(2\beta v)$. If $0<v<\beta$, the mean value theorem [F10] gives $c\in (v,\beta )$ with $r(\beta )-r(v)=r^{\mathrm{\prime }}(c)(\beta -v)>0$. If $v>\beta$, it gives $c\in (\beta ,v)$ with $r(v)-r(\beta )=r^{\mathrm{\prime }}(c)(v-\beta )<0$. Thus $v=\beta =3\mathrm{/}4$ is the unique global maximum. The target variance is $1$ and covariance is $1\mathrm{/}2$ by step 1.1, whereas the optimizing independent factors have variance $3\mathrm{/}4$ and covariance $0$ by step 2.1. Thus the unique ELBO optimizer differs from the exact target.

Each row is nonnegative and sums to one. The laws differ because $P_0(\{1\})=1\mathrm{/}4$ whereas $P_1(\{1\})=3\mathrm{/}8$; hence the identity estimand is identifiable by [F3]. The finite identity summary is measurable because the source and target carry their stated power-set and Borel sigma-algebras.

Under $P_0$, $E[S]=(0+1+2+3)\mathrm{/}4=3\mathrm{/}2$ and $E[S^2]=(0+1+4+9)\mathrm{/}4=7\mathrm{/}2$. Under $P_1$, the corresponding weighted sums are $(9+6+21)\mathrm{/}24=3\mathrm{/}2$ and $(9+12+63)\mathrm{/}24=7\mathrm{/}2$. Thus both variances equal $7\mathrm{/}2-(3\mathrm{/}2{)}^2=5\mathrm{/}4>0$, so the one-dimensional covariance is positive definite and the ideal Gaussian synthetic likelihood is defined by [F2], [F4].

The exact likelihoods at $y=1$ are $1\mathrm{/}4$ and $3\mathrm{/}8$, so the equal-prior exact posterior mass at $\theta =1$ is $(3\mathrm{/}8)\mathrm{/}(1\mathrm{/}4+3\mathrm{/}8)=3\mathrm{/}5$. The ideal Gaussian synthetic likelihoods coincide because their means and variances coincide. Their common positive value cancels in normalization against the equal prior, so the synthetic-likelihood posterior has mass $1\mathrm{/}2$ at $\theta =1$.

The two posterior measures differ on event $\{1\}$ by $\mathrm{\mid }3\mathrm{/}5-1\mathrm{/}2\mathrm{\mid }=1\mathrm{/}10$, so their total-variation distance is at least $1\mathrm{/}10$ by [F5]. Thus matching the first two moments does not remove this posterior target bias.

Put the discrete metric on $\mathrm{\Theta }$ and $\mathsf{Y}$. A Cauchy sequence is eventually constant because its tail has pairwise distances below $1\mathrm{/}2$; each finite space is separable since the whole finite set is a countable dense subset. Every subset is open, so the Borel sigma-algebra is the power set. Thus both are standard Borel by [F5]. The real target is standard Borel by [F5].

The event $A=\{(z,w):z=0\}$ has probabilities $P_0(A)=1$ and $P_1(A)=0$. Thus the two full-data laws are not observationally equivalent, and the identity parameter is identifiable by [F3].

The full-data likelihood at $y=(0,0)$ is $1\mathrm{/}2$ under $\theta =0$ and $0$ under $\theta =1$. Its prior evidence is $(1\mathrm{/}2)(1\mathrm{/}2)+(1\mathrm{/}2)0=1\mathrm{/}4$, so [F4] gives exact posterior ${\mathrm{\Pi }}_y={\delta }_0$.

For each $\theta$, exactly two pairs $(z,w)$ have $z=\theta$, each with mass $1\mathrm{/}2$, so $P_{\theta }$ is a probability law; because the parameter sigma-algebra is also the power set, the displayed family is a probability kernel. The map $S$ is nonconstant, since $S(0,0)=0$ and $S(0,1)=1$, and it is measurable because every subset of the finite source is measurable.

The summary law is Bernoulli$(1\mathrm{/}2)$ under each parameter. The zero-tolerance ball in $\mathbb{R}$ centered at $S(y)=0$ is $\{0\}$, and each parameter has acceptance probability $1\mathrm{/}2$, so prior-predictive acceptance is $1\mathrm{/}2>0$. Conditioning the equal prior on this event leaves ${\mathrm{\Pi }}_{\mathrm{A}\mathrm{B}\mathrm{C},0}\{\theta =0\}=\frac{(1\mathrm{/}2)(1\mathrm{/}2)}{1\mathrm{/}2}=1\mathrm{/}2$ by [F1].

On event $\{\theta =0\}$ the exact posterior gives mass $1$ and the zero-tolerance ABC target gives mass $1\mathrm{/}2$. By [F6], their total-variation distance is at least $1\mathrm{/}2$. The full model is identifiable, but the nonconstant summary has parameter-independent law and its ABC target remains the prior.

Each row sums to one and has nonnegative entries, so it defines a probability measure at each parameter. Every event probability is measurable in the finite parameter space, hence the table defines a probability kernel by [F6]. Under the discrete metric a Cauchy sequence is eventually constant (take tail distances below $1\mathrm{/}2$); each finite set is separable and all of its subsets are open. Hence both are standard Borel. The summary map $S$ is measurable to the real Borel sigma-algebra because every preimage is a subset of the finite measurable data space, by [F3].

At observation $1$, the likelihoods are $1\mathrm{/}5$ and $4\mathrm{/}5$, so the exact posterior mass at $\theta =1$ is $(4\mathrm{/}5)\mathrm{/}(1\mathrm{/}5+4\mathrm{/}5)=4\mathrm{/}5$ by [F4]. The closed $ϵ$-ball around $1$ is $\{1\mathrm{/}2,1\}$. Its conditional probabilities are $P_0(B)=3\mathrm{/}5+1\mathrm{/}5=4\mathrm{/}5$ and $P_1(B)=1\mathrm{/}10+4\mathrm{/}5=9\mathrm{/}10$. Therefore the prior-predictive acceptance probability is $\frac{1}{2}(4\mathrm{/}5+9\mathrm{/}10)=17\mathrm{/}20>0$.

Conditioning the prior-simulator law on this ball gives ABC mass at $\theta =1$ equal to $\frac{(1\mathrm{/}2)(9\mathrm{/}10)}{17\mathrm{/}20}=9\mathrm{/}17$ by [F1]. The exact and ABC posterior masses differ by $\mathrm{\mid }4\mathrm{/}5-9\mathrm{/}17\mathrm{\mid }=23\mathrm{/}85$.

The measurable events in $\mathrm{\Theta }=\{0,1\}$ are $\mathrm{\varnothing }$, $\{0\}$, $\{1\}$, and $\mathrm{\Theta }$. The exact and ABC probabilities agree on $\mathrm{\varnothing }$ and $\mathrm{\Theta }$; on $\{1\}$ their difference has absolute value $\mathrm{\mid }4\mathrm{/}5-9\mathrm{/}17\mathrm{\mid }=23\mathrm{/}85$, and on $\{0\}$ it has the same absolute value because both are probability measures. Thus [F5] gives total-variation distance exactly $23\mathrm{/}85$. If the proposal-simulator pairs $({\mathrm{\Theta }}_j,Y_j)$ are IID under the prior-kernel joint law, [F2] identifies this ABC target as the accepted-parameter law; the finite calculations make no additional convergence or efficiency claim.