Approximate Bayesian computation and simulation-based inference — Examples

Equip ${\mathbb{N}}_{\ge 1}$ with the discrete metric $d(n,m)=0$ for $n=m$ and $1$ otherwise. A Cauchy sequence, meaning that its tail terms are pairwise less than every positive radius apart, is eventually constant by taking radius $1\mathrm{/}2$, so this metric is complete. The set is countable and dense in itself; each singleton is open, so every subset is open and its Borel sigma-algebra is the power set. The identity is therefore a Borel isomorphism to this Polish presentation, and [F1] gives standard Borelness.

Put ${\pi }_n=1\mathrm{/}[n(n+1)]$. The partial sums telescope: ${\sum }_{n=1}^N{\pi }_n={\sum }_{n=1}^N(\frac{1}{n}-\frac{1}{n+1})=1-\frac{1}{N+1}⟶1$ by [F4]. Therefore $\mathrm{\Pi }={\sum }_{n\ge 1}{\pi }_n{\delta }_n$ is a probability measure by [F3].

Define $b(s)=\frac{1}{4}{\mathbf{1}}_{[-1,1]}(s)$, $w_n=1\mathrm{/}(100n^3)$, ${\varphi }_n(s)=(4w_n{)}^{-1}{\mathbf{1}}_{[1\mathrm{/}n-w_n,1\mathrm{/}n+w_n]}(s)$, and $m_n=b+{\varphi }_n$. The two summands are nonnegative measurable functions with integrals $1\mathrm{/}2$ each, so $m_n$ is a probability density. Let $K(n,ds)=m_n(s)ds$. It is a probability kernel: every section is a probability measure, and every function of $n$ is measurable on the discrete parameter space. For measurable $A\subseteq {\mathbb{N}}_{\ge 1}$ and Lebesgue-measurable $C\subseteq \mathbb{R}$, the ABC definition gives $M(A\times C)={\sum }_{n\in A}{\pi }_n{\int }_C{m}_n(s)ds$. To extend this identity from rectangles, let $E\subseteq {\mathbb{N}}_{\ge 1}\times \mathbb{R}$ be product-measurable and put $E_n=\{s:(n,s)\in E\}$. The section map $s\mapsto (n,s)$ is measurable, so each $E_n$ is Lebesgue-measurable; because ${\mathbb{N}}_{\ge 1}$ is countable discrete, $E={⨆}_{n\ge 1}(\{n\}\times E_n)$. Countable additivity and the rectangle identity give $M(E)={\sum }_{n\ge 1}{\pi }_n{\int }_{E_n}{m}_n(s){\lambda }_1(ds)\mathrm{.}$ The function $m(n,s):=m_n(s)$ is product-measurable because ${\mathbb{N}}_{\ge 1}$ is countable discrete and every $m_n$ is measurable. The prior $\mathrm{\Pi }$ is sigma-finite and Lebesgue measure is sigma-finite, so [F7] applied to ${\mathbf{1}}_{E}m$ gives ${\int }_{E}m(n,s)(\mathrm{\Pi }\times {\lambda }_1)(d(n,s))={\sum }_{n\ge 1}{\pi }_n{\int }_{E_n}{m}_n(s){\lambda }_1(ds)\mathrm{.}$ Thus $M$ has the specified jointly measurable density on the full product sigma-algebra, not only on rectangles. The identity summary is measurable.

For each fixed $n$, the spike interval is a positive distance from $0$, since $1\mathrm{/}n-w_n>0$. On a sufficiently small neighborhood of $0$, $m_n(s)=1\mathrm{/}4$; hence $m_n$ is continuous at $0$ and $m_n(0)=1\mathrm{/}4$. Thus $z_0={\sum }_n{\pi }_n{m}_n(0)=1\mathrm{/}4$, and the density-selected zero-tolerance posterior is $\mathrm{\Pi }$ itself. However, for every $\delta >0$, all sufficiently large $n$ have $1\mathrm{/}n<\delta$, and on the neighborhood $(-\delta ,\delta )$ any common pointwise envelope $H(n)$ must satisfy $H(n)\ge m_n(1\mathrm{/}n)=\frac{1}{4}+25n^3\mathrm{.}$ Since ${\pi }_n\ge 1\mathrm{/}(2n^2)$, ${\sum }_n{\pi }_{n}H(n)\ge \frac{25}{2}{\sum }_{n}n=\mathrm{\infty }$. Every prior atom has positive mass, so an almost-everywhere envelope condition also forces this bound for every $n$. No integrable local envelope exists.

For integer $q\ge 2$, take ${ϵ}_q=1\mathrm{/}q$ and $B_q=[-1\mathrm{/}q,1\mathrm{/}q]$. The base density contributes $1\mathrm{/}(2q)$ to every parameter's ball probability by the box-volume formula [F5]. If $n<q$, then $1\mathrm{/}n-w_n>1\mathrm{/}q$: indeed $(q-n)\mathrm{/}(nq)\ge 1\mathrm{/}(n(n+1))>1\mathrm{/}(100n^3)$. If $n>q$, then $1\mathrm{/}n+w_n<1\mathrm{/}q$: here $(n-q)\mathrm{/}(qn)\ge 1\mathrm{/}(q(q+1))>1\mathrm{/}(100(q+1{)}^3)\ge 1\mathrm{/}(100n^3)$. Thus spikes for $n<q$ contribute zero; the $n=q$ ball cuts its symmetric spike at the center and captures half its mass, namely $1\mathrm{/}4$; and spikes for $n>q$ lie wholly in the ball and contribute $1\mathrm{/}2$.

By [F6] and the telescoping tail ${\sum }_{n>q}{\pi }_n=1\mathrm{/}(q+1)$, the acceptance probability is $p_q=\frac{1}{2q}+\frac{1}{4q(q+1)}+\frac{1}{2(q+1)},q{p}_q⟶1.$ For the fixed parameter event $A=\{1\}$, its unnormalized accepted mass is ${\pi }_1\mathrm{/}(2q)=1\mathrm{/}(4q)$, because its spike lies outside $B_q$. Hence ${\mathrm{\Pi }}_{{ϵ}_q}(A\mid 0)=(1\mathrm{/}(4q))\mathrm{/}{p}_q\to 1\mathrm{/}4$, whereas the density-selected limit assigns $A$ mass $\mathrm{\Pi }(A)=1\mathrm{/}2$. The total-variation distance is at least this event-probability difference, which tends to $1\mathrm{/}4$. Thus the claimed convergence fails without the integrable-envelope condition.

On either finite set, the discrete metric has values $0$ and $1$. Since Cauchy means that tail distances are eventually below every positive radius, a Cauchy sequence is eventually constant at radius $1\mathrm{/}2$, so the metric is complete. Each finite set is countable and dense in itself, and all singletons are open; thus its Borel sigma-algebra is the power set. The identity is a Borel isomorphism to this Polish presentation, so [F4] shows both spaces are standard Borel. By [F5], the usual metric on $\mathbb{R}$ is complete and separable and induces its usual Borel sigma-algebra, so the summary target is standard Borel. The constant map $S$ is measurable.

The sampling laws are $P_0={\delta }_0$ and $P_1={\delta }_1$, hence $P_0\mathrm{\ne }{P}_1$. By [F1], the identity parameter is identifiable. The marginal probability of the observation $y_0=0$ is $1\mathrm{/}2$; its prior-weighted likelihoods are $1\mathrm{/}2$ at parameter $0$ and $0$ at parameter $1$. Therefore the full-data conditional law is ${\delta }_0$, satisfying the posterior event identity [F7].

For every $ϵ\ge 0$, the event $\mathrm{\mid }S(Y)-0\mathrm{\mid }\le ϵ$ is the whole data space. It has probability one, so the ABC target is the unchanged prior, with mass $1\mathrm{/}2$ at each parameter. For rejection sampling supplied with the IID pairs of [F6], the first proposal is accepted and its output has this same law. The target's total-variation distance from the full-data posterior is at least the discrepancy on $\{0\}$, namely $\mathrm{\mid }1\mathrm{/}2-1\mathrm{\mid }=1\mathrm{/}2$, by [F7].

The statistic is not sufficient. If a common kernel $Q$ existed, taking summary event $C=\{0\}$ in [F2]'s identity would force $K(\theta ,D)=Q(0,D)$ for every data event $D$ and both parameters. For $D=\{0\}$ this would require both $Q(0,\{0\})=1$ and $Q(0,\{0\})=0$, a contradiction. Thus identifiability does not prevent persistent distortion from an insufficient summary.

Give $\mathrm{\Theta }$ its discrete metric. Since Cauchy means that tail distances are eventually below every positive radius, a Cauchy sequence is eventually constant at radius $1\mathrm{/}2$, so the metric is complete; the finite set is countable and dense in itself, and its singletons are open. Its Borel sigma-algebra is the power set, so [F3] gives standard Borelness. By [F2], the usual metric on $\mathbb{R}$ is complete and has countable dense subset $\mathbb{Q}$; its balls are ordinary intervals, so it induces the usual topology. Thus both the data and summary Borel spaces are standard Borel by [F3]. Each displayed density is nonnegative measurable and integrates to one because its supporting interval has length one. Thus $K(\theta ,ds)=m_{\theta }(s)ds$ is a probability kernel on the finite parameter space, and the identity summary is measurable. For every product-measurable $E\subseteq \mathrm{\Theta }\times \mathbb{R}$, the section maps $s\mapsto (\theta ,s)$ are measurable, so $E_{\theta }=\{s:(\theta ,s)\in E\}$ is measurable, and $E={⨆}_{\theta \in \mathrm{\Theta }}\{\theta \}\times E_{\theta }$. Since $\mathrm{\Theta }$ is finite, the rectangle identity extends by this finite disjoint decomposition to $M(E)={\sum }_{\theta \in \mathrm{\Theta }}\mathrm{\Pi }(\{\theta \}){\int }_{E_{\theta }}{m}_{\theta }(s)ds={\int }_E{m}_{\theta }(s)(\mathrm{\Pi }\times {\lambda }_1)(d\theta ,ds)\mathrm{.}$ Thus the finite-parameter summary-predictive law has the specified joint density required by the zero-tolerance theorem.

For $0<ϵ<1\mathrm{/}2$, the tolerance ball about $1\mathrm{/}2$ is $[1\mathrm{/}2-ϵ,1\mathrm{/}2+ϵ]$. Its intersection with $[0,1]$ has length $2ϵ$, while it is disjoint from $[1,2]$ because its upper endpoint is strictly below $1$. The parameter-conditional acceptance probabilities are therefore $2ϵ$ at $0$ and $0$ at $1$. Averaging against the equal prior gives $p_{ϵ}=ϵ>0$, and conditioning gives ABC target ${\delta }_0$. In particular $p_{ϵ}\to 0$. By [F5], the accepted parameter also has law ${\delta }_0$.

On the neighborhood $[1\mathrm{/}4,3\mathrm{/}4]$ of $s_0=1\mathrm{/}2$, $m_0$ is identically $1$ and $m_1$ is identically $0$. Both densities are continuous at $s_0$, and the constant envelope $H(\theta )=1$ is integrable under $\mathrm{\Pi }$. The averaged density is $z_0=\frac{1}{2}{m}_0(1\mathrm{/}2)+\frac{1}{2}{m}_1(1\mathrm{/}2)=1\mathrm{/}2$, so the density-selected posterior is ${\delta }_0$.

The hypotheses of [F6] hold under the Countable Choice assumption [F4] (with, for example, neighborhood radius $1\mathrm{/}4$), so the ABC targets converge in total variation to ${\delta }_0$ as $ϵ\downarrow 0$. Here step 2.1 shows the stronger exact identity ${\mathrm{\Pi }}_{ϵ}={\delta }_0$ for every $0<ϵ<1\mathrm{/}2$, even though the acceptance probability vanishes.

Give each finite set its discrete metric $d(x,x^{\mathrm{\prime }})=0$ for equality and $1$ otherwise. Since Cauchy means that tail distances are eventually below every positive radius, a Cauchy sequence is eventually constant at radius $1\mathrm{/}2$; the metric is complete. Each finite set is countable and dense in itself, and its singletons are open, so its Borel sigma-algebra is the power set. The identity is a Borel isomorphism to this Polish presentation, so [F4] gives that $\mathrm{\Theta }$ and $\mathsf{Y}$ are standard Borel. For $\mathbb{R}$, [F5] gives a complete usual metric and countable dense subset $\mathbb{Q}$; each metric ball is an ordinary open interval, so this metric induces the usual topology. Thus $\mathbb{R}$ is Polish and its usual Borel space is standard Borel by [F4].

The prior is proper. For each $\theta$, summing the likelihood over the eight binary triples gives $(\theta +(1-\theta ){)}^3=1$, so $K$ is a probability kernel by [F1]. The count $S$ is measurable because its domain is finite.

For $k\in \{0,1,2,3\}$ let $F_k=\{y\in \mathsf{Y}:S(y)=k\}$ and define $Q(k,D)=\mathrm{\mid }D\cap F_k\mathrm{\mid }\mathrm{/}\left(\genfrac{}{}{0px}{}{3}{k}\right)$. For $t\mathrm{\notin }\{0,1,2,3\}$ set $Q(t,\cdot )={\delta }_{(0,0,0)}$. Each value is a probability measure by [F7], and for fixed $D$ the map $t\mapsto Q(t,D)$ is Borel, so $Q$ is a probability kernel. For every $\theta$, $D\subseteq \mathsf{Y}$, and Borel $C\subseteq \mathbb{R}$, $K(\theta ,D\cap S^{-1}(C))={\sum }_{k\in C\cap \{0,1,2,3\}}\mathrm{\mid }D\cap F_k\mathrm{\mid }{\theta }^k(1-\theta {)}^{3-k}$ $={\sum }_{k\in C\cap \{0,1,2,3\}}Q(k,D)\left(\genfrac{}{}{0px}{}{3}{k}\right){\theta }^k(1-\theta {)}^{3-k}={\int }_{C}Q(t,D)K_S(\theta ,dt)\mathrm{.}$ This is the common-kernel criterion [F3], so $S$ is sufficient.

The full-data likelihood at $y_0=(1,1,0)$ is ${\theta }^2(1-\theta )$. The prior-weighted likelihoods at $1\mathrm{/}3$ and $2\mathrm{/}3$ are respectively $1\mathrm{/}27$ and $2\mathrm{/}27$, so their normalization gives posterior masses $1\mathrm{/}3$ and $2\mathrm{/}3$. For $0\le ϵ<1$, the condition $\mathrm{\mid }S(Y)-2\mathrm{\mid }\le ϵ$ is exactly $S(Y)=2$. Its likelihood is $3{\theta }^2(1-\theta )$, three times the full-data likelihood at either parameter; the prior-weighted masses are $1\mathrm{/}9$ and $2\mathrm{/}9$, giving the same normalized posterior.

The event in step 3.2 has positive prior-predictive probability $1\mathrm{/}3$, so the ABC target is defined. By [F6], the rejection algorithm's accepted parameter has that target law; hence it equals the full-data posterior just computed. Explicitly, the summary likelihood differs from the full-data likelihood by the same positive factor $3$ at both parameters, which cancels from the finite posterior normalization in step 3.2; sufficiency itself was proved independently in step 3.1.