Adaptive empirical-Bayes credible-set calibration and failure

By [F1]–[F2], the independent prior coordinates and independent normal errors can be realized on countable product probability spaces. Fix one coordinate and put $s_i=i^{-r}$. By [F4], [F9] and [F11], the prior and sampling densities with respect to Lebesgue measure are $g_i(t)=(2\pi s_i{)}^{-1\mathrm{/}2}{e}^{-t^2\mathrm{/}(2s_i)},q_i(x\mid t)=\sqrt{\frac{n}{2\pi }}{e}^{-n(x-{\kappa }_{i}t{)}^2\mathrm{/}2}\mathrm{.}$ Their product has exponent $-\frac{1}{2}[(s_i^{-1}+n{\kappa }_i^2)t^2-2n{\kappa }_{i}xt+n{x}^2]$. Completing the square in $t$ gives conditional variance $v_i=(s_i^{-1}+n{\kappa }_i^2{)}^{-1}$ and conditional mean $m_i(x)=v_{i}n{\kappa }_{i}x$. Integrating the same product over $t$ gives the prior-predictive density $\frac{\exp \{-x^2\mathrm{/}[2(n^{-1}+{\kappa }_i^2{s}_i)]\}}{\sqrt{2\pi (n^{-1}+{\kappa }_i^2{s}_i)}}\mathrm{.}$ This is the density of $N(0,n^{-1}+{\kappa }_i^2{s}_i)$ by [F4], [F9] and [F11]. In the procedure's parameter range, [F8] identifies the same coordinate update.

The scalar posterior densities in step 1.1 are probability densities by [F4], [F9] and [F11]. For each fixed data sequence, [F2] defines the product posterior measure $K(x,\cdot )$. On a finite-coordinate parameter cylinder $A$, $K(x,A)$ is a finite product of normal probabilities, hence is Borel in $x$. The scalar Bayes identity and independence of the coordinate pairs give the conditional-law identity for such $A$ and every finite-coordinate data cylinder. For fixed $A$, both sides, as functions of the data event, are finite measures; the scalar Bayes identity with the full data event shows that both have total mass equal to the prior probability of $A$. If that mass is positive, normalize both measures and apply [F6]; if it is zero, both measures are identically zero. The data cylinders generate the data sigma-algebra by [F3], so the identity holds for every data event. The class of parameter sets $A$ for which $K(\cdot ,A)$ is Borel and the identity holds for every data event contains the parameter-cylinder $\pi$-system. It is a Dynkin system because complements use $1-K(x,A)$ and differences of finite measures, while countable disjoint unions use pointwise limits of measurable partial sums and countable additivity of the joint measures. A second use of [F3] and [F6] proves the identity for every product-measurable $A$. Therefore $K$ is a posterior probability kernel.

Write $a_i=n{\kappa }_i^2{s}_i$. The density of the marginal $N(0,n^{-1}(1+a_i))$ coordinate relative to $N(0,n^{-1})$ is $L_i(x_i)=(1+a_i{)}^{-1\mathrm{/}2}\exp \{n{a}_i{x}_i^2\mathrm{/}[2(1+a_i)]\}$. Since ${\sum }_i{a}_i\le n{C}^2{\sum }_i{i}^{-1-2\alpha -2p}<\mathrm{\infty }$, [F5] and [F10] give ${\sum }_i{a}_i{Z}_i^2<\mathrm{\infty }$ almost surely under $Q_{n,0}$, where $Z_i=\sqrt{n}{X}_i$. Hence ${\sum }_i\mathrm{\mid }\log L_i(X_i)\mathrm{\mid }<\mathrm{\infty }$ there. Choose $J$ so $a_i<1$ for $i>J$. Direct Gaussian integration gives $E_{Q_{n,0}}{L}_i^2=(1-a_i^2{)}^{-1\mathrm{/}2}$ for $i>J$, and ${\prod }_{i>J}(1-a_i^2{)}^{-1\mathrm{/}2}<\mathrm{\infty }$ because ${\sum }_i{a}_i^2<\mathrm{\infty }$. The finite tail products are mean-one martingales and are Cauchy in $L^2$: for $m>k>J$, independence gives $E\mathrm{\mid }{\prod }_{i=J+1}^m{L}_i-{\prod }_{i=J+1}^k{L}_i{\mathrm{\mid }}^2=E({\prod }_{i=J+1}^k{L}_i^2)\{{\prod }_{i=k+1}^m(1-a_i^2{)}^{-1\mathrm{/}2}-1\}$, which tends uniformly to zero as $k\to \mathrm{\infty }$. The finite initial product is independent of the tail and has mean one, so multiplication preserves $L^1$ convergence. By absolute convergence of the log series, the nonnegative limit is ${\prod }_i{L}_i=\exp ({\sum }_i\log L_i)$ and has expectation one. It agrees with the marginal product law on every finite cylinder, so [F6] identifies it as $d{Q}_{n,\alpha }\mathrm{/}d{Q}_{n,0}$. Taking logarithms and substituting $s_i=i^{-r}$ gives the displayed ${\mathrm{\ell }}_n(\alpha ;x)$.

Under $P_{{\theta }_0}^{(n)}$, [F10] and the independence in the model give $E{X}_i^2={\kappa }_i^2{\theta }_{0,i}^2+n^{-1}$. The sum ${\sum }_i{a}_{i}E(n{X}_i^2)$ is finite: its noise part is ${\sum }_i{a}_i$, and its signal part is $n^2{\sum }_i{\kappa }_i^4{i}^{-r}{\theta }_{0,i}^2\le n^2{C}^4{\sum }_i{\theta }_{0,i}^2<\mathrm{\infty }$. Thus the same absolute-convergence bound for ${\sum }_i\mathrm{\mid }\log L_i(X_i)\mathrm{\mid }$ holds under the true sampling law.

By [F5] and [F10], the prior has expected squared norm ${\sum }_i{i}^{-r}<\mathrm{\infty }$, hence belongs to ${\mathrm{\ell }}^2$ almost surely. The posterior variances satisfy ${\sum }_i{v}_i\le {\sum }_i{i}^{-r}<\mathrm{\infty }$. For its mean, write $m_i(X_i)=w_i{\theta }_{0,i}+\sqrt{n}{\kappa }_i(i^r+n{\kappa }_i^2{)}^{-1}{Z}_i$, where $w_i=n{\kappa }_i^2\mathrm{/}(i^r+n{\kappa }_i^2)\le 1$. The cross term has mean zero because $E{Z}_i=0$ by [F10], so $E_{{\theta }_0}^{(n)}{\sum }_i{m}_i(X_i{)}^2\le {\sum }_i{\theta }_{0,i}^2+n{C}^2{\sum }_i{i}^{-2r-2p}<\mathrm{\infty }\mathrm{.}$ Therefore the posterior mean is in ${\mathrm{\ell }}^2$ almost surely; for almost every such observation its posterior expected squared norm is ${\sum }_i(m_i(X_i{)}^2+v_i)<\mathrm{\infty }$, so the posterior gives ${\mathrm{\ell }}^2$ mass one.

Uniformly for $\alpha \le A$, for all sufficiently large $n$ we have $\log N_{\alpha }\ge 8$, $\lceil N_{\alpha }\mathrm{/}\rho \rceil \ge N_0$, $\log N_{\alpha }\ge 2\log \rho$ and $(1+\rho \mathrm{/}{N}_{\alpha }{)}^{r_{\alpha }}\le 2$. Indeed $N_A$ grows exponentially in $\sqrt{\log n}$ whereas $r_A=O(\sqrt{\log n})$. We henceforth impose these cutoffs. For $q\ge 1$, $0\le k\le 4$ and $\log N\ge 8$, the bound $\log i\mathrm{/}\log N\le \exp (q\log (i\mathrm{/}N)\mathrm{/}(2k))$ for $i>N$ and $k>0$ follows from $1+x\le e^x$ and $\log N\ge 2k\mathrm{/}q$. Summing the resulting powers and comparing a decreasing function with its integral gives ${\sum }_{i>N}{i}^{-1-q}(\log i{)}^k\le 3N^{-q}(\log N{)}^k$; $k=0$ follows directly by integration.

Temporarily suppress $\alpha$ from $r,s,N,V$. The positive deterministic noise series $D={\sum }_i{n}^2\log i\mathrm{/}(i^r+n{)}^2$ satisfies $V\mathrm{/}8\le D\le 3V$. For the lower bound retain $i\le \lfloor N\rfloor$, replace the denominator by $4n^2$, and use ${\sum }_{i\le \lfloor N\rfloor }\log i\ge N\log N\mathrm{/}2$ for $N\ge e^8$, obtained by integrating $\log x$. For the upper bound the terms below $N$ total at most $V$. The remaining terms are bounded by $n^2{\sum }_{i>N}{i}^{-2r}\log i$, which is at most $V\{(2r-1{)}^{-1}+[(2r-1{)}^2\log N{]}^{-1}+N^{-1}\}<2V$, by integrating $x^{-2r}\log x$ and including its value at $N$.

For a polished-tail truth put $m=\lceil N\mathrm{/}\rho \rceil$, $B=\{m,\dots ,\lfloor \rho m\rfloor \}$ and $E_B={\sum }_{i\in B}{\theta }_i^2$. On $B$ one has $N\mathrm{/}\rho \le i\le N+\rho$, $\log i\ge \log N\mathrm{/}2$ and $i^r+n\le 3n$. Thus $S_{\alpha }\ge {\rho }^{-s}{N}^s(\log N)E_B\mathrm{/}18$, and ${\sum }_{i>N}{\theta }_i^2\le L_0{E}_B$. For $i>N$, $f_{\alpha }(i)\log i\le n^2{i}^{-1-2\alpha -4p}(\log i{)}^2\le N^s(\log N{)}^2$, since the function on the right before evaluation at $N$ decreases beyond $N$. Below $N$, $\log i\le \log N$. Consequently ${\sum }_i{f}_{\alpha }(i)(\log i){\theta }_i^2\le (1+18L_0{\rho }^s)(\log N)S_{\alpha }\mathrm{.}$ All these inequalities hold also when $S_{\alpha }=0$.

For every $i$, $f_{\alpha }(i)\le 2N^s\log N$. Below $N$ use the denominator bound $n^2$; above $N$ write $i=Nx$ and $q=1+2\alpha +4p\ge 1$, obtaining the upper bound $N^s{x}^{-q}(\log N+\log x)$, and use $x^{-q}\log x\le 1\mathrm{/}(eq)$. For $a\le b$ and $2\le i\le N_b$, $f_a(i)\mathrm{/}{f}_b(i)=i^{-2(b-a)}[(i^{r_b}+n)\mathrm{/}(i^{r_a}+n){]}^2\le 4$. Apply step 2.2 at $b$ to the energy above $N_b$, and the kernel supremum at $a$, to obtain $S_a\le 4S_b+36L_0{\rho }^{s_b}\frac{N_a^{s_a}\log N_a}{N_b^{s_b}\log N_b}{S}_b\mathrm{.}$ Multiply by $V_b\mathrm{/}{V}_a$. The identities $N_{\alpha }^{s_{\alpha }}=n{N}_{\alpha }^{-2p}$, $\log N_b\mathrm{/}\log N_a\le 1$ and $(N_b\mathrm{/}{N}_a{)}^{2p}\le 1$ give the stated comparison of $h_a,h_b$, without dividing by a possibly zero signal.

Differentiation under the signal sum is valid by domination by a bounded multiplier of ${\theta }_i^2$, since its tail and each of its first derivatives are bounded by a constant times $i^{-1-4p}(\log i{)}^k$. It gives $S_{\alpha }^{\mathrm{\prime }}=2{\sum }_i{f}_{\alpha }(i)(\log i)(n-i^{r_{\alpha }})\mathrm{/}(n+i^{r_{\alpha }}){\theta }_i^2$. Step 2.2 bounds its absolute value by $2(1+18L_0{\rho }^{s_{\alpha }})(\log N_{\alpha })S_{\alpha }$. Moreover $V_{\alpha }^{\mathrm{\prime }}\mathrm{/}{V}_{\alpha }=-2(1+\log N_{\alpha })\mathrm{/}{r}_{\alpha }$. Therefore $G_{\alpha }=V_{\alpha }+S_{\alpha }$ is positive and continuously differentiable, with $\mathrm{\mid }{G}_{\alpha }^{\mathrm{\prime }}\mathrm{\mid }\mathrm{/}{G}_{\alpha }\le K{L}_0{\rho }^{1+2A}\log n$.

Set $t_i=i^{1+2\alpha }{\kappa }_i^{-2}$ and $a_i=n^2{t}_i\log i\mathrm{/}(t_i+n{)}^2$. Direct differentiation of finite likelihood sums, followed by centering, gives $E_{\theta }{M}_n(\alpha )={\sum }_i\frac{n^2\log i}{(t_i+n{)}^2}-{\sum }_i\frac{n^2{i}^{1+2\alpha }(\log i){\theta }_i^2}{(t_i+n{)}^2},Q_n(\alpha )=-{\sum }_i{a}_i(\alpha )(X_i^2-E_{\theta }{X}_i^2)\mathrm{.}$ Ellipticity compares $t_i+n$ to $i^{r_{\alpha }}+n$ with factors $C^{\pm 2}$. Thus step 2.1 proves both asserted expectation inequalities. By [F4], $E{Z}_i=0$ and $E{Z}_i^2=1$. Two integrations by parts using ${\varphi }^{\mathrm{\prime }}(x)=-x\varphi (x)$ give $E{Z}_i^4=3$; the boundary terms vanish because every polynomial times $\varphi$ tends to zero at infinity. Hence $E\mathrm{\mid }{Z}_i{\mathrm{\mid }}^3\le (E{Z}_i^2{)}^{1\mathrm{/}2}(E{Z}_i^4{)}^{1\mathrm{/}2}<\mathrm{\infty }$ by [F3]. The density $\varphi$ is even, so the integrable odd moment satisfies $E{Z}_i^3=0$. Expanding $X_i^2-E_{\theta }{X}_i^2$ now gives $\mathrm{Var}(X_i^2)=2\mathrm{/}{n}^2+4{\kappa }_i^2{\theta }_i^2\mathrm{/}n$. Independence makes variances additive. Finally $a_i^{\mathrm{\prime }}=2(\log i)a_i(n-t_i)\mathrm{/}(n+t_i)$, so $\mathrm{\mid }{a}_i^{\mathrm{\prime }}\mathrm{\mid }\le 2(\log i)a_i$.

For $j=0,1$ the preceding identities give ${\mathrm{Var}}_{\theta }{Q}_n^{(j)}(\alpha )\le K{N}_{\alpha }(\log N_{\alpha }{)}^{2+2j}(1+h_{\alpha })\mathrm{.}$ Here are the series estimates in full. The noise contribution is bounded by $K{\sum }_i[n{i}^{r_{\alpha }}\mathrm{/}(i^{r_{\alpha }}+n{)}^2{]}^2(\log i{)}^{2+2j}$. Below $N$ bound the bracket by one; above $N$ bound it by $n{i}^{-r}$ and use step 1.1 with $q=2r-1\ge 1$. The result is $KN(\log N{)}^{2+2j}$. The signal contribution is bounded by $K{\sum }_i{f}_{\alpha }(i){\theta }_i^2[n{i}^r(\log i{)}^{1+2j}\mathrm{/}(i^r+n{)}^2]$. The bracket is at most $K(\log N{)}^{1+2j}$: below $N$ this is immediate, and above $N$ it follows from $x^{-r}(\log N+\log x{)}^{1+2j}\le K(\log N{)}^{1+2j}$ for $x=i\mathrm{/}N\ge 1$, $r\ge 1$. The signal contribution is therefore at most $K{S}_{\alpha }(\log N{)}^{1+2j}=KN(\log N{)}^{2+2j}{h}_{\alpha }$. This calculation uses only $\theta \in {\mathrm{\ell }}^2$, not polished tails.

The random sums have common versions suitable for integration and suprema. On each fixed $[0,A]$, the variance-series tails for $Q_n,Q_n^{\mathrm{\prime }}$ tend to zero uniformly: coarse dominating summands are constants times $i^{-2-4p}(\log i{)}^4$ and $i^{-1-4p}(\log i{)}^4{\theta }_i^2$, both summable. For a finite centered tail $T$, the fundamental theorem of calculus and Cauchy–Schwarz give $E\sup \mathrm{\mid }T\mathrm{\mid }\le \sqrt{ET(0{)}^2}+{\int }_0^A\sqrt{E{T}^{\mathrm{\prime }}(u{)}^2}du$. Thus partial sums converge in expected uniform norm to a continuous $Q_n$. Their derivatives converge in $L^1(dPd\alpha )$, so the limit has absolutely continuous paths and the fundamental theorem passes to the limit. Deterministic expectation sums converge uniformly by the analogous kernel domination. Choose an almost surely uniformly convergent subsequence of score sums. On the procedure's full-probability domain, the finite likelihood sums converge at zero to its prescribed value. Integrating those finite scores and passing along this subsequence gives ${\mathrm{\ell }}_n(\alpha )={\mathrm{\ell }}_n(0)+{\int }_0^{\alpha }{M}_n(u)du$ simultaneously on $[0,A]$. Its continuous integrand is therefore the asserted derivative, including the right endpoint derivative at zero.

Put $R_{\alpha }=Q_n(\alpha )\mathrm{/}{G}_{\alpha }$. By step 4.1, $E\mathrm{\mid }{R}_{\alpha }\mathrm{\mid }\le K\mathrm{/}\sqrt{N_{\alpha }}$. The quotient rule, step 3.2 and Cauchy–Schwarz give $E\mathrm{\mid }{R}_{\alpha }^{\mathrm{\prime }}\mathrm{\mid }\le K{L}_0{\rho }^{1+2A}(\log n)\mathrm{/}\sqrt{N_{\alpha }}$. The absolutely continuous version from step 5.1 satisfies ${\sup }_{[0,A]}\mathrm{\mid }R\mathrm{\mid }\le \mathrm{\mid }{R}_0\mathrm{\mid }+{\int }_0^A\mathrm{\mid }{R}_{\alpha }^{\mathrm{\prime }}\mathrm{\mid }d\alpha$. Taking expectations by Tonelli and using $N_{\alpha }\ge N_A$ proves the first bound. For arbitrary truths on an interval where $h\le H$, instead normalize by $V$ alone. Step 4.1 and $\mathrm{\mid }{V}^{\mathrm{\prime }}\mathrm{/}V\mathrm{\mid }\le 3\log n$ yield $E\mathrm{\mid }Q\mathrm{/}V\mathrm{\mid }\le K_H\mathrm{/}\sqrt{N_A}$ and $E\mathrm{\mid }(Q\mathrm{/}V{)}^{\mathrm{\prime }}\mathrm{\mid }\le K_H\log n\mathrm{/}\sqrt{N_A}$. The same integral inequality proves the final asserted bound, including $c=0$.

Write $\lambda =\log (\rho \vee e)\ge 1$ and $t=\sqrt{\log n}$. At the maximal allowed $A$, $2A\log \rho \le t\sqrt{\lambda }\mathrm{/}2$, whereas $\frac{1}{2}\log N_A=t^2\mathrm{/}[2(1+2A+2p)]\ge t\sqrt{\lambda }-O(1)$. Consequently ${\rho }^{2A}\mathrm{/}\sqrt{N_A}\le K\exp (-t\sqrt{\lambda }\mathrm{/}2)$, which dominates every polynomial in $\log n$. The first bound tends to zero; the arbitrary-truth bound is smaller apart from a fixed factor. For smaller $A$, the numerators only decrease and $N_A$ increases. This proves all uniform convergence assertions.

Set $u=(16C^8{)}^{-1}$ and $v=8C^8$. Continuity of $h_n$ implies $h_n\le u$ on $[0,{\underset{‾}{\alpha }}_n]$ whenever ${\underset{‾}{\alpha }}_n>0$, and $h_n\ge v$ on $[{\bar{\alpha }}_n,A]$ whenever ${\bar{\alpha }}_n<A$. These implications include the empty-set conventions. Moreover ${\underset{‾}{\alpha }}_n\le {\bar{\alpha }}_n$: otherwise any point strictly between them would have $h_n<u$ by the first-infimum definition and $h_n>v$ by the last-supremum definition, a contradiction. At a nontrivial threshold, continuity gives $h_n({\underset{‾}{\alpha }}_n)\ge u$ if ${\underset{‾}{\alpha }}_n<A$ and $h_n({\bar{\alpha }}_n)\le v$ if ${\bar{\alpha }}_n>0$.

Write $V_{\alpha }=N_{n,\alpha }\log N_{n,\alpha }$. The expectation bounds in F2 imply $E{M}_n\mathrm{/}{V}_{\alpha }\ge 1\mathrm{/}(16C^4)$ when $h_n\le u$, and $E{M}_n\mathrm{/}(V_{\alpha }{h}_n)\le -5\mathrm{/}(8C^4)$ when $h_n\ge v$. Therefore $E{M}_n\mathrm{/}[V_{\alpha }(1+h_n)]$ has a positive margin at least $1\mathrm{/}(32C^4)$ on the lower interval and a negative margin of magnitude at least $1\mathrm{/}(32C^4)$ on the upper interval. For a nonnegative variable $Y$, integrating $\epsilon {\mathbf{1}}_{\{Y>\epsilon \}}\le Y$ proves $P(Y>\epsilon )\le EY\mathrm{/}\epsilon$. The uniform centered-score bound and this inequality show that, uniformly in polished-tail truths, with probability tending to one the likelihood derivative is positive throughout the lower interval and negative throughout the upper interval. Integration shows strict increase and decrease, including comparisons with the interval endpoints. Every global maximizer lies in the asserted bracket, proving the first conclusion.

For an arbitrary square-summable truth with ${\underset{‾}{\alpha }}_n>0$, the lower interval still has $h_n\le u$. Apply the final, arbitrary-truth bound in F2 to $[0,{\underset{‾}{\alpha }}_n]$ with $H=u$. The positive expectation margin $1\mathrm{/}(16C^4)$ is unchanged. Markov's inequality makes the derivative positive throughout that interval with probability tending to one uniformly in all such truths. Thus no global maximizer lies below ${\underset{‾}{\alpha }}_n$. If this endpoint is zero, the conclusion holds automatically.