Marginal likelihoods, Bayes factors, and empirical Bayes

Let $k,r\ge 2$, let $\mu \in \mathbb{R}$, ${\tau }^2\ge 0$, and ${\sigma }^2>0$. For $i=1,\dots ,k$, let the group blocks $U_i=(a_i,{\mathit{ϵ}}_i)$ be independent random elements on one probability space, where ${\mathit{ϵ}}_i=({ϵ}_{i1},\dots ,{ϵ}_{ir})$. Within each block, $a_i$ and ${\mathit{ϵ}}_i$ are independent and the coordinates of ${\mathit{ϵ}}_i$ are mutually independent. Assume $a_i\sim N(0,{\tau }^2)$ and ${ϵ}_{ij}\sim N(0,{\sigma }^2)$. Put $Y_{ij}=\mu +a_i+{ϵ}_{ij}$, ${\bar{Y}}_{i\cdot }=r^{-1}{\sum }_{j=1}^r{Y}_{ij}$, and ${\bar{Y}}_{\cdot \cdot }=k^{-1}{\sum }_{i=1}^k{\bar{Y}}_{i\cdot }$. Define $MSB=\frac{r}{k-1}{\sum }_{i=1}^k({\bar{Y}}_{i\cdot }-{\bar{Y}}_{\cdot \cdot }{)}^2,MSW=\frac{1}{k(r-1)}{\sum }_{i=1}^k{\sum }_{j=1}^r(Y_{ij}-{\bar{Y}}_{i\cdot }{)}^2\mathrm{.}$ Then $\mathbb{E}[MSB]={\sigma }^2+r{\tau }^2,\mathbb{E}[MSW]={\sigma }^2\mathrm{.}$ Equating these two population moments to the observed mean squares gives the raw method-of-moments estimates ${\widehat{{\sigma }^2}}_{\mathrm{M}\mathrm{O}\mathrm{M}}=MSW,{\widehat{{\tau }^2}}_{\mathrm{M}\mathrm{O}\mathrm{M}}=\frac{MSB-MSW}{r}\mathrm{.}$ The second estimate is not constrained to be nonnegative.

Let $I=\{1,\dots ,N\}$ with $N\ge 2$ be a finite model index set. For each $i\in I$, let $({\mathrm{\Theta }}_i,{\mathcal{T}}_i)$ be a measurable parameter space, let ${\mathrm{\Pi }}_i$ be a probability measure (Probability measures and probability spaces) on it, and let $Q_i$ be a sampling kernel for a statistical experiment (Statistical experiments) on one common data space $(\mathcal{Y},\mathcal{A})$. Assume the experiments are dominated (Dominated statistical experiments) by the same supplied sigma-finite measure $\mu$. Fix a specified likelihood density (Likelihood densities) $f_i:{\mathrm{\Theta }}_i\times \mathcal{Y}\to [0,\mathrm{\infty }]$ such that $Q_i(\theta ,B)={\int }_B{f}_i(\theta ,y)\mu (dy)(B\in \mathcal{A})\mathrm{.}$ Require that each $f_i$ is jointly measurable in parameter and data; separate likelihood versions do not imply this joint measurability. Assign prior model weights $w_i>0$ with ${\sum }_i{w}_i=1$. The marginal likelihood, or model evidence, at $y$ is the possibly extended value $m_i(y):={\int }_{{\mathrm{\Theta }}_i}{f}_i(\theta ,y){\mathrm{\Pi }}_i(d\theta )\in [0,\mathrm{\infty }]\mathrm{.}$ This is the BCS-02 evidence normalizer (Evidence normalizer and fiberwise posterior update) specialized to a proper model prior, and is the prior-predictive density relative to the specified common data measure. For models $i,j$ whose evidences at $y$ are both finite and positive, the Bayes factor for $j$ against $i$ is $B{F}_{ji}(y):=\frac{m_j(y)}{m_i(y)}\mathrm{.}$ The corresponding prior model odds are $w_j\mathrm{/}{w}_i$. No Bayes factor is defined here using an arbitrarily scaled improper parameter prior.

A Bayes factor is an odds multiplier. This page supplies no universal cutoff that converts its numerical magnitude into a binary decision about which model is true.

Assume the finite dominated model setup of Marginal likelihoods, proper model evidence, and Bayes factors. Fix a data value $y$ such that $0<m_i(y)<\mathrm{\infty }$ for every $i\in I$. A version of the posterior model probabilities at this fiber is $\mathrm{Pr}(M=i\mid y)=\frac{w_i{m}_i(y)}{{\sum }_{\mathrm{\ell }\in I}{w}_{\mathrm{\ell }}{m}_{\mathrm{\ell }}(y)}\mathrm{.}$ For any two models $i,j\in I$, the posterior odds satisfy $\frac{\mathrm{Pr}(M=j\mid y)}{\mathrm{Pr}(M=i\mid y)}=\frac{w_j}{w_i}B{F}_{ji}(y)\mathrm{.}$ The displayed version uses the specified jointly measurable density representatives; no quotient is formed if an evidence is zero or infinite.

Assume a finite-group hierarchy as in Finite-group hierarchical Bayesian model, with hyperparameter $\eta \in H$, group parameters ${\theta }_1,\dots ,{\theta }_m$, conditional probability kernel $G_{\eta }$ (Measure kernel and probability kernel) on the group-parameter space, and specified jointly measurable likelihood density $L_y({\theta }_{1:m})={\prod }_{j=1}^m{f}_j({\theta }_j,y_j)$ against the common product data measure. For fixed $\eta$, define its marginal likelihood $m(y\mid \eta ):={\int }_{{\mathrm{\Theta }}^m}{L}_y({\theta }_{1:m})G_{\eta }^{\otimes m}(d{\theta }_{1:m})\in [0,\mathrm{\infty }]\mathrm{.}$ Where the function $\eta \mapsto m(y\mid \eta )$ is finite-valued and attains a positive finite maximum, a Type-II maximum marginal likelihood choice is any $\widehat{\eta }(y)\in {\mathrm{arg\hspace{0.17em}max}}_{\eta \in H}m(y\mid \eta )\mathrm{.}$ This definition does not assert that a maximum exists. A data-indexed choice requires a selected measurable maximizer on its domain.

Only when $0<m(y\mid \widehat{\eta }(y))<\mathrm{\infty }$ define the corresponding empirical-Bayes plug-in posterior on a measurable $A\subseteq {\mathrm{\Theta }}^m$ by ${\mathrm{\Pi }}_{\widehat{\eta }(y),y}(A):=\frac{{\int }_A{L}_y({\theta }_{1:m})G_{\widehat{\eta }(y)}^{\otimes m}(d{\theta }_{1:m})}{m(y\mid \widehat{\eta }(y))}\mathrm{.}$ The finite-positive condition makes this a probability update by Fiberwise posterior propriety is equivalent to finite positive evidence.

Under a proper hyperprior $H_0$, at a data fiber with finite positive joint evidence $0<Z(y):={\int }_{H}m(y\mid \eta )H_0(d\eta )<\mathrm{\infty }$, full Bayes instead retains the posterior law of $\eta$ jointly with the group parameters, as in Hierarchical posterior normalization on positive finite evidence. Integrating over a shared uncertain hyperparameter can induce dependence between group parameters, while a plug-in update fixes that hyperparameter at one selected value; special cases can leave the groups independent. This definition asserts no generic empirical-Bayes risk, consistency, dominance, or credible-coverage guarantee.