PAC-Bayes priors, posteriors, and relative entropy

Let $(\mathrm{\Theta },\mathcal{A})$ be a measurable space. A prior is a probability measure $\pi$ on $(\mathrm{\Theta },\mathcal{A})$ fixed independently of the training sample. A posterior relative to $\pi$ is a probability measure $\rho$ on $(\mathrm{\Theta },\mathcal{A})$ that is absolutely continuous with respect to $\pi$, meaning that $\pi (A)=0$ implies $\rho (A)=0$ for every $A\in \mathcal{A}$.

For such $\rho$ and $\pi$, define their Kullback–Leibler relative entropy by $\mathrm{K}\mathrm{L}(\rho \mathrm{\parallel }\pi ):={\sup }_{\mathcal{P}}{\sum }_{A\in \mathcal{P}}\rho (A)\log \frac{\rho (A)}{\pi (A)},$ where the supremum is over all finite measurable partitions $\mathcal{P}$ of $\mathrm{\Theta }$. A term with $\rho (A)=0$ is $0$; a term with $\rho (A)>0=\pi (A)$ is $+\mathrm{\infty }$. The value may be $+\mathrm{\infty }$.

Measurable spaces and measurable sets: a measurable space is a set equipped with a specified sigma-algebra.

Probability measures and probability spaces: a probability measure is a measure of total mass one.

The natural logarithm as the inverse of the exponential function: $\log$ is the inverse of the real exponential on the positive reals.

Alquier, User-friendly Introduction to PAC-Bayes Bounds, Chapter 2: uses a sample-independent prior and relative entropy between posterior and prior.