Finite KL-regularized policy improvement

Let $Y$ be a finite nonempty set. A probability vector on $Y$ is a family $p=(p_y{)}_{y\in Y}$ with $p_y\ge 0$ and ${\sum }_{y\in Y}{p}_y=1$. For probability vectors $p,q$, define the finite relative entropy $D(p\mathrm{\parallel }q)={\sum }_{y:p_y>0}{p}_y\log \frac{p_y}{q_y}$ when $p_y=0$ whenever $q_y=0$, and define $D(p\mathrm{\parallel }q)=+\mathrm{\infty }$ otherwise. Terms with $p_y=0$ are omitted. Given a real reward $r:Y\to \mathbb{R}$ and a coefficient $\beta >0$, define $F_{q,r,\beta }(p)={\sum }_{y\in Y}{p}_{y}r(y)-\beta D(p\mathrm{\parallel }q)\mathrm{.}$ The finite KL-regularized policy-improvement problem is to maximize $F_{q,r,\beta }$ over all probability vectors $p$ on $Y$. The reference $q$ has full support when $q_y>0$ for every $y\in Y$; in that case $F_{q,r,\beta }(p)$ is finite for every $p$.

Finite KL divergence: uses the same directed finite sum, omits zero first-coordinate terms, and assigns $+\mathrm{\infty }$ when $p$ has positive mass where $q$ is zero.

Natural logarithm: defines $\log$ on $(0,\mathrm{\infty })$ as the inverse of the exponential function.