Finite Rényi divergence

Let $P=(p_x)$ and $Q=(q_x)$ be probability vectors on the same finite nonempty alphabet. For order $\alpha \in (0,\mathrm{\infty })$ with $\alpha \mathrm{\ne }1$, put $S_{\alpha }(P,Q):={\sum }_{x:q_x>0}{p}_x^{\alpha }{q}_x^{1-\alpha }\mathrm{.}$ If $\alpha >1$ and $p_x>0=q_x$ for some $x$, define $S_{\alpha }(P,Q)=+\mathrm{\infty }$; if $0<\alpha <1$, such coordinates contribute zero. In the displayed sum, $p_x=0<q_x$ contributes zero. If the sum is zero, then necessarily $\alpha <1$ and the logarithmic expression below is defined as $+\mathrm{\infty }$. The Rényi divergence of order $\alpha$ from $P$ to $Q$ is $D_{\alpha }(P\mathrm{\parallel }Q):=\frac{\log S_{\alpha }(P,Q)}{\alpha -1}\mathrm{.}$ At order one define $D_1(P\mathrm{\parallel }Q):=D(P\mathrm{\parallel }Q),$ using Finite Kullback–Leibler divergence. All divergences are extended-real-valued; the direction is always first argument to second.

For $0<\alpha <1$, a point with $p_x>0=q_x$ contributes zero to the affinity $S_{\alpha }$ rather than infinity. For $\alpha >1$, the same support mismatch makes $D_{\alpha }(P\mathrm{\parallel }Q)=+\mathrm{\infty }$. The sign of $\alpha -1$ is part of the definition and reverses inequalities when $0<\alpha <1$.