The finite Rényi divergence limit is KL

Suppose $P\ll Q$. On the finite set $\mathrm{supp}P$, every $p_x,q_x$ is positive. Put $r_x=p_x\mathrm{/}{q}_x$ and define $S(h):={\sum }_{p_x>0}{p}_x{r}_x^h$ for every real $h$. Then $S(h)>0$, $S(0)=1$, and $S(h)=S_{1+h}(P,Q)$ for $0<\mathrm{\mid }h\mathrm{\mid }<1$ by [F3]. By [F4] and [F5], $S^{\mathrm{\prime }}(0)={\sum }_{p_x>0}{p}_x\log r_x=D(P\mathrm{\parallel }Q)$. The chain rule [F6] gives $(\log S{)}^{\mathrm{\prime }}(0)=D(P\mathrm{\parallel }Q)$. Since $\log S(0)=0$, the definition of this derivative proves the two-sided limit.

Suppose $P\ll ̸Q$. Let $s:={\sum }_{x:q_x>0}{p}_x<1$. For $0<\alpha <1$, the finite sum in [F3] tends to $s$ as $\alpha \uparrow 1$, because each term with $q_x>0$ tends to $p_x$. If $s=0$, that sum is identically zero and $D_{\alpha }=+\mathrm{\infty }$. If $s>0$, then $\log S_{\alpha }\to \log s<0$ while $\alpha -1\to 0$ through negative values, so $D_{\alpha }\to +\mathrm{\infty }$ from below order one.

For every $\alpha >1$, the positive $P$-mass at a zero $Q$-coordinate makes $S_{\alpha }=+\mathrm{\infty }$, hence $D_{\alpha }=+\mathrm{\infty }$ by [F3]. Also $D(P\mathrm{\parallel }Q)=+\mathrm{\infty }$ by [F2]. The left and right limits therefore both equal the stated extended value.