Absolute convergence and tail bound for bounded discounted rewards

For every $j\ge 0$, $0\le {\gamma }^j\mathrm{\mid }{\rho }_{j+1}\mathrm{\mid }\le M{\gamma }^j$. By [F1] and the scaling part of [F3], the majorant series ${\sum }_{j\ge 0}M{\gamma }^j$ converges (and when $M=0$ it is the zero series); [F2] gives convergence of ${\sum }_{j\ge 0}{\gamma }^j\mathrm{\mid }{\rho }_{j+1}\mathrm{\mid }$. Thus the stated series converges absolutely by definition of absolute convergence.

Every finite partial sum satisfies $\mid {\sum }_{j=0}^{N-1}{\gamma }^j{\rho }_{j+1}\mid \le M{\sum }_{j=0}^{N-1}{\gamma }^j\le M{\sum }_{j=0}^{\mathrm{\infty }}{\gamma }^j=M\mathrm{/}(1-\gamma )$. The second inequality is [F7] applied to the nonnegative geometric series in [F1]. At $N=0$ the sum is empty and the inequality holds as well.

For $K\ge N$, [F4] gives $\mid {\sum }_{j=N}^{K-1}{\gamma }^j{\rho }_{j+1}\mid \le M{\sum }_{j=N}^{K-1}{\gamma }^j\le M{\sum }_{j=N}^{\mathrm{\infty }}{\gamma }^j=M{\gamma }^N\mathrm{/}(1-\gamma )$, where [F7] applies to the shifted geometric series and [F3] evaluates its sum. The finite tail sums converge to $G-{\sum }_{j<N}{\gamma }^j{\rho }_{j+1}$ by step 1.1, and each lies between the negatives and positives of the displayed bound; [F6] passes both inequalities to the limit. Therefore the required tail estimate holds.

Taking $N=0$ in step 2.1 gives $\mathrm{\mid }G\mathrm{\mid }\le M\mathrm{/}(1-\gamma )$. The right side of the tail bound tends to zero by [F5], so the partial sums converge to $G$ with the stated uniform remainder control.