First-step equations for nonnegative exit costs

On $E^{{\mathbb{N}}_0}$, put $T(\omega )=\inf \{n\ge 0:{\omega }_n\in D^c\}$, extend $f$ by zero on $D$ and $c$ by zero on $D^c$, and define $B(\omega )={\sum }_{n\ge 0}f({\omega }_n){\mathbf{1}}_{\{T(\omega )=n\}}$, $C(\omega )={\sum }_{m\ge 0}c({\omega }_m){\mathbf{1}}_{\{m<T(\omega )\}}$, and $R=B+C$. The hitting events are measurable by [F22], the restrictions by [F9], the nonnegative series by [F10], and their sum by [F16]; thus $R$ is measurable, $B=0$ on $T=\mathrm{\infty }$ by [F17], and no coordinate at infinity is evaluated.

If $x\in D^c$, then $X_0=x$ almost surely under ${\mathbb{P}}_x$ by [F5], so $T=0$ by [F1], $B=f(x)$ and $C=0$; hence $u(x)={\mathbb{E}}_{x}R=f(x)$. This includes an empty $D$ and a start already on the boundary.

If $x\in D$, then every path starting at $x$ has $T\ge 1$; its shifted path exits at time $T-1$ when $T<\mathrm{\infty }$ and never exits when $T=\mathrm{\infty }$, so $R(\omega )=c(x)+R({\omega }_1,{\omega }_2,\dots )$ in extended nonnegative arithmetic. Additivity [F8] gives $u(x)=c(x)+{\mathbb{E}}_{x}R(X_1,X_2,\dots )$ without subtraction, also when the tail expectation is infinite.

For $N\ge 1$, let $H_N=R\wedge N$ and $h_N(y)={\mathbb{E}}_y{H}_N(X_0,X_1,\dots )$; then $h_N$ is measurable and bounded by $N$ by [F3], and its conditional future-path identity at time $1$, followed by [F15], gives ${\mathbb{E}}_x{H}_N(X_1,X_2,\dots )={\mathbb{E}}_x{h}_N(X_1)$.

The bounded one-step identity [F4], expectation preservation [F15], and $X_0=x$ under ${\mathbb{P}}_x$ [F5] give ${\mathbb{E}}_x{h}_N(X_1)=K{h}_N(x):={\int }_E{h}_N(y)K(x,dy)$.

For a nonnegative simple $s={\sum }_{i=1}^r{a}_i{\mathbf{1}}_{A_i}$ with disjoint measurable $A_i$, [F19] identifies its nonnegative integral with its simple integral [F13], and countable singleton weights [F6] give ${\int }_{E}s(y)K(x,dy)={\sum }_i{a}_{i}K(x,A_i)={\sum }_i{a}_i{\sum }_{y\in A_i}p(x,y)={\sum }_{y:p(x,y)>0}p(x,y)s(y)$. For general nonnegative measurable $g$, choose simple $s_j\uparrow g$ by [F12]; MCT [F7] passes the integrals to the limit, while [F20] fixes the nonnegative integral and [F21] ensures the increments $s_{i+1}-s_i$ are finite-valued nonnegative simple functions. Thus $s_j={\sum }_{i<j}(s_{i+1}-s_i)$ with $s_0=0$; Tonelli [F14] interchanges the increment and state sums when $E$ is countably infinite, using its fixed enumeration, while for finite $E$ the limit passes through the finite sum. It follows that ${\int }_{E}g(y)K(x,dy)={\sum }_{y:p(x,y)>0}p(x,y)g(y)$, the support-restricted action [F18].

As $N\to \mathrm{\infty }$, $H_N(X_1,X_2,\dots )\uparrow R(X_1,X_2,\dots )$ and $h_N(y)\uparrow u(y)$ for every $y$; [F7] and measurability of the increasing limit [F11] give ${\mathbb{E}}_{x}R(X_1,X_2,\dots )={\int }_{E}u(y)K(x,dy)={\sum }_{y:p(x,y)>0}p(x,y)u(y)=Pu(x)$ by steps 1.4–1.6 and [F2]. Combining with step 1.3 proves $u(x)=c(x)+Pu(x)$ on $D$, including the value $+\mathrm{\infty }$.

If $E=\mathrm{\varnothing }$, there is no probability law of an $E$-valued chain and no state to check; if $D=\mathrm{\varnothing }$, step 1.2 covers every state; if $D=E$, the boundary payoff is zero and the equation still holds when $T=\mathrm{\infty }$ and $u=+\mathrm{\infty }$. If $f=c=0$, then $u=0$; a one-state absorbing chain in $D$ with positive cost has $u=+\mathrm{\infty }=c+Pu$, and deterministic rows obey the same shift calculation. The endpoint $T=0$ is handled in step 1.2, whereas on $D$ one has $T\ge 1$ and the exit-time cost is excluded by $m<T$. AC [A1] is used for the canonical laws and conditional-expectation/Markov identities [F3]–[F5], [F15]; countability supplies the fixed row representation, with no additional choice principle. The two equations form no biconditional.