Hitting probability as minimal harmonic extension

If $E$ is finite or countably infinite, fix an increasing finite exhaustion $E_j\uparrow E$ (using a fixed enumeration when $E$ is infinite), and for $\varphi :E\to [0,+\mathrm{\infty }]$ put ${\varphi }_j(y)={\mathbf{1}}_{E_j}(y)(\varphi (y)\wedge j)$. Each ${\varphi }_j$ is bounded and simple by [F12]; the nonnegative integral [F13], its simple-function agreement [F15], the simple integral formula [F14], the atomic weights [F4] and [F2] give $K{\varphi }_j(x)={\int }_E{\varphi }_j(y)K(x,dy)={\sum }_{y\in E_j}p(x,y)(\varphi (y)\wedge j)$. As $j\uparrow \mathrm{\infty }$, [F11] passes the integrals to $K\varphi (x)$, while the finite sums increase to the support-restricted extended row sum by [F16] and [F3]; thus $K\varphi (x)=P\varphi (x)$, with zero weights omitted and no $0\cdot (+\mathrm{\infty })$ formed.

If $E=\mathrm{\varnothing }$, there is no state or initial law to check; if $A=\mathrm{\varnothing }$, then $T_A=\mathrm{\infty }$ by [F1, F17], so $h=0$ and $h\le g$ follows from $g\ge 0$; if $A=E$, every start has $T_A=0$ and $h=1$, while every admissible $g$ is also $1$; in general, [F1, F7] give $h(x)=1$ whenever $x\in A$.

Define $H(\omega )={\mathbf{1}}_{\{\mathrm{\exists }n\ge 0:{\omega }_n\in A\}}$, a bounded product-measurable path functional. By [F5], $h$ is measurable and ${\mathbb{E}}_x[H(X_1,X_2,\dots )\mid {\mathcal{F}}_1]=h(X_1)$; for $x\in A^c$, hitting $A$ is equivalent to the shifted path hitting it, so [F9] gives $h(x)={\mathbb{E}}_{x}h(X_1)$. The bounded one-step identity [F6], $X_0=x$ [F7], and expectation preservation [F9] identify this as $Kh(x)$; step 1.1 then gives $Kh(x)=Ph(x)$.

For any finite-valued nonnegative $g$ and each $n$, use the finite-support truncations $g_j$ from step 1.1. The bounded one-step identity [F6] and step 1.1 give $\mathbb{E}[g_j(X_{n+1})\mid {\mathcal{F}}_n]=K{g}_j(X_n)=P{g}_j(X_n)$; since $g_j\uparrow g$, conditional monotone convergence [F8] and the row-sum limit in step 1.1 yield $\mathbb{E}[g(X_{n+1})\mid {\mathcal{F}}_n]=Pg(X_n)$ almost surely, with its event-integral characterization available even when the conditional value is infinite.

Fix $x\in A^c$, put $T=T_A$, $Y_n=g(X_{n\wedge T})$, and $S_n=\{T>n\}\in {\mathcal{F}}_n$ by [F1]. If ${\mathbb{E}}_x{Y}_n=g(x)<\mathrm{\infty }$, then on $S_n$ one has $X_n\in A^c$ and $Pg(X_n)=g(X_n)$; [F8] and step 2.2 give ${\mathbb{E}}_x[{\mathbf{1}}_{S_n}g(X_{n+1})]={\mathbb{E}}_x[{\mathbf{1}}_{S_n}Pg(X_n)]={\mathbb{E}}_x[{\mathbf{1}}_{S_n}g(X_n)]$. On $S_n^c$, $Y_{n+1}=Y_n=1$, and on $S_n$, $Y_n=g(X_n)$ and $Y_{n+1}=g(X_{n+1})$. Since $X_0=x$ by [F7], induction from $Y_0=g(x)$ proves ${\mathbb{E}}_x{Y}_n=g(x)$ and integrability for every finite $n$.

On $\{T<\mathrm{\infty }\}$, $Y_n=1$ for every $n\ge T$, while on $\{T=\mathrm{\infty }\}$ all $Y_n\ge 0$; hence ${\mathbf{1}}_{\{T<\mathrm{\infty }\}}\le {\mathrm{lim\hspace{0.17em}inf}}_n{Y}_n$. Fatou [F10] and step 3.1 give $h(x)\le {\mathbb{E}}_x{\mathrm{lim\hspace{0.17em}inf}}_n{Y}_n\le {\mathrm{lim\hspace{0.17em}inf}}_n{\mathbb{E}}_x{Y}_n=g(x)$ for $x\in A^c$, and step 1.2 covers $x\in A$, $A=\mathrm{\varnothing }$, and $A=E$. A one-state absorbing chain is included by those same two set cases, and the finite-row argument in step 1.1 covers deterministic transitions. AC [A1] is used for the canonical laws and the conditional-expectation/Markov suppliers; the row exhaustion uses the supplied countability witness, with no extra choice. The result asserts minimality and no biconditional.