Markov Kernels and Markov Chains

Assume [F1]. If $s={\sum }_{j=1}^r{a}_j{1}_{A_j}$ is a nonnegative measurable [F1, F2] simple function, then, for $B\in {\mathcal{F}}_n$, finite additivity of the integral and [F1] give $\mathbb{E}[1_{B}s(X_{n+1})]={\sum }_j{a}_j\mathbb{E}[1_B{1}_{\{X_{n+1}\in A_j\}}]=\mathbb{E}[1_{B}Ks(X_n)]\mathrm{.}$ The variable $Ks(X_n)$ is ${\mathcal{F}}_n$-measurable by [F2], so it is a version of $\mathbb{E}[s(X_{n+1})\mid {\mathcal{F}}_n]$.

Let $0\le f\le M$. By [F3], choose simple $s_j\uparrow f$, replacing [F2, F3, F4, F5, step 1.1] $s_j$ by $s_j\wedge M$ if necessary. Then $K{s}_j(x)\to Kf(x)$ for every $x$; this follows from [F4] with the probability measure $K(x,\cdot )$ and majorant $M$. For each $B\in {\mathcal{F}}_n$, [F4] under $\mathbb{P}$ on both sides of the identity from step 1.1 gives $\mathbb{E}[1_{B}f(X_{n+1})]=\mathbb{E}[1_{B}Kf(X_n)]\mathrm{.}$ Thus $Kf(X_n)$ is a conditional-expectation version; [F5] makes the equality an equality of almost-everywhere classes.

For bounded real $f$, apply step 2.1 to $f^{+}$ and $f^{-}$. Subtracting the [F2, F5, step 2.1] two defining event-integral identities shows that $Kf(X_n)=K{f}^{+}(X_n)-K{f}^{-}(X_n)$ is a version of the desired conditional expectation. This proves the bounded-function form.

Conversely, put $f=1_A$. Then $Kf(X_n)=K(X_n,A)$, so the asserted [F1, step 3.1] bounded-function identity is exactly [F1] for $A$. Together with the forward direction through step 3.1, this proves the equivalence.

Assume (1), fix bounded $g$, and write [F1, F2, F3] $W=\mathbb{E}[g(Z)\mid \mathcal{G}]$. For $C\in \mathcal{G}$ and measurable $A$, [F1] and the defining event-integral property give $\begin{array}{rl}{\displaystyle \mathbb{E}[1_C{1}_{\{Y\in A\}}g(Z)]}& {\displaystyle =\mathbb{E}[1_C\mathbb{E}(1_{\{Y\in A\}}g(Z)\mid \mathcal{G})]}\\ {\displaystyle }& {\displaystyle =\mathbb{E}[1_C\mathbb{E}(1_{\{Y\in A\}}\mid \mathcal{G})W]}\\ {\displaystyle }& {\displaystyle =\mathbb{E}[1_C{1}_{\{Y\in A\}}W],}\end{array}$ where the last equality follows by [F3]. The events $C\cap \{Y\in A\}$ form a pi-system generating $\mathcal{G}\vee \sigma (Y)$. For fixed $g$, the events on which the first and last integrals agree form a lambda-system, so [F2] extends the equality to the whole join. Since $W$ is measurable for that join, it is a version of $\mathbb{E}[g(Z)\mid \mathcal{G}\vee \sigma (Y)]$. This proves (2), including $A=\mathrm{\varnothing }$ and $A$ equal to the whole state space.

Conversely assume (2), put $\mathcal{H}=\mathcal{G}\vee \sigma (Y)$, and take [F1, F3, F4] bounded $f,g$. By [F3], (2), and [F4], $\begin{array}{rl}{\displaystyle \mathbb{E}[f(Y)g(Z)\mid \mathcal{G}]}& {\displaystyle =\mathbb{E}[\mathbb{E}(f(Y)g(Z)\mid \mathcal{H})\mid \mathcal{G}]}\\ {\displaystyle }& {\displaystyle =\mathbb{E}[f(Y)\mathbb{E}(g(Z)\mid \mathcal{H})\mid \mathcal{G}]}\\ {\displaystyle }& {\displaystyle =\mathbb{E}[f(Y)W\mid \mathcal{G}]=\mathbb{E}[f(Y)\mid \mathcal{G}]W\mathrm{.}}\end{array}$ This is [F1], so (1) follows.

If $\varphi$ and $\psi$ are measurable, substitute $f\circ \varphi$ and [F1] $g\circ \psi$ in [F1]; boundedness and measurability are preserved. Hence $\varphi (Y)\perp \text{\hspace{0.17em}⁣}\text{\hspace{0.17em}⁣}\text{\hspace{0.17em}⁣}\perp \psi (Z)\mid \mathcal{G}$.

If $V$ is $\mathcal{G}$-measurable, then [step 1.1, step 1.2] $\mathcal{G}\vee \sigma (Y,V)=\mathcal{G}\vee \sigma (Y)$. Thus criterion (2) is unchanged after replacing $Y$ by $(Y,V)$. Symmetry gives the corresponding claim on the $Z$ side. Constant, one-point, and zero-valued $V$ are included.

Apply [F1] to the coordinate pair on [F1, F2] $(S_2\times S_3,{\mu }_{23})$ and then [F2]. This gives a probability kernel $Q:S_2\to S_3$ such that, for $A_2,A_3$ measurable, ${\mu }_{23}(A_2\times A_3)={\int }_{A_2}Q(s_2,A_3){\mu }_2(d{s}_2)\mathrm{.}$ Choice is used precisely by [F1]--[F2] to select and factor the conditional law.

Lift $Q$ to the kernel [F3, F4, F7, step 1.1] $\tilde{Q}((s_1,s_2),\cdot )=Q(s_2,\cdot )$ and, for a measurable $C\subseteq S_1\times S_2\times S_3$, set $\mu (C)={\int }_{S_1\times S_2}Q(s_2,C_{(s_1,s_2)}){\mu }_{12}(d(s_1,s_2))\mathrm{.}$ Each section $C_{(s_1,s_2)}$ is measurable by [F7], and the integrand is measurable by [F3], applied to $1_C$ and $\tilde{Q}$. For disjoint $C_j$, their sections are disjoint and [F4] passes the increasing partial sums through the outer integral. Thus $\mu$ is countably additive; $\mu (\mathrm{\varnothing })=0$ and $\mu (S_1\times S_2\times S_3)=1$. Hence it is a probability measure, including when one of the displayed test sets below is empty.

Taking $C=A_1\times A_2\times S_3$ in step 2.1 gives [F6, step 1.1, step 2.1] $\mu (C)={\mu }_{12}(A_1\times A_2)$. Taking $C=S_1\times A_2\times A_3$ and using step 1.1 gives $\mu (C)={\mu }_{23}(A_2\times A_3)$. The rectangle pi-systems and [F6] therefore identify both required marginals on their full product sigma-algebras.

For bounded measurable $g:S_3\to \mathbb{R}$, write [F3, F5, step 1.1, step 2.1, step 3.1] $Qg(s_2)=\int g(s_3)Q(s_2,d{s}_3)$. The construction in step 2.1, first for indicators, then for simple functions, and then for positive and negative parts, shows ${\mathbb{E}}_{\mu }[g(S_3)\mid \sigma (S_1,S_2)]=Qg(S_2)\mathrm{.}$ The $(2,3)$ marginal and step 1.1 likewise show ${\mathbb{E}}_{\mu }[g(S_3)\mid \sigma (S_2)]=Qg(S_2)$. Criterion [F5] now gives $S_1\perp \text{\hspace{0.17em}⁣}\text{\hspace{0.17em}⁣}\text{\hspace{0.17em}⁣}\perp S_3\mid S_2$.

Let $\nu$ be any other law with the two marginals and the stated [F5, F6, step 2.1, step 3.1] conditional independence. Its $(2,3)$ marginal makes $Qg(S_2)$ a conditional expectation of $g(S_3)$ given $S_2$; [F5] then makes it the conditional expectation given $(S_1,S_2)$. Consequently, for every measurable rectangle, $\nu (A_1\times A_2\times A_3)=\int 1_{A_1}(s_1)1_{A_2}(s_2)Q(s_2,A_3)d{\mu }_{12},$ which equals the value of $\mu$ from step 2.1. Rectangles form a pi-system containing the whole space, so [F6] gives $\nu =\mu$.

The identity kernel is a two-sided identity: directly from its Dirac [F1, F2] sections, $(IK)(x,A)=K(x,A)$ and $(KI)(x,A)=\int 1_A(y)K(x,dy)=K(x,A)$. Thus $K^{m+0}=K^m=K^m{K}^0$, including $m=0$. If $K^{m+n}=K^m{K}^n$, then [F1]--[F2] give $K^{m+n+1}=K^{m+n}K=(K^m{K}^n)K=K^m(K^{n}K)=K^m{K}^{n+1}\mathrm{.}$ Induction proves the kernel identity for all $m,n\ge 0$.

Fix $m$ and bounded measurable $f$. For $n=0$, $f(X_m)$ is [F1, F3, F4, step 1.1] ${\mathcal{F}}_m$-measurable and hence is its own conditional expectation; it is also $K^{0}f(X_m)$. Suppose the formula holds at $n$. By [F3] at time $m+n$, [F4], and the induction hypothesis applied to the bounded measurable function $Kf$, $\begin{array}{rl}{\displaystyle \mathbb{E}[f(X_{m+n+1})\mid {\mathcal{F}}_m]}& {\displaystyle =\mathbb{E}[\mathbb{E}(f(X_{m+n+1})\mid {\mathcal{F}}_{m+n})\mid {\mathcal{F}}_m]}\\ {\displaystyle }& {\displaystyle =\mathbb{E}[Kf(X_{m+n})\mid {\mathcal{F}}_m]=K^n(Kf)(X_m)=K^{n+1}f(X_m)\mathrm{.}}\end{array}$ The last equality is the definition of kernel composition from step 1.1. Induction proves the conditional-expectation formula. Choice is used only by [F3]--[F4], which operate on conditional-expectation classes.

Taking $f=1_A$ in step 2.1 gives the displayed conditional-probability [F3, step 2.1] formula; conversely that formula for all $A$ gives the bounded-function formula by the preceding lemma. Empty and full $A$ yield respectively zero and one on both sides.

Define backward, starting with $G_r=f_r$, by [F2, F3] $G_j(x)=f_j(x)K^{n_{j+1}-n_j}{G}_{j+1}(x),0\le j<r\mathrm{.}$ Every $G_j$ is bounded and measurable because kernel integration preserves measurability. Applying [F2] at time $n_{r-1}$, multiplying by the bounded ${\mathcal{F}}_{n_{r-1}}$-measurable preceding product, and using [F3] removes $f_r(X_{n_r})$ and replaces it by $K^{n_r-n_{r-1}}{f}_r(X_{n_{r-1}})$. Repeating finitely many times gives $\mathbb{E}{\prod }_{j=0}^r{f}_j(X_{n_j})=\mathbb{E}[G_0(X_{n_0})]\mathrm{.}$

A final application of [F2] from time $0$ to time $n_0$, followed by [F1, F2, step 1.1] integration against [F1], gives $\mathbb{E}[G_0(X_{n_0})]=\int \mu (dx)\int K^{n_0}(x,d{x}_0)G_0(x_0),$ which expands to the displayed iterated integral. If $r=0$, this same line is the whole calculation; if $n_0=0$, the inner identity kernel simply evaluates $G_0(x)$. Choice is used in [F2]--[F3] and nowhere in the finite algebraic unwinding.

Put $f_j=1_{A_j}$. The left side is the joint law's value on the rectangle [F4, step 1.1, step 2.1] $A_0\times \cdots \times A_r$, and the right side is the announced cylinder integral. These rectangles include empty factors and the full rectangle, form a pi-system, and generate the finite product sigma-algebra. By [F4] their values determine the joint law uniquely. Conversely, the stated joint law integrates every bounded product test by the same iterated-integration calculation, so the two displayed formulations are equivalent.

Construct prefix probabilities recursively. Given ${\mu }_n$ on [F1] $E_0\times \cdots \times E_n$, define ${\mu }_{n+1}(A)=\int K_n(x_0,\dots ,x_n,A_{(x_0,\dots ,x_n)})d{\mu }_n\mathrm{.}$ The integrand is measurable by [F1]. For disjoint $A_j$, sectionwise countable additivity and [F8] move the increasing partial sums through the outer integral; the empty set has mass $0$ and the whole product has mass $1$. Thus ${\mu }_{n+1}$ is a probability measure. Since $K_n(x,E_{n+1})=1$, its marginal on the first $n+1$ coordinates is ${\mu }_n$. Induction gives consistent prefix laws and hence consistent laws for arbitrary finite coordinate sets by marginalization.

The spaces are nonempty along a compatible history: ${\mu }_0(E_0)=1$, and a [F2, step 1.1] probability section of each $K_n$ has nonempty target. Choice supplies one compatible infinite coordinate sequence. Together with the prefix construction in step 1.1, this shows the product is nonempty and [F2] defines a finitely additive probability $P_0$ on the cylinder algebra $\mathcal{A}$.

Starting from the cylinder law in step 2.1, we prove continuity at the empty set, the missing premeasure condition. Let [F1, F5, step 2.1] $C_j\downarrow \mathrm{\varnothing }$ be cylinders and suppose instead that $P_0(C_j)\downarrow a>0$. Represent $C_j$ using the first $r_j+1$ coordinates, enlarging so that $r_j$ increases. For a prefix $x^{(k)}=(x_0,\dots ,x_k)$ and $j$ with $r_j\ge k$, let $p_j^{(k)}(x^{(k)})$ be the probability, under the remaining kernels through time $r_j$, that the completed prefix lies in $C_j$. These functions are measurable by repeated [F1], lie in $[0,1]$, and decrease in $j$. Put $m_k={\lim }_j{p}_j^{(k)}$. Dominated convergence [F5] gives the recursion $m_k(x^{(k)})=\int m_{k+1}(x^{(k)},y)K_k(x^{(k)},dy),$ while another use of [F5] gives $a=\int m_0(x_0){\mu }_0(d{x}_0)$.

Since $a>0$, some $x_0$ has $m_0(x_0)>0$. Whenever [step 3.1] $m_k(x^{(k)})>0$, the recursion in step 3.1 implies that some $x_{k+1}$ has $m_{k+1}(x^{(k)},x_{k+1})>0$. Choice selects these coordinates recursively. For fixed $j$, once the selected prefix reaches $r_j$, monotonicity gives $0<m_{r_j}(x^{(r_j)})\le p_j^{(r_j)}(x^{(r_j)})=1_{C_j}(x^{(r_j)})\mathrm{.}$ Thus the selected infinite point belongs to every $C_j$, contradicting their empty intersection. Therefore $P_0(C_j)\downarrow 0$. This is the exact nonempty-choice use in the proof; no compactness or tail measure is assumed.

If disjoint cylinders $A_j$ have cylinder union $A$, then [F3, F4, step 4.1] $R_N=A\setminus {\bigcup }_{j<N}{A}_j$ is a decreasing cylinder sequence with empty intersection. Finite additivity and step 4.1 give $P_0(A)={\sum }_{j<N}{P}_0(A_j)+P_0(R_N)⟶{\sum }_j{P}_0(A_j)\mathrm{.}$ Hence, by [F3], $P_0$ is a finite premeasure. Since Choice implies countable choice, [F4] extends it to a probability $P$ on the product sigma-algebra.

If $P^{\mathrm{\prime }}$ is another extension, it agrees with $P$ on every cylinder by the [F6, step 5.1] prescribed finite laws. Cylinders form a pi-system containing the whole product and generate the product sigma-algebra, so [F6] gives $P^{\mathrm{\prime }}=P$. This also handles zero cylinder events and the total-mass-one cylinder.

Under the probability $P$ constructed and identified in step 6.1, in the homogeneous last-coordinate specialization, let [F7, step 1.1, step 6.1] ${\mathcal{F}}_n=\sigma (X_0,\dots ,X_n)$. The recursive prefix law in step 1.1 says for every $B\in {\mathcal{F}}_n$ and $A\in \mathcal{E}$ that ${\mathbb{E}}_P[1_B{1}_{\{X_{n+1}\in A\}}]={\mathbb{E}}_P[1_{B}K(X_n,A)]\mathrm{.}$ The right-hand random variable is ${\mathcal{F}}_n$-measurable, so it is the conditional probability. Hence [F7] says that the coordinates form the homogeneous Markov chain and they have initial law ${\mu }_0$.

In [F1], take $E_n=E$ for all $n$, initial law $\mu$, and [F1] $K_n(x_0,\dots ,x_n,A)=K(x_n,A)\mathrm{.}$ The last display is a probability kernel because it is the composition of the measurable last-coordinate projection with each measurable evaluation of $K$.

The theorem [F1] therefore gives a unique law on the product sigma-algebra and says that [F1, step 1.1] the coordinates are the required $(\mu ,K)$-chain. Choice is exactly the assumption of [F1]. The construction includes Dirac initial laws, one-point spaces, and $n=0$; an empty $E$ admits no probability $\mu$ and so cannot meet the hypotheses.

For every finite increasing time list, [F1] gives the same iterated [F1] integral for both processes because their $\mu$ and $K$ agree. This includes a single time, time zero, empty rectangle events, and the full rectangle. Therefore all their finite-dimensional distributions coincide.

Push both processes forward by their path maps. The two induced [F2, step 1.1] probabilities have the finite-dimensional distributions compared in step 1.1, so [F2] makes them equal on the cylinder sigma-algebra. Choice enters through [F1]'s conditional-expectation argument; [F2] adds no new selection.

Let $C=A_0\times \cdots \times A_r\times E\times E\times \cdots$ be a [F1, F2, F5] rectangular path cylinder. Backward kernel integration gives the measurable function $h_C(x)=1_{A_0}(x)K(1_{A_1}K(\cdots K{1}_{A_r}))(x)\mathrm{.}$ By [F5], it equals ${\mathbb{P}}_x(C)$ under the canonical chain started from $x$. Starting at time $n+r-1$ and applying [F1] backward $r$ times, with $1_{A_0}(X_n)$ and the already exposed factors left outside, gives $\mathbb{E}[1_C(X_n,X_{n+1},\dots )\mid {\mathcal{F}}_n]=h_C(X_n)\mathrm{.}$ The formula also covers $r=0$, an empty $A_j$, and all $A_j=E$.

Let $\mathcal{D}$ be the path events $D$ for which [F3, F4, step 1.1] $h_D(x)={\mathbb{P}}_x(D)$ is measurable and the conditional identity in step 1.1 holds with $D$. The whole path space belongs to $\mathcal{D}$, with $h=1$. Complements remain in $\mathcal{D}$ because $h_{D^c}=1-h_D$. For pairwise disjoint $D_j\in \mathcal{D}$, countable additivity gives $h_{\cup D_j}={\sum }_j{h}_{D_j}$; measurable partial sums increase to this function, and dominated convergence in the defining event integrals gives the conditional identity for the union. Hence $\mathcal{D}$ is a lambda-system. Rectangular cylinders form a pi-system and belong by step 1.1, so [F3] yields every product-measurable path event.

Finite real linear combinations of event indicators now satisfy both [F4, step 2.1] measurability and the identity. If $0\le H\le M$, choose simple $H_j\uparrow H$ by [F4]. Then $h_j(x)={\mathbb{E}}_x{H}_j\to {\mathbb{E}}_{x}H=h(x)$ by dominated convergence, making $h$ measurable. The same theorem passes the limit through all event tests for conditional expectation and proves the displayed identity. Apply this to the positive and negative parts of a general bounded real $H$ and subtract. Zero, one, constant, and degenerate one-point path functionals are included. Choice enters through the canonical laws ${\mathbb{P}}_x$ and conditional-expectation versions used by [F1].

Assume (1), take bounded $U$ measurable for ${\mathcal{F}}_n$ and bounded $V$ [F1] measurable for ${\mathcal{T}}_n^{+}$, and put $W=\mathbb{E}[V\mid \sigma (X_n)]$. Conditioning first on ${\mathcal{F}}_n$ gives $\mathbb{E}[UV\mid \sigma (X_n)]=\mathbb{E}[U\mathbb{E}(V\mid {\mathcal{F}}_n)\mid \sigma (X_n)]=\mathbb{E}[UW\mid \sigma (X_n)]=\mathbb{E}[U\mid \sigma (X_n)]W\mathrm{.}$ This is the sigma-algebra form of conditional independence in [F1], so (2) holds. Constants, zero, and one cause no exception.

Conversely assume (2) and keep $V,W$ as above. For every [F1] $A\in {\mathcal{F}}_n$, the conditional product identity gives $\mathbb{E}[1_{A}V]=\mathbb{E}[\mathbb{E}(1_{A}V\mid \sigma (X_n))]=\mathbb{E}[\mathbb{E}(1_A\mid \sigma (X_n))W]=\mathbb{E}[1_{A}W]\mathrm{.}$ Since $W$ is ${\mathcal{F}}_n$-measurable, this is exactly the defining event test for $W=\mathbb{E}[V\mid {\mathcal{F}}_n]$. Hence (1). Empty and full $A$ are included.

If $X$ is a homogeneous $K$-chain, apply [F2] to every bounded measurable [F2, step 1.1, step 1.2] path functional $H$ and $V=H(X_n,X_{n+1},\dots )$. Such variables generate the bounded ${\mathcal{T}}_n^{+}$-measurable variables by the event/simple-function argument in [F2], and [F2] gives a $\sigma (X_n)$-measurable version $h(X_n)$. Thus (1), and hence (2), holds.

For the converse qualification, take $V=g(X_{n+1})$ in (1). Combining (1) [step 1.1, step 1.2] with the stated present-state kernel identity gives $\mathbb{E}[g(X_{n+1})\mid {\mathcal{F}}_n]=Kg(X_n)\mathrm{.}$ Indicators recover the $K$-chain definition. Without the single-$K$ hypothesis, conditional independence alone allows time-inhomogeneous present-state kernels, so it would not justify the stronger homogeneous conclusion. Choice is used by the conditional-expectation interfaces throughout.

Since $h$ is measurable, $(h(X_n))$ is adapted. Applying [F3] with value [F1, F3] zero at infinity shows that $R_h$ is ${\mathcal{F}}_{\tau }$-measurable. Moreover $\mathrm{\mid }{Z}_H\mathrm{\mid },\mathrm{\mid }{R}_h\mathrm{\mid }\le \parallel H{\parallel }_{\mathrm{\infty }}$, so both variables are integrable. This also covers $H=0$, constant $H=1$, and the event $\{\tau =\mathrm{\infty }\}$, where both variables vanish by definition.

Fix $A\in {\mathcal{F}}_{\tau }$. For every finite $n$, [F2] and [F1] give [F1, F2, F4, step 1.1] $\begin{array}{rl}{\displaystyle }& {\displaystyle \mathbb{E}[1_{A\cap \{\tau =n\}}H(X_n,X_{n+1},\dots )]}\\ {\displaystyle }& {\displaystyle =\mathbb{E}[1_{A\cap \{\tau =n\}}h(X_n)]\mathrm{.}}\end{array}$ Sum from $n=0$ to $N$. The partial sums on either side are bounded in absolute value by $\parallel H{\parallel }_{\mathrm{\infty }}$ and converge pointwise to $1_A{Z}_H$ and $1_A{R}_h$. By [F4], letting $N\to \mathrm{\infty }$ gives $\mathbb{E}[1_A{Z}_H]=\mathbb{E}[1_A{R}_h]$. Together with step 1.1 this is the defining event test for the displayed conditional expectation. Empty $A$, full $A$, $\tau =0$, and an almost-surely infinite $\tau$ require no separate argument.

If $\mathbb{P}(\tau <\mathrm{\infty })=1$, the exceptional infinity event is null, so [step 2.1] $Z_H=H(X_{\tau },X_{\tau +1},\dots )$ and $R_h=h(X_{\tau })$ almost surely for any arbitrary values assigned there. This proves the finite form. Choice is used only by [F1] and the conditional expectation in the conclusion.