LemmaStatement: Literature-sourcedProof: AI-generatedPipeline-generatedjudge pass (gpt-6-luna)

How statement and proof provenance work

The first chip identifies the source of the statement or construction; the second identifies the source of its local proof or verification.

Literature-sourced: the exact statement appears in a cited source; only wording and notation differ.
AI-adapted: a semantically identical restatement of literature-sourced material, modulo indexing, notation, and boundary cases adopted by the library.
AI-generated: a genuinely novel statement formulated by AI, with no source for the claim itself.

These labels describe origin, not correctness: citations and verification chips remain separate evidence.

The weighted feature projection is well-defined

Statement

Let $S$ be a nonempty finite state set of a finite Markov decision process, let $0 \leq γ < 1$ , and let $π$ be a stationary Markov policy. Define the policy Bellman operator by

(T^{π} f) (s) = \sum_{a \in A (s)} π (a ∣ s) [r (s, a) + γ \sum_{s^{'} \in S} P (s^{'} ∣ s, a) f (s^{'})] .

Let $μ : S \to [0, 1]$ satisfy $\sum_{s \in S} μ (s) = 1$ , and let $ϕ_{1}, \dots, ϕ_{d} : S \to R$ be a finite list of feature functions. Define $f \sim_{μ} g$ when $\sum_{s \in S} μ (s) (f (s) - g (s))^{2} = 0$ , let $H_{μ}$ be the quotient of $R^{S}$ by this relation, and define $⟨ [f]_{μ}, [g]_{μ} ⟩_{μ} = \sum_{s \in S} μ (s) f (s) g (s)$ . Then this quotient is a finite-dimensional real inner-product space. The span $V_{μ}$ of $[ϕ_{1}]_{μ}, \dots, [ϕ_{d}]_{μ}$ is finite-dimensional, and every $h \in H_{μ}$ has a unique orthogonal projection $Π_{μ} h \in V_{μ}$ . Here orthogonal means that $h - Π_{μ} h$ has inner product zero with every vector in $V_{μ}$ . For each fixed $w \in R^{d}$ , put $v_{w} (s) = \sum_{j = 1}^{d} ϕ_{j} (s) w_{j}$ and $δ_{w} = [T^{π} v_{w} - v_{w}]_{μ}$ . Then $∥ δ_{w} ∥_{μ}^{2}$ and $∥ Π_{μ} δ_{w} ∥_{μ}^{2}$ are well-defined real numbers, including when some state weights vanish or the features are linearly dependent.

Statement references

The finite transition/reward model and stationary Markov policies.

The expected discounted policy Bellman operator.

Finite indexed sums over the state set.

Every vector in a real inner-product space has a unique orthogonal projection onto a finite-dimensional subspace.

Finite sums of nonnegative terms vanish only when each term vanishes.

Sutton and Barto, Reinforcement Learning: An Introduction, 2nd ed., §11.4.

Facts & Assumptions

Given: A nonempty finite set $S$ , probability weights $μ$ on $S$ , and finitely many real-valued feature functions on $S$ .

[F1]

The sum $\sum_{i \in S} a_{i}$ over a finite index set, and its product form: a sum over a finite set is defined by enumerating its elements through a bijection with an initial segment of the natural numbers, with the value independent of that enumeration.

[F2]

Laws of finite sums and finite products: a finite sum of nonnegative real numbers is zero only if every summand is zero; finite sums also preserve addition and scalar multiplication.

[F3]

Projection onto a finite-dimensional subspace of an inner-product space: in a real inner-product space, every vector has a unique orthogonal projection onto any finite-dimensional subspace, characterized by the residual being orthogonal to that subspace.

Proof

1.1givenF1F2algebra

Put $S_{+} = {s \in S : μ (s) > 0}$ . By [F1] and [F2], $f \sim_{μ} g$ holds exactly when $f (s) = g (s)$ for every $s \in S_{+}$ . This is an equivalence relation, and equality on $S_{+}$ is preserved under pointwise addition and scalar multiplication, so the quotient inherits the real vector-space operations. If representatives change within their equivalence classes, every summand with positive weight stays unchanged and every summand with zero weight remains zero; therefore the displayed inner product is independent of representatives. It is symmetric and bilinear by finite-sum rules, and $⟨ [f]_{μ}, [f]_{μ} ⟩_{μ} = 0$ implies $f = 0$ on $S_{+}$ , so $[f]_{μ} = 0$ . The classes of the finitely many coordinate indicator functions span $H_{μ}$ , hence this is a finite-dimensional real inner-product space.

2.1givenF3step 1.1∎

The subspace $V_{μ}$ is spanned by the finite list of feature classes, so it is finite-dimensional. Apply [F3] to $h \in H_{μ}$ and $V_{μ}$ to obtain a unique vector $Π_{μ} h$ whose residual is orthogonal to $V_{μ}$ . For each fixed $w$ , the Bellman residual is a function on $S$ , so it determines the class $δ_{w}$ ; its norm and the norm of its unique projection are finite real numbers. This conclusion does not identify parameters whose feature functions agree only on the support of $μ$ , since the Bellman operator can depend on values outside that support.

Depends on

Used by

State-weighted Bellman error and projected Bellman error Definition

Dependency tree · two levels

36 results within two dependency steps of this one, each drawn at its shortest distance from it. An arrow runs from a result to what uses it, so the chart reads left to right and ends at this result, which carries a heavier outline. Every node is a link to that result. Click elsewhere on the chart to enlarge it.

Sources

Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, 2nd ed. (standard reference, not scraped)