How statement and proof provenance work
The first chip identifies the source of the statement or construction; the second identifies the source of its local proof or verification.
- Literature-sourced: the exact statement appears in a cited source; only wording and notation differ.
- AI-adapted: a semantically identical restatement of literature-sourced material, modulo indexing, notation, and boundary cases adopted by the library.
- AI-generated: a genuinely novel statement formulated by AI, with no source for the claim itself.
These labels describe origin, not correctness: citations and verification chips remain separate evidence.
The sample CPO update has a well-defined proposal and finite backtracking search
Statement
Under the inputs and assumptions of the sample-based constrained policy
optimization update, with finite parameter dimension and nonnegative
integer backtracking budget , the update has a unique mathematical output,
either no-step or an accepted parameter. Its proposals satisfy these properties.
If its matrix is symmetric positive definite and its local quadratic program is feasible, the program has a nonempty compact global-maximizer set and a unique lexicographically least global maximizer. This is the proposal selected by the update and satisfies every linearized cost constraint and the quadratic trust-region constraint.
If the program is infeasible with one cost and , the inverse used
by the update exists, , and its recovery proposal is
It is the unique minimizer of on ,
satisfies , and decreases the linearized cost by
. It need not make the infeasible cost constraint feasible.
The non-positive-definite, zero-recovery-gradient and multiple-cost infeasible
branches return no-step as specified.
Once the specified exact subproblem decisions and operations have been
performed, backtracking terminates after at most candidate tests.
It returns the first passing candidate, which belongs to the parameter
domain and satisfies the empirical KL and all empirical cost tests, or
returns no-step if none passes. These are well-definedness and finite-search
properties of the exact mathematical update. They do not bound the cost of
exact feasibility/optimization operations or certify the true expected costs,
exact population KL, policy improvement, or convergence of sampled iterates.
Statement references
Facts & Assumptions
Given: The complete inputs and hypotheses of the algorithm, its real sample coefficients , radius , factor , and finite integer budget . The algorithm's Axiom of Choice hypothesis for its theoretical KL comparison is retained; the finite-dimensional proposal argument below makes no additional use of choice.
Sample-based constrained policy optimization update specifies all proposal branches, the lexicographic tie rule and the ordered sample-test search.
A definite quadratic form has a uniform signed bound on the Euclidean unit sphere gives with on the unit sphere for a positive definite quadratic form in positive dimension.
Heine-Borel in : with the Euclidean metric a subset of is compact if and only if it is closed and bounded, and the proof by bisection uses no choice principle; the same holds on the real line makes a closed bounded Euclidean set compact, and A closed subset of a compact metric space is compact preserves compactness under closed restriction.
A continuous real-valued function on a nonempty compact metric space is bounded and attains a greatest and a least value gives attained maxima and minima for a continuous real function on a nonempty compact metric space.
Invertible matrix theorem: invertibility, full pivot rank, RREF , trivial nullspace and unique solvability are equivalent makes a square real matrix with trivial nullspace invertible.
Cauchy-Schwarz with its equality case, the triangle inequality for , the parallelogram law and polarisation and The Euclidean inner product on give Euclidean Cauchy–Schwarz, the triangle inequality, coordinate bounds and bilinear norm identities.
Proof
First consider and symmetric positive definite . By homogeneity, [F2] gives for all , with zero handled directly. Put . Coordinate bounds in [F6] and the identity give , so the quadratic form is continuous. The ellipsoid is therefore closed and contained in the ball of radius , hence compact by [F3]. Each linear functional is continuous because . Thus the feasible set is a closed subset of , and is compact.
If is nonempty, [F4] gives an attained maximum of on . Its maximizer set is nonempty and compact by closedness and [F3]. Successively for coordinates , let be the minimum of coordinate on and set . Each minimum exists by [F4], and each restriction remains nonempty and compact by [F3]. After restrictions all coordinates are fixed, so is a singleton. For any other maximizer, the first coordinate where it differs from this singleton must be greater, by the construction of that coordinate's minimum. The singleton is therefore exactly the unique lexicographically least maximizer. As a member of , it satisfies all stated constraints.
If , then and positive definiteness forces . Thus [F5] supplies . In the single-cost recovery branch write , . Then , and . For every , nonnegativity of the quadratic form at gives , using symmetry. Consequently . For , direct substitution gives and , attaining the bound. If another point attains it, equality in the preceding inequalities gives , hence by positive definiteness. The recovery proposal is therefore unique, and its linearized cost is . Since it is the minimum over , infeasibility implies that this last value is strictly positive, rather than implying feasibility.
The remaining proposal branches are the explicit returns in [F1]: non-positive-definite , infeasible single-cost problem with , and infeasible problem with all return no-step. If , the only parameter displacement is the zero vector, so a feasible program has that unique maximizer, and in the infeasible case all and one of the specified no-step branches applies. Thus zero dimension introduces no missing proposal case. Together with steps 2.1 and 2.2, every branch either gives one uniquely specified proposal or returns no-step.
For a proposal , the candidates for integer form a prescribed ordered list of points. The parameter-domain check precedes evaluation of the policy tests, so no value outside the domain is required. On the domain, the finite common-support policy probabilities define the empirical KL and surrogates in [F1]. Either there is a least passing index in this finite list, or there is none; [F1] returns its candidate or no-step, respectively. Inspecting successive indices therefore terminates in at most candidate tests. The definition of passing supplies exactly the stated sample inequalities. All choices and returns are single-valued by steps 2.1, 2.2 and 3.1 and this least-index rule, establishing the asserted output and finite-search properties. No step in this argument compares the sample estimates to population expectations.
Remarks
The source's practical approximate CPO implementation motivates the quadratic program and the recovery direction. This lemma proves properties of the library's explicitly specified exact finite-data update, including its local lexicographic tie rule and finite search budget. It does not attribute those additional conventions or their proof to the paper.
Depends on
- Sample-based constrained policy optimization update
- A definite quadratic form has a uniform signed bound on the Euclidean unit sphere
- Heine-Borel in $\mathbb{R}^n$: with the Euclidean metric a subset of $\mathbb{R}^n$ is compact if and only if it is closed and bounded, and the proof by bisection uses no choice principle; the same holds on the real line
- A continuous real-valued function on a nonempty compact metric space is bounded and attains a greatest and a least value
- A closed subset of a compact metric space is compact
- Invertible matrix theorem: invertibility, full pivot rank, RREF $I$, trivial nullspace and unique solvability are equivalent
- Cauchy-Schwarz $\lvert\langle x,y\rangle\rvert \le \lVert x\rVert_2\lVert y\rVert_2$ with its equality case, the triangle inequality for $\lVert\cdot\rVert_2$, the parallelogram law and polarisation
- The Euclidean inner product $\langle x,y\rangle = \sum_{k<n} x_k y_k$ on $\mathbb{R}^n$
Used by
Nothing in the library uses this result yet.
Cited to discharge well-definedness by Sample-based constrained policy optimization update.
Dependency tree · two levels
76 results within two dependency steps of this one, each drawn at its shortest distance from it. An arrow runs from a result to what uses it, so the chart reads left to right and ends at this result, which carries a heavier outline. Every node is a link to that result. Click elsewhere on the chart to enlarge it.
Sources
- Achiam et al., Constrained Policy Optimization, Sections 6.1–6.3 (standard reference, not scraped)