Exact Newton-Kleinman iteration preserves stability, decreases cost matrices, and converges quadratically locally

Suppose $A_j$ is Hurwitz. By [F3], the exact solution $X_j$ of the policy Lyapunov equation is $X_j={\int }_0^{\mathrm{\infty }}{e}^{t{A}_j^{\ast }}(Q+K_j^{\ast }R{K}_j)e^{t{A}_j}dt⪰0.$ For ${\mathrm{\Delta }}_j=K_{j+1}-K_j$, expansion using $B^{\ast }{X}_j=R{K}_{j+1}$ gives $A_{j+1}^{\ast }{X}_j+X_j{A}_{j+1}+Q+K_{j+1}^{\ast }R{K}_{j+1}=-{\mathrm{\Delta }}_j^{\ast }R{\mathrm{\Delta }}_j\mathrm{.}$ If $A_{j+1}v=\lambda v$ with $\mathrm{Re}\lambda \ge 0$, the quadratic form of the left side is $2\mathrm{Re}(\lambda )v^{\ast }{X}_{j}v+v^{\ast }Qv+(K_{j+1}v{)}^{\ast }R(K_{j+1}v)\ge 0,$ while the right side is nonpositive. Equality forces $K_{j+1}v=0$ and ${\mathrm{\Delta }}_{j}v=0$, hence $K_{j}v=0$ and $A_{j}v=Av=\lambda v$, contradicting that $A_j$ is Hurwitz. Starting from $K_0$, induction proves every $A_j$ Hurwitz and every $X_j⪰0$.

Subtracting the Lyapunov equations for $X_j$ and $X_{j+1}$ after expressing both under $A_{j+1}$ gives $A_{j+1}^{\ast }(X_j-X_{j+1})+(X_j-X_{j+1})A_{j+1}=-{\mathrm{\Delta }}_j^{\ast }R{\mathrm{\Delta }}_j\mathrm{.}$ The Lyapunov integral [F3] therefore yields $X_j-X_{j+1}⪰0$. The decreasing sequence is bounded below by zero, so each quadratic form converges. Evaluating at $e_i$, $e_i+e_j$, and $e_i+\mathrm{i}{e}_j$ shows convergence of every matrix entry to a Hermitian limit $X_{\mathrm{\infty }}⪰0$. Since $K_{j+1}=R^{-1}{B}^{\ast }{X}_j$, the gains also converge. Passing to the limit in the policy Lyapunov equation gives the CARE for $X_{\mathrm{\infty }}$; [F2] identifies this positive-semidefinite root as $X_{\ast }$.

Let $E_j=X_j-X_{\ast }$ and $A_{\ast }=A-B{R}^{-1}{B}^{\ast }{X}_{\ast }$. Put $D_j=K_{j+1}-K_{\ast }=R^{-1}{B}^{\ast }{E}_j$. Subtract the CARE identity at $X_{\ast }$ from the policy Lyapunov equation under $A_{j+1}$ to obtain $A_{j+1}^{\ast }{E}_{j+1}+E_{j+1}{A}_{j+1}=-D_j^{\ast }R{D}_j\mathrm{.}$ By [F2], $A_{\ast }$ is Hurwitz. Choose $W={\int }_0^{\mathrm{\infty }}{e}^{t{A}_{\ast }^{\ast }}I{e}^{t{A}_{\ast }}dt\succ 0$ using [F3], so $A_{\ast }^{\ast }W+W{A}_{\ast }=-I$. Since $A_{j+1}\to A_{\ast }$, for all large $j$ one has $A_{j+1}^{\ast }W+W{A}_{j+1}⪯-\frac{1}{2}I$. Along $\dot{x}=A_{j+1}x$, differentiation gives $\frac{d}{dt}(x^{\ast }Wx)\le -\frac{1}{2}\mathrm{\parallel }x{\mathrm{\parallel }}_2^2\le -\frac{1}{2\mathrm{\parallel }W{\mathrm{\parallel }}_2}{x}^{\ast }Wx\mathrm{.}$ Consequently $\mathrm{\parallel }{e}^{t{A}_{j+1}}{\mathrm{\parallel }}_2\le M{e}^{-\alpha t}$ with $M=(\mathrm{\parallel }W{\mathrm{\parallel }}_2\mathrm{/}{\lambda }_{\min }(W){)}^{1\mathrm{/}2}$ and $\alpha =1\mathrm{/}(4\mathrm{\parallel }W{\mathrm{\parallel }}_2)$, uniformly for all sufficiently large $j$. Applying the Lyapunov integral to the error equation yields $\mathrm{\parallel }{E}_{j+1}{\mathrm{\parallel }}_F\le \left({\int }_0^{\mathrm{\infty }}{M}^2{e}^{-2\alpha t}dt\right)\mathrm{\parallel }R{\mathrm{\parallel }}_2\mathrm{\parallel }{R}^{-1}{B}^{\ast }{\mathrm{\parallel }}_2^2\mathrm{\parallel }{E}_j{\mathrm{\parallel }}_F^2\mathrm{.}$ The integral is finite and supplies the stated local quadratic constant.