Finite basis projector energies and variational electronic structure

Write $A=R+iS$ with $R,S$ real. Hermitian symmetry gives $R^{\mathsf{T}}=R$ and $S^{\mathsf{T}}=-S$. Define $\mathcal{R}(A)=\left(\begin{array}{cc}R& -S\\ S& R\end{array}\right)$ and $\mathrm{\Phi }(x+iy)=(x,y)$. Direct multiplication gives $\mathrm{\Phi }(Az)=\mathcal{R}(A)\mathrm{\Phi }(z)$ and $\mathrm{\parallel }\mathrm{\Phi }(z){\mathrm{\parallel }}_2=\mathrm{\parallel }z{\mathrm{\parallel }}_2$; the transpose identities show $\mathcal{R}(A)$ is real symmetric. Applying the same construction to $A-B$ yields $\mathrm{\parallel }\mathcal{R}(A)-\mathcal{R}(B){\mathrm{\parallel }}_2=\mathrm{\parallel }A-B{\mathrm{\parallel }}_2$ by the supremum definition in [F4].

By [F1], $A=U\mathrm{diag}({\lambda }_1(A),\dots ,{\lambda }_n(A))U^{\ast }$ for a unitary $U$. Its realification is orthogonal, and the realification of the diagonal matrix has each ${\lambda }_j(A)$ twice on its diagonal. Therefore the descending eigenvalues of $\mathcal{R}(A)$ are ${\lambda }_1(A),{\lambda }_1(A),\dots ,{\lambda }_n(A),{\lambda }_n(A)$; the same holds for $B$.

Put $M=\mathcal{R}(B)$, $N=\mathcal{R}(A-B)$, and $\delta =\mathrm{\parallel }A-B{\mathrm{\parallel }}_2$. By [F3] and step 1.1, every eigenvalue of the real symmetric $N$ lies in $[-\delta ,\delta ]$ using [F4]. Applying [F2] with the perturbation's largest eigenvalue for the upper bound and smallest eigenvalue for the lower bound gives $\mathrm{\mid }{\lambda }_k(M+N)-{\lambda }_k(M)\mathrm{\mid }\le \delta$ for every $k$. Taking $k=2j-1$ and using step 2.1 proves the claimed complex-Hermitian eigenvalue bound.

Each $m_j$ exists by [F1]. If $j\le k$, then $K_j\subseteq K_k$, so the minimum over the larger set satisfies $m_k\le m_j$. Thus $(m_j)$ is nonincreasing. Let $m:=\inf \{m_j:j\ge 1\}$ in the extended-real convention of [F2]. If the set of values is bounded below, then for every $\epsilon >0$ some $j_0$ has $m\le m_{j_0}<m+\epsilon$ by the greatest-lower-bound property; for $j\ge j_0$, monotonicity gives $m\le m_j\le m_{j_0}<m+\epsilon$, so $m_j\to m$. If the set is unbounded below, for every real $R$ some $j_0$ has $m_{j_0}<R$, and monotonicity gives $m_j<R$ for every $j\ge j_0$, so $m_j\to -\mathrm{\infty }$. Thus the extended-real limit exists and equals $m$.

Set $a={\inf }_{P\in {\cup }_j{K}_j}E(P)$ in the extended-real convention of [F2]. Since each minimum is attained by [F1], every $m_j$ is a value of $E$ on the union, so $a$ is a lower bound for $\{m_j:j\ge 1\}$ and [F2] gives $a\le m$. For every $P\in {\cup }_j{K}_j$, choose an index $j_0$ with $P\in K_{j_0}$. For all $j\ge j_0$, $m_j\le E(P)$; since $m$ is a lower bound for all $m_j$, this gives $m\le E(P)$. Thus $m$ is an extended-real lower bound for all values of $E$ on the union, so $m\le a$ by [F2]. Therefore $m=a$.

Under the additional recovery hypothesis, fix $P\in K$. If $a=-\mathrm{\infty }$, then $a\le E(P)$. Otherwise, for every $\epsilon >0$, the convergence $E(P_j)\to E(P)$ gives an index $j$ with $E(P_j)<E(P)+\epsilon$. Since $a\le E(P_j)$, we have $a<E(P)+\epsilon$ for every $\epsilon >0$, hence $a\le E(P)$. Taking the extended-real infimum over $P\in K$ gives $a\le {\inf }_{K}E$. The reverse inequality follows from ${\cup }_j{K}_j\subseteq K$, so the two extended-real infima are equal.

By [F1], choose an orthonormal eigenbasis for $P$ with eigenvalues ${\lambda }_1,\dots ,{\lambda }_n$. Idempotence gives ${\lambda }_j^2={\lambda }_j$, so each ${\lambda }_j$ is $0$ or $1$.

Let $U$ be the unitary matrix whose columns are the eigenbasis from step 1.1, so $U^{\ast }PU=\mathrm{diag}({\lambda }_1,\dots ,{\lambda }_n)$. Since $U$ is unitary, $U^{\ast }=U^{-1}$ and [F3] gives $\mathrm{tr}P=\mathrm{tr}(U^{\ast }PU)={\sum }_j{\lambda }_j$, where the last equality is the diagonal definition of trace. The image is spanned exactly by the eigenvectors with eigenvalue $1$, so by [F2] its dimension, and thus the rank, is the number of indices with ${\lambda }_j=1$. Therefore trace and rank agree.

The matrix $D_{\ast }=(r\mathrm{/}n)I$ has eigenvalues $r\mathrm{/}n\in (0,1)$ and trace $r$, so [F1] gives $D_{\ast }\in {\mathcal{D}}_r$. Thus the domain is nonempty.

The conditions $D=D^{\ast }$, $D⪰0$, $I-D⪰0$, and $\mathrm{tr}D=r$ define a closed subset of the real coordinates of ${\mathbb{H}}_n$: the first and last are continuous equalities, while positive semidefiniteness is closed because it is the intersection of the closed inequalities $⟨Dx,x⟩\ge 0$ over all fixed vectors $x$.

If $d_1,\dots ,d_n\in [0,1]$ are the eigenvalues of $D\in {\mathcal{D}}_r$, then $\mathrm{\parallel }D{\mathrm{\parallel }}_F^2=\mathrm{tr}(D^2)={\sum }_j{d}_j^2\le {\sum }_j{d}_j=r$. Thus ${\mathcal{D}}_r$ is bounded.

The real vector space ${\mathbb{H}}_n$ has dimension $n^2$ and is isomorphic to ${\mathbb{R}}^{n^2}$ by its real diagonal entries and real and imaginary off-diagonal coordinates. Closedness from step 1.2 and boundedness from step 1.3 give compactness by [F3].

For $0<x<1$, put $t=1\mathrm{/}x$. By [F3], $0\le \log t={\int }_1^{t}ds\mathrm{/}s\le {\int }_1^{t}ds\mathrm{/}\sqrt{s}=2(\sqrt{t}-1)$, so $\mathrm{\mid }x\log x\mathrm{\mid }=(\log t)\mathrm{/}t\le 2\mathrm{/}\sqrt{t}\to 0$. The other term $(1-x)\log (1-x)$ tends to $0$ as $x\downarrow 0$ by continuity of $y\mapsto y\log y$ at $y=1$. Since $\varphi (1-x)=\varphi (x)$, the same limits hold at $1$. Thus the stipulated endpoint values make $\varphi$ continuous on $[0,1]$.

By [F2], if $D_k\to D$ in Frobenius norm, then every ordered eigenvalue $d_{k,j}$ tends to the corresponding $d_j$; indeed [F4] gives $\mathrm{\parallel }{D}_k-D{\mathrm{\parallel }}_2\le \mathrm{\parallel }{D}_k-D{\mathrm{\parallel }}_F\to 0$. All these eigenvalues lie in $[0,1]$ by the domain definition.

Spectral functional calculus gives $\mathrm{tr}\varphi (D_k)={\sum }_{j=1}^n\varphi (d_{k,j})$ and likewise for $D$. Scalar continuity from step 1.1 makes each of the finitely many summands converge, so $\mathrm{tr}\varphi (D_k)\to \mathrm{tr}\varphi (D)$.

For $x>0$, [F1] gives ${\int }_0^R((1+s{)}^{-1}-(x+s{)}^{-1})ds=\log (1+R)-\log (x+R)+\log x$. The difference of the first two terms tends to zero: it is ${\int }_{x+R}^{1+R}dt\mathrm{/}t$, whose absolute value is at most $\mathrm{\mid }1-x\mathrm{\mid }\mathrm{/}(R+\min \{1,x\})$. Thus $\log x={\int }_0^{\mathrm{\infty }}((1+s{)}^{-1}-(x+s{)}^{-1})ds$. Diagonalizing a positive-definite Hermitian $X$ by [F4] gives the same identity for $\log X$. If $X⪰mI$ for some $m>0$, the resolvent $R_s=(X+sI{)}^{-1}$ has norm at most $(m+s{)}^{-1}$.

Let $H$ be small enough that $\mathrm{\parallel }H{\mathrm{\parallel }}_2<m\mathrm{/}2$, so $X+H⪰(m\mathrm{/}2)I$. For each $s\ge 0$, the resolvent identity [F2] gives $R_s(X+H)-R_s(X)=-R_s(X+H)H{R}_s(X)$. The induced operator norm is submultiplicative: for any matrices $A,B$ and unit vector $x$, $\mathrm{\parallel }ABx{\mathrm{\parallel }}_2\le \mathrm{\parallel }A{\mathrm{\parallel }}_2\mathrm{\parallel }Bx{\mathrm{\parallel }}_2\le \mathrm{\parallel }A{\mathrm{\parallel }}_2\mathrm{\parallel }B{\mathrm{\parallel }}_2$, hence $\mathrm{\parallel }AB{\mathrm{\parallel }}_2\le \mathrm{\parallel }A{\mathrm{\parallel }}_2\mathrm{\parallel }B{\mathrm{\parallel }}_2$ by [F6]. Using the integral representation from step 1.1, the remainder after the candidate linear term is $\log (X+H)-\log X-{\int }_0^{\mathrm{\infty }}{R}_s(X)H{R}_s(X)ds=-{\int }_0^{\mathrm{\infty }}{R}_s(X+H)H{R}_s(X)H{R}_s(X)ds\mathrm{.}$ By step 1.1, $\mathrm{\parallel }{R}_s(X+H){\mathrm{\parallel }}_2\le (m\mathrm{/}2+s{)}^{-1}$ and $\mathrm{\parallel }{R}_s(X){\mathrm{\parallel }}_2\le (m+s{)}^{-1}$. Therefore the remainder has norm at most $\mathrm{\parallel }H{\mathrm{\parallel }}_2^2{\int }_0^{\mathrm{\infty }}\frac{ds}{(m\mathrm{/}2+s)(m+s{)}^2}\le \frac{2\mathrm{\parallel }H{\mathrm{\parallel }}_2^2}{m^2}=o(\mathrm{\parallel }H{\mathrm{\parallel }}_2)\mathrm{.}$ This proves the real Fr'echet derivative formula $D\log (X)[H]={\int }_0^{\mathrm{\infty }}{R}_{s}H{R}_{s}ds$.

By the product rule [F5] and trace differential [F3], $D\mathrm{tr}(X\log X)[H]=\mathrm{tr}(H\log X)+{\int }_0^{\mathrm{\infty }}\mathrm{tr}(X{R}_{s}H{R}_s)ds$. The trace identity $\mathrm{tr}(AB)=\mathrm{tr}(BA)$ follows by expanding the diagonal sums. Since $X$ commutes with $R_s$, it rewrites the integral as $\mathrm{tr}(H{\int }_0^{\mathrm{\infty }}X{R}_s^{2}ds)$. Diagonalizing $X$ and using ${\int }_0^{\mathrm{\infty }}x(x+s{)}^{-2}ds=1$ for each eigenvalue $x>0$ makes the matrix integral $I$. Thus $D\mathrm{tr}(X\log X)[H]=\mathrm{tr}(H(\log X+I))$.

Apply step 3.1 first with $X=D$ and direction $H$, then with $X=I-D$ and direction $-H$. The identity terms cancel, leaving $D\mathcal{S}(D)[H]=\mathrm{tr}(H(\log D-\log (I-D)))$.

Since $P_0^2=P_0$, every $x\in {\mathbb{C}}^n$ decomposes as $P_{0}x+(I-P_0)x$ with the first term in $U$ and the second in $\ker P_0$ by [F4]. If $u=P_{0}y\in U$ and $z\in \ker P_0$, then $⟨u,z⟩=⟨P_{0}y,z⟩=⟨y,P_{0}z⟩=0$ by self-adjointness. Hence $\ker P_0=U^{\perp }$, giving the stated orthogonal block decomposition.

For any $X:U\to U^{\perp }$, $J_X^{\ast }{J}_X=I_U+X^{\ast }X$ is positive definite because $⟨(I_U+X^{\ast }X)u,u⟩=\mathrm{\parallel }u{\mathrm{\parallel }}^2+\mathrm{\parallel }Xu{\mathrm{\parallel }}^2>0$ for $u\mathrm{\ne }0$. The displayed $P(X)$ is Hermitian, and $P(X{)}^2=J_X(J_X^{\ast }{J}_X{)}^{-1}{J}_X^{\ast }{J}_X(J_X^{\ast }{J}_X{)}^{-1}{J}_X^{\ast }=P(X)$. Its range is $\mathrm{range}{J}_X=V_X$, of dimension $r$; [F2] gives $\mathrm{tr}P(X)=r$.

Let $T={\pi }_U{\mathrm{\mid }}_V:V\to U$, whose adjoint is $T^{\ast }=P_V{\mathrm{\mid }}_U:U\to V$. The block $A={\pi }_U{P}_V{\mathrm{\mid }}_U$ is $T{T}^{\ast }$, so it is invertible exactly when $T$ is invertible: $T{T}^{\ast }$ is positive definite exactly when $T^{\ast }$ is injective, and the domain and codomain of $T$ have the same finite dimension. Thus the chart condition is the open condition that $A$ be invertible; it is open because $P_V\mapsto A$ and determinant are continuous, and at $P_0$ we have $A=I_U$. In that domain, projection from $V$ to $U$ is bijective, so every $u\in U$ has a unique vector $u+Xu\in V$; this defines a unique linear $X$. In blocks $P_V=\left(\begin{array}{cc}A& B^{\ast }\\ B& C\end{array}\right)$, that map is $X=B{A}^{-1}$. The formula in step 1.2 is the unique orthogonal projector onto this graph. Matrix inversion is smooth on invertible matrices by [F5], so the graph map and $P(X)$ are inverse smooth coordinates.

Let $P(t)$ be a differentiable curve in ${\mathcal{P}}_r$ with $P(0)=P_0$ and derivative $H$. Differentiating $P(t{)}^{\ast }=P(t)$ and $P(t{)}^2=P(t)$ gives $H=H^{\ast }$ and $P_{0}H+H{P}_0=H$. In block form these equations force $H=\left(\begin{array}{cc}0& B^{\ast }\\ B& 0\end{array}\right)$. Conversely, for any such $B$, the graph curve $P(tB)$ from step 1.2 has derivative $\left(\begin{array}{cc}0& B^{\ast }\\ B& 0\end{array}\right)$ at $t=0$, since $(I+t^2{B}^{\ast }B{)}^{-1}=I+O(t^2)$. Thus these are exactly all tangent directions.

By [F3], each frame $Y$ with $Y^{\ast }Y=I$ and range $U$ exists. If $Y$ and $Z$ are two such frames, then $Q=Y^{\ast }Z$ satisfies $YQ=Y{Y}^{\ast }Z=P_{0}Z=Z$ and $Q^{\ast }Q=Z^{\ast }Y{Y}^{\ast }Z=Z^{\ast }Z=I$. Thus $Q$ is unitary and $Z=YQ$. Conversely, $YQ(YQ{)}^{\ast }=Y{Y}^{\ast }$ for every unitary $Q$. Hence projectors identify exactly the unitary changes of occupied frame.

The diagonal matrix $P_{\ast }:=\mathrm{diag}(I_r,0)$ satisfies $P_{\ast }^{\ast }=P_{\ast }$, $P_{\ast }^2=P_{\ast }$, and $\mathrm{tr}{P}_{\ast }=r$, so it belongs to ${\mathcal{P}}_r$ by [F1]. It has rank $r$ by [F2], making the domain nonempty.

The equations $P=P^{\ast }$, $P^2=P$, and $\mathrm{tr}P=r$ are closed conditions on the real coordinates of ${\mathbb{H}}_n$, since adjoint, multiplication, and trace are continuous finite-dimensional maps. Thus ${\mathcal{P}}_r$ is closed. For $P\in {\mathcal{P}}_r$, $\mathrm{\parallel }P{\mathrm{\parallel }}_F^2=\mathrm{tr}(P^{\ast }P)=\mathrm{tr}(P^2)=r$, so it is bounded.

The real dimension of ${\mathbb{H}}_n$ is $n^2$: there are $n$ real diagonal entries and two real coordinates for each of the $n(n-1)\mathrm{/}2$ off-diagonal conjugate pairs. Therefore ${\mathbb{H}}_n$ is linearly homeomorphic to ${\mathbb{R}}^{n^2}$. By [F3], the closed bounded nonempty subset ${\mathcal{P}}_r$ is compact.

If $A,B$ are Hermitian, then $\overline{\mathrm{tr}(AB)}=\mathrm{tr}((AB{)}^{\ast })=\mathrm{tr}(BA)=\mathrm{tr}(AB)$ by the trace identity [F4]. Hence $\mathrm{tr}(hP)$ and $\mathrm{tr}(PG(P))$ are real; $E_{xc}$ is real by [F3]. The first two terms are polynomial in the real coordinates of $P$, so together with the $C^2$ term they give a real $C^2$ function.

For real $t$, real-linearity gives $G(P+tH)=G(P)+tG(H)$. Expanding and collecting the coefficient of $t$ yields ${\frac{d}{dt}\mid }_{t=0}\frac{1}{2}\mathrm{tr}((P+tH)G(P+tH))=\frac{1}{2}\mathrm{tr}(HG(P))+\frac{1}{2}\mathrm{tr}(PG(H))$. The linear trace term differentiates to $\mathrm{tr}(hH)$ by [F1], and the $E_{xc}$ term differentiates to $⟨\mathrm{\nabla }{E}_{xc}(P),H{⟩}_F$ by its gradient definition.

Self-adjointness of $G$ gives $\mathrm{tr}(PG(H))=⟨P,G(H){⟩}_F=⟨G(P),H{⟩}_F=\mathrm{tr}(G(P)H)$. Since these traces are real, the two quadratic contributions in step 1.2 combine to $\mathrm{tr}(G(P)H)$. Adding the other derivatives gives $DE(P)[H]=⟨h+G(P)+\mathrm{\nabla }{E}_{xc}(P),H{⟩}_F$. This is the unique Frobenius gradient pairing, so it equals $⟨F(P),H{⟩}_F$.

The energy $\mathcal{E}$ is continuous by [F1], and the entropy trace is continuous by [F3], so their weighted sum ${\mathcal{F}}_{\beta }$ is continuous on ${\mathcal{D}}_r$.

The domain is nonempty and compact by [F2]. Apply [F4] to ${\mathcal{F}}_{\beta }$ to obtain a minimizer in ${\mathcal{D}}_r$.

Diagonalize $D$ by [F2], with eigenvalues $d_j\in [0,1]$ and ${\sum }_j{d}_j=r$. If $d_i=0$, then $r>0$ gives an index $j$ with $d_j>0$. For small $t>0$, $D_t=D+t(e_i{e}_i^{\ast }-e_j{e}_j^{\ast })$ remains in ${\mathcal{D}}_r$. If $d_j\in (0,1)$, ${\varphi }^{\mathrm{\prime }}$ is bounded near $d_j$, so [F4] gives $\varphi (d_j-t)-\varphi (d_j)=O(t)$. Also $(1-t)\log (1-t)=O(t)$ because this scalar function is differentiable at $0$. Thus the entropy change is $\varphi (t)+\varphi (d_j-t)-\varphi (0)-\varphi (d_j)=t\log t+O(t)$ by [F3]. If $d_j=1$, it is $\varphi (t)+\varphi (1-t)=2t\log t+O(t)$ by $\varphi (1-t)=\varphi (t)$. The energy change is $O(t)$ by Fréchet differentiability of $\mathcal{E}$ at $D$ along the fixed direction $e_i{e}_i^{\ast }-e_j{e}_j^{\ast }$. Dividing the total free-energy change by $t$ gives ${\beta }^{-1}\log t+O(1)$ (or $2{\beta }^{-1}\log t+O(1)$), which is negative for sufficiently small $t$. This contradicts minimality.

If $d_i=1$, then $r<n$ gives $j$ with $d_j<1$. Transfer $t$ from the $i$th eigendirection to the $j$th: $D_t=D+t(e_j{e}_j^{\ast }-e_i{e}_i^{\ast })$. If $d_j\in (0,1)$, the term that leaves the endpoint $1$ is $\varphi (1-t)=\varphi (t)=t\log t+O(t)$ and the interior term changes by $O(t)$ by [F4]. If $d_j=0$, the two endpoint terms change by $2\varphi (t)=2t\log t+O(t)$. The energy change is $O(t)$ by differentiability as in step 1.1. Dividing by $t$ again gives a negative free-energy change for small $t$, contradicting minimality. No eigenvalue can equal $0$ or $1$, proving the claim.