Normal equations for best affine $L^2$ prediction

Let $V:=\mathrm{span}([X],[1],[Y_1],\dots ,[Y_n])\subseteq L^2(\mathbb{P})\mathrm{.}$ This space is finite-dimensional because it is spanned by $n+2$ vectors, and $W$ is a subspace of $V$. Equip $V$ with the usual $L^2$ inner product. Applying [L4] to the vector $[X]\in V$ and the subspace $W\subseteq V$ gives a unique class $[Z_{\ast }]\in W$ minimizing $\mathrm{\parallel }[X]-[Z]{\mathrm{\parallel }}_2$ over $[Z]\in W$.

By [L3], every class in $W$ has an affine representative of the form $Z=a_0+{\sum }_{j=1}^n{a}_j{Y}_j$. Put $\tilde{X}:=X-\mathbb{E}[X]$ and ${\tilde{Y}}_j:=Y_j-\mathbb{E}[Y_j]$. Then $X-Z=\tilde{X}-{\sum }_{j=1}^n{a}_j{\tilde{Y}}_j+(\mathbb{E}[X]-a_0-{\sum }_{j=1}^n{a}_j\mathbb{E}[Y_j])\mathrm{.}$ Expanding the square and using [L1] shows that the mixed term with the final constant vanishes, so $\mathbb{E}[(X-Z{)}^2]=\mathbb{E}\left[{(\tilde{X}-{\sum }_{j=1}^n{a}_j{\tilde{Y}}_j)}^2\right]+{(\mathbb{E}[X]-a_0-{\sum }_{j=1}^n{a}_j\mathbb{E}[Y_j])}^2\mathrm{.}$ Hence every affine representative of the minimizing class must satisfy $a_0=\mathbb{E}[X]-{\sum }_{j=1}^n{a}_j\mathbb{E}[Y_j]\mathrm{.}$

After imposing the intercept from step 2.1, define $Q(a_1,\dots ,a_n):=\mathbb{E}\left[{(\tilde{X}-{\sum }_{j=1}^n{a}_j{\tilde{Y}}_j)}^2\right]\mathrm{.}$ Using [L1] and [L2], $Q(a)=\mathrm{Var}(X)-2{\sum }_{j=1}^n{a}_j\mathrm{Cov}(X,Y_j)+{\sum }_{i=1}^n{\sum }_{j=1}^n{a}_i{a}_j\mathrm{Cov}(Y_i,Y_j)\mathrm{.}$

Let $h=(h_1,\dots ,h_n)\in {\mathbb{R}}^n$. Substituting $a+h$ into step 3.1 and subtracting gives $Q(a+h)-Q(a)=\mathbb{E}\left[{\left({\sum }_{j=1}^n{h}_j{\tilde{Y}}_j\right)}^2\right]+2{\sum }_{i=1}^n{h}_i({\sum }_{j=1}^n\mathrm{Cov}(Y_i,Y_j)a_j-\mathrm{Cov}(X,Y_i))\mathrm{.}$ Therefore, if $a$ satisfies the normal equations, then $Q(a+h)-Q(a)\ge 0$ for every $h$, so $a$ minimizes $Q$.

Conversely, suppose $a$ minimizes $Q$. Fix $i$ and take $h=t{e}_i$, where $e_i$ is the $i$th standard basis vector. Then step 4.1 gives $Q(a+t{e}_i)-Q(a)=t^2\mathrm{Var}(Y_i)+2t({\sum }_{j=1}^n\mathrm{Cov}(Y_i,Y_j)a_j-\mathrm{Cov}(X,Y_i))\ge 0$ for every real $t$. The coefficient of $t$ must therefore be $0$, since the same inequality holds for both signs of $t$. This is exactly the $i$th normal equation, and $i$ was arbitrary.

Let $a$ and $b$ be two coefficient vectors satisfying the normal equations, and let $D:={\sum }_{j=1}^n(a_j-b_j){\tilde{Y}}_j\mathrm{.}$ Subtracting the two linear systems gives ${\sum }_j\mathrm{Cov}(Y_i,Y_j)(a_j-b_j)=0$ for every $i$. Multiplying the $i$th equation by $a_i-b_i$ and summing over $i$, then using bilinearity from [L1], yields $\mathbb{E}[D^2]={\sum }_{i=1}^n{\sum }_{j=1}^n(a_i-b_i)(a_j-b_j)\mathrm{Cov}(Y_i,Y_j)=0.$ Because $D^2\ge 0$, [L5] gives $D=0$ almost surely. Thus the corresponding predictors agree almost surely, so the minimizing class in $L^2(\mathbb{P})$ is unique even when the coefficient vector is not.

Step 1.1 gives existence and uniqueness of the minimizing class, step 2.1 identifies the optimal intercept, steps 4.1 and 5.1 characterize the optimal centered coefficients by the covariance normal equations, and step 6.1 proves that all coefficient solutions yield the same predictor almost surely. This is exactly the best affine $L^2$ prediction statement.