Fixed-design regression, estimability, and Gauss–Markov — Examples

For $b=(b_1,b_2{)}^T$, put $s=b_1+b_2$. Then $Xb=(s,s{)}^T$, and the squared residual is $(1-s{)}^2+(3-s{)}^2=2(s-2{)}^2+2.$ It is minimized exactly when $s=2$. Thus the full coefficient solution set is $(1,1{)}^T+t(1,-1{)}^T$ for $t\in \mathbb{R}$, its fitted vector is $(2,2{)}^T$, and its residual is $(-1,1{)}^T$, in agreement with [F1].

The row space of $X$ is $\mathrm{span}\{(1,1{)}^T\}$. Hence $c=(1,1{)}^T$ is in the row space and is estimable; indeed $X^T(1\mathrm{/}2,1\mathrm{/}2{)}^T=(1,1{)}^T$, so $(Y_1+Y_2)\mathrm{/}2$ estimates ${\beta }_1+{\beta }_2$ unbiasedly. The vector $(1,0{)}^T$ is not in that row space, since every vector $X^{T}a$ has equal coordinates. Therefore neither single coefficient is linearly estimable by [F2].

The weights $1\mathrm{/}4$ are nonnegative and sum to one. For a disjoint sequence of subsets of $\mathrm{\Omega }$, at most four sets are nonempty, so additivity of cardinality gives countable additivity; thus $\mathbb{P}$ is a probability measure by [F4]. In the displayed formula, $P_{\beta }({\mathbb{R}}^2)=1$; for any disjoint Borel sequence, each of the four outcomes contributes to at most one term, so $P_{\beta }$ is countably additive and hence a probability measure on $\mathcal{B}({\mathbb{R}}^2)$. Direct weighted sums using [F3] give $\mathbb{E}U=\mathbb{E}V=0$, $\mathbb{E}{U}^2=\mathbb{E}{V}^2=1$, and $\mathbb{E}(UV)=0$. Thus $\mathbb{E}ϵ=0$ and, by [F5], $\mathrm{Cov}(ϵ)=\mathrm{diag}(1,9)$, which is not a scalar multiple of $I_2$. By linearity [F2], ${\mathbb{E}}_{\beta }Y=X\beta$.

Here $X^{T}X=2$, so the unique OLS estimate is $\widehat{\beta }=\frac{Y_1+Y_2}{2}=\beta +\frac{U+3V}{2}\mathrm{.}$ It is unbiased by [F2], and bilinearity gives $\mathrm{Var}(\widehat{\beta })=(1+9)\mathrm{/}4=5\mathrm{/}2$.

The alternative estimator $\tilde{\beta }=\frac{9}{10}{Y}_1+\frac{1}{10}{Y}_2=\beta +\frac{9}{10}U+\frac{3}{10}V$ is linear and unbiased because its coefficients sum to one and both error means vanish. Its variance is $\frac{81}{100}\mathrm{Var}(U)+\frac{9}{100}\mathrm{Var}(V)+\frac{54}{100}\mathrm{Cov}(U,V)=\frac{9}{10}<\frac{5}{2}\mathrm{.}$ So OLS is not BLUE in this model. The covariance premise in [F1] fails, which is the stated boundary of the Gauss–Markov theorem.

The predictor mean is $\bar{x}=26\mathrm{/}5$. Put $z=x-\bar{x}\mathbf{1}=(-26,-21,-16,-11,74{)}^T\mathrm{/}5$. Then ${\mathbf{1}}^{T}z=0$ and $z^{T}z=\frac{676+441+256+121+5476}{25}=\frac{1394}{5}\mathrm{.}$ Since $x=z+\bar{x}\mathbf{1}$, the column space is $C=\mathrm{span}\{\mathbf{1},z\}$, with these two generators nonzero and orthogonal.

For every $y\in {\mathbb{R}}^5$, the vector $p(y)=\frac{{\mathbf{1}}^{T}y}{5}\mathbf{1}+\frac{z^{T}y}{z^{T}z}z$ lies in $C$, and $y-p(y)$ is orthogonal to both generators, hence to all of $C$. By [F2], $p(y)=P_{C}y$; [F1] identifies this as the fitted vector. Therefore $H=\frac{{\mathbf{11}}^T}{5}+\frac{z{z}^T}{z^{T}z},h_{55}=\frac{1}{5}+\frac{(74\mathrm{/}5{)}^2}{1394\mathrm{/}5}=\frac{1}{5}+\frac{2738}{3485}=\frac{687}{697}\mathrm{.}$

Since $h_{55}=1-10\mathrm{/}697$ is close to one and its formula involves only the design vector, this is a high-leverage design row. The calculation makes no claim that the fifth response has a large residual or is influential; those are separate diagnostics.