Fixed-design regression, estimability, and Gauss–Markov

Let $n,p\ge 1$ be integers and let $X\in {\mathbb{R}}^{n\times p}$ be a fixed matrix. A fixed-design first-moment linear regression model is a parametric statistical experiment on $({\mathbb{R}}^n,\mathcal{B}({\mathbb{R}}^n))$ with parameter set ${\mathbb{R}}^p$, laws $(P_{\beta }{)}_{\beta \in {\mathbb{R}}^p}$, and observed vector $Y(y)=y$, such that every coordinate of $Y$ is integrable under $P_{\beta }$ and ${\mathbb{E}}_{\beta }Y=X\beta (\beta \in {\mathbb{R}}^p)\mathrm{.}$ Here $\mathcal{B}({\mathbb{R}}^n)$ is the Borel sigma-algebra, and expectation is taken under the particular law $P_{\beta }$. The residual vector ${ϵ}_{\beta }:=Y-X\beta$ therefore has coordinatewise mean zero. The phrase fixed design means that $X$ is held constant in this experiment; it carries no probability law or assignment mechanism for $X$.

This definition states only the first-moment model. It imposes no covariance, independence, or Gaussian assumption. Such assumptions must be stated where used.

Residual mean. For each fixed $\beta$ and coordinate $j$, put $c_j=(X\beta {)}_j$. Both $Y_j$ and the constant $c_j$ are integrable under the probability law $P_{\beta }$, and the constant has expectation $c_j$. By Linearity, monotonicity, and the modulus bound for expectation, ${\mathbb{E}}_{\beta }[{ϵ}_{\beta ,j}]={\mathbb{E}}_{\beta }[Y_j-c_j]={\mathbb{E}}_{\beta }[Y_j]-c_j=(X\beta {)}_j-c_j=0.$ Thus the residual-mean conclusion in the Definition uses expectation linearity coordinatewise under each law; it adds no covariance or distributional assumption.

This is an abstract mathematical regression model, not a complete scientific model contract for an application. The Definition of Scientific statistical model contracts requires a field record for a specified analysis, including its target population; this pair has no specified application or target population. The following notes record the model-side fields that can be stated at this abstract level.

• Borel convention. $\mathcal{B}({\mathbb{R}}^n)$ is generated by the metric topology of the standard Euclidean metric $d_2$: $d_2$ is a metric by ${\mathbb{R}}^n$ as the set of functions $n\to \mathbb{R}$, and $d_1$, $d_2$, $d_{\mathrm{\infty }}$ are metrics on it, and its metric-open sets form a topology by The metric topology: a set is open when every one of its points has a ball around it inside the set; closed means open complement, Arbitrary unions and finite intersections of open sets are open, open balls are open and closed balls are closed, and Topology on a set, open and closed sets, clopen sets, the closed-set axiomatisation, and the coarser/finer comparison. Thus this is the Borel sigma-algebra specified by The Borel sigma-algebra of a topological space.

• Record, units, and indices. One observed data object is the vector $Y\in {\mathbb{R}}^n$ on the Borel space ${\mathbb{R}}^n$. The analysis unit is this vector; coordinate $i$ is an abstract response slot, not a specified biological, physical, sampling, or assignment unit. No enrollment process is modeled, so enrollment status is not applicable; this says nothing about enrollment in a later application. There is no time index, and distinct coordinates need not be independent or distinct scientific units.

• Fixed, random, and observed quantities. $X$ is fixed; $\beta$ is a fixed parameter indexing $P_{\beta }$; $Y$ is random under that law and observed as $y$; ${ϵ}_{\beta }$ is its random residual. No latent quantity or randomized assignment is specified.

• Target, mechanism, law, and conditioning. $\beta$ indexes the formal parameter. For a contrast $c$, ${\psi }_c(\beta )=c^T\beta$ is a formal model estimand; no target population is specified. The law is directly $P_{\beta }$ on ${\mathbb{R}}^n$, with no separate mechanism from a fuller scientific state and no additional support, density, or common-support assumption. All moments are under $P_{\beta }$ with $X$ fixed; there is no averaging over a random design.

• Adequacy. In a specified application, the mean relation ${\mathbb{E}}_{\beta }Y=X\beta$ is empirically assessable: data or validation can challenge it but cannot prove it. This pair supplies no application data and does not assert adequacy. Additional moment, covariance, independence, or Gaussian conditions are theorem-specific mathematical hypotheses, not part of this first-moment model.

The scientific application, including its target population and substantive adequacy evidence, remains unspecified.

Let $n\ge 1$ and $a\in {\mathbb{R}}^n$. The linear map ${\mathrm{\ell }}_a:{\mathbb{R}}^n\to \mathbb{R},{\mathrm{\ell }}_a(y)=a^{T}y,$ is measurable from the Euclidean Borel space ${\mathbb{R}}^n$ to the Borel space $\mathbb{R}$.

For integers $n,p\ge 1$, a matrix $X\in {\mathbb{R}}^{n\times p}$, and a response vector $y\in {\mathbb{R}}^n$, define the least-squares solution set ${\mathcal{B}}_{\mathrm{L}\mathrm{S}}(y):={\mathrm{arg\hspace{0.17em}min}}_{b\in {\mathbb{R}}^p}\mathrm{\parallel }y-Xb{\mathrm{\parallel }}_2^2\mathrm{.}$ This is a set-valued definition; it does not select one coefficient vector. The next theorem proves that the set is nonempty and identifies the fitted vector and residual as unique, including when $X$ is rank-deficient.

In a fixed-design first-moment regression model with parameter $\beta \in {\mathbb{R}}^p$, a linear contrast is the estimand ${\psi }_c(\beta ):=c^T\beta ,c\in {\mathbb{R}}^p\mathrm{.}$ The contrast is linearly estimable if there is a fixed vector $a\in {\mathbb{R}}^n$ such that the statistic $a^{T}Y$ is unbiased for ${\psi }_c$ at every parameter value: ${\mathbb{E}}_{\beta }[a^{T}Y]=c^T\beta \text{for every }\beta \in {\mathbb{R}}^p\mathrm{.}$ The vector $a$ may depend on the fixed design and on $c$, but not on the observed response or on the unknown parameter. The map $y\mapsto a^{T}y$ is a measurable statistic by Linear statistics on Euclidean data are Borel-measurable and Statistics and estimators.

Let $n,p\ge 1$, $X\in {\mathbb{R}}^{n\times p}$, $y\in {\mathbb{R}}^n$, and $C=\mathrm{col}(X)\subseteq {\mathbb{R}}^n$. For the least-squares solution set ${\mathcal{B}}_{\mathrm{L}\mathrm{S}}(y)$ of Least-squares solution set, the following hold:

1. ${\mathcal{B}}_{\mathrm{L}\mathrm{S}}(y)$ is nonempty, and $b$ is in it exactly when $X^T(Xb-y)=0$.
2. If $b_0\in {\mathcal{B}}_{\mathrm{L}\mathrm{S}}(y)$, then ${\mathcal{B}}_{\mathrm{L}\mathrm{S}}(y)=b_0+\ker X$.
3. Every least-squares solution has the same fitted vector $P_{C}y$ and residual $y-P_{C}y$.
4. The direction space $\ker X$ of the affine solution set has dimension $p-\mathrm{rank}(X)$.

The theorem makes no statistical moment or distributional assumption.

In a fixed-design first-moment regression model with $X\in {\mathbb{R}}^{n\times p}$, a contrast $c^T\beta$ is linearly estimable if and only if $c\in \mathrm{row}(X)=\mathrm{im}(X^T),$ equivalently if and only if there exists $a\in {\mathbb{R}}^n$ with $X^{T}a=c$. Whenever this holds, $a^{T}Y$ is unbiased for $c^T\beta$ for every $\beta$. Here $\mathrm{row}(X)$ is defined as the image of the transpose map.

Let $X\in {\mathbb{R}}^{n\times p}$ have full column rank (its columns are linearly independent) in a fixed-design first-moment regression model. Suppose for some ${\sigma }^2>0$ the error vector ${ϵ}_{\beta }:=Y-X\beta$ has finite second moments, ${\mathbb{E}}_{\beta }{ϵ}_{\beta }=0$, and ${\mathrm{Cov}}_{\beta }({ϵ}_{\beta })={\sigma }^2{I}_n$ for every $\beta$, where the vector covariance matrix has entries ${\mathrm{Cov}}_{\beta }({ϵ}_{\beta ,i},{ϵ}_{\beta ,j})$. Then the unique least-squares estimator is $\widehat{\beta }=(X^{T}X{)}^{-1}{X}^{T}Y,$ it is unbiased for $\beta$, and its covariance matrix is ${\mathrm{Cov}}_{\beta }(\widehat{\beta })={\sigma }^2(X^{T}X{)}^{-1}\mathrm{.}$ No Gaussian assumption is made.

Let $X\in {\mathbb{R}}^{n\times p}$ have full column rank in a fixed-design first-moment regression model. Suppose ${\mathrm{Cov}}_{\beta }({ϵ}_{\beta })={\sigma }^2{I}_n$ for a fixed ${\sigma }^2>0$, where the vector covariance matrix has entries ${\mathrm{Cov}}_{\beta }({ϵ}_{\beta ,i},{ϵ}_{\beta ,j})$. For every contrast $c^T\beta$, the OLS estimate $c^T\widehat{\beta }$ is unbiased and has variance no larger than any other linear unbiased estimator of that contrast. Thus OLS is best linear unbiased (BLUE) for every estimable contrast. Here “best” means minimum variance within the class of linear unbiased estimators; Gaussianity is not assumed.