The covariance decomposition of a fixed K-fold score

By [F1], reordering the sample coordinates preserves their common law and mutual independence: the probability of any coordinate rectangle is the product of the same marginal probabilities in either order. Applying [F2] to the original and permuted tuples therefore gives the same joint law $P^{\otimes n}$, so the sample distribution is invariant under every coordinate permutation.

Consider a permutation of indices that maps every fold onto a fold. The product law is invariant under it by step 1.1. Because each learning rule is symmetric, permuting a training tuple within or between folds leaves its fitted predictor unchanged, while the held-out observation is relabelled with its fold loss. Thus the vector $(e_1,\dots ,e_n)$ is invariant in law under all such block permutations. These permutations act transitively on single indices, on ordered distinct pairs in one fold, and on ordered pairs from different folds. Therefore all marginal variances equal ${\sigma }^2$, all within-fold covariances equal $\omega$, and all between-fold covariances equal $\gamma$.

By [F3], the variance of the average is $n^{-2}{\sum }_{i=1}^n{\sum }_{i^{\mathrm{\prime }}=1}^n\mathrm{Cov}(e_i,e_{i^{\mathrm{\prime }}})$. There are $n$ diagonal ordered pairs, $Km(m-1)=n(m-1)$ within-fold off-diagonal ordered pairs, and $n(n-m)$ ordered pairs from different folds. Substitution of the three covariance values from step 2.1 gives $\frac{1}{n^2}\{n{\sigma }^2+n(m-1)\omega +n(n-m)\gamma \}=\frac{{\sigma }^2}{n}+\frac{m-1}{n}\omega +\frac{n-m}{n}\gamma$. For $m=1$, the within-fold count is zero, so the declared value $\omega =0$ contributes nothing.