Duplicate inputs force the population Gram matrix to be singular

The rows $i,j$ of $K^{(0)}$ coincide because $K_{i\mathrm{\ell }}^{(0)}=⟨x_i,x_{\mathrm{\ell }}⟩=⟨x_j,x_{\mathrm{\ell }}⟩=K_{j\mathrm{\ell }}^{(0)}$ for every $\mathrm{\ell }$. This is the base case.

Assume rows $i,j$ of $K^{(h-1)}$ coincide. By [F2], the matrix is symmetric, so the corresponding columns coincide as well. For each $\mathrm{\ell }$, the covariance matrices $A_{i\mathrm{\ell }}^{(h)}$ and $A_{j\mathrm{\ell }}^{(h)}$ are equal. Thus the Gaussian activation expectations agree for $h<H$, and for $h=H$ both the derivative expectations and prefactors $K_{i\mathrm{\ell }}^{(H-1)}$, $K_{j\mathrm{\ell }}^{(H-1)}$ agree. Hence rows $i,j$ of the fully connected $K^{(h)}$ coincide.

For the fixed $H$, let $P(h)$ assert that rows $i,j$ of $K^{(h)}$ coincide for $0\le h\le H$. Step 1.1 gives the base, and step 1.2 proves $P(h)$ from $P(h-1)$ for every $1\le h\le H$. By [F4], the fully connected terminal rows coincide. For $H=1$, the residual terminal matrix equals this fully connected terminal matrix by definition, so its rows also coincide.

Now assume $H\ge 2$. For $1\le h<H$, let $Q(h)$ assert both that rows $i,j$ of $S^{(h)}$ coincide and that $b_i^{(h)}=b_j^{(h)}$. The base $S^{(0)}=K^{(0)}$ has equal rows by step 1.1. At layer $1$, this equality and symmetry from [F2] make the pair covariance matrices for indices $(i,\mathrm{\ell })$ and $(j,\mathrm{\ell })$ identical for every $\mathrm{\ell }$, and give $S_{ii}^{(0)}=S_{jj}^{(0)}$. Hence the Gaussian marginal means agree, so the defining formulas give $S_{i\mathrm{\ell }}^{(1)}=S_{j\mathrm{\ell }}^{(1)}$ for every $\mathrm{\ell }$ and $b_i^{(1)}=b_j^{(1)}$; this proves $Q(1)$. For $2\le h<H$, assume $Q(h-1)$. Symmetry from [F2] and equal rows of $S^{(h-1)}$ give equal columns, so the covariance matrices for $(i,\mathrm{\ell })$ and $(j,\mathrm{\ell })$ coincide and ${\mu }_i^{(h-1)}={\mu }_j^{(h-1)}$. In the mean recursion this gives $b_i^{(h)}=b_j^{(h)}$. In the second-moment recursion, the old-row terms agree, the terms $b_i^{(h-1)}{\mu }_{\mathrm{\ell }}^{(h-1)}+b_{\mathrm{\ell }}^{(h-1)}{\mu }_i^{(h-1)}$ agree with their $j$-row counterparts, and the Gaussian activation expectations agree; thus $S_{i\mathrm{\ell }}^{(h)}=S_{j\mathrm{\ell }}^{(h)}$ for every $\mathrm{\ell }$. This proves $Q(h)$. Finite induction gives $Q(H-1)$. The terminal residual formula then has equal prefactors $S_{i\mathrm{\ell }}^{(H-1)}=S_{j\mathrm{\ell }}^{(H-1)}$ and equal derivative expectations because the corresponding pair covariance matrices coincide. Therefore the terminal residual rows coincide.

For either terminal matrix, let $v=e_i-e_j\mathrm{\ne }0$. Symmetry and equal rows imply $K^{(H)}v=0$. By [F2]–[F3], this matrix is positive semidefinite and has a nonzero null vector, so zero is an eigenvalue and the smallest eigenvalue is zero. This proves the residual branch and completes the example.