A rounded subtractive covariance update can become indefinite

Since $0<c<1$, for $(x_1,x_2)\mathrm{\ne }(0,0)$, $(x_1,x_2)P^{-}(x_1,x_2{)}^T=(x_1+c{x}_2{)}^2+(1-c^2)x_2^2>0.$ Thus $P^{-}$ is positive definite. Also $1-c^2=2^{-23}-2^{-48}>0$. The exact innovation is $S=H{P}^{-}{H}^T+R=4-2^{-23}+2^{-22}=4+2^{-23}>0.$

In precision $p=24$, the exact sum $1+c=2-2^{-24}$ is halfway between the representable predecessor $2-2^{-23}$ and $2$; ties-to-even rounds it to $2$. Hence the computed $B$ is $(2,2{)}^T$. The next exact addition is $4+2^{-22}$, halfway between $4$ and its next representable successor $4+2^{-21}$; ties-to-even rounds it to $S_{\mathrm{f}\mathrm{l}}=4$. All these operands and results are normal.

The computed gain is $K=(1\mathrm{/}2,1\mathrm{/}2{)}^T$. Multiplication gives $(KS)K^T=\left[\begin{array}{cc}1& 1\\ 1& 1\end{array}\right]$ exactly, so the rounded subtractive update is $P_{\mathrm{s}\mathrm{u}\mathrm{b}}=\left[\begin{array}{cc}0& -2^{-24}\\ -2^{-24}& 0\end{array}\right]\mathrm{.}$ Its eigenvalues are $2^{-24}$ and $-2^{-24}$, so it is indefinite. This shows cancellation can lose positive semidefiniteness even with an exact positive-definite prior and positive measurement variance. By contrast, when the factor procedure succeeds, [F3] gives a represented covariance that is an exact factor product; this example makes no claim that rounding a later materialization of that product preserves the property.