Latent factors, ICA, NMF, and representation ambiguity — Examples

Put $a=\sqrt{2}$. Direct integration gives ${\int }_0^{\mathrm{\infty }}{e}^{-as}ds=1\mathrm{/}a$ and ${\int }_0^{\mathrm{\infty }}{s}^2{e}^{-as}ds=[-e^{-as}(s^2\mathrm{/}a+2s\mathrm{/}{a}^2+2\mathrm{/}{a}^3){]}_0^{\mathrm{\infty }}=2\mathrm{/}{a}^3$. Hence ${\int }_{\mathbb{R}}f(s)ds=1$, symmetry gives mean zero, and $\mathbb{E}{S}_j^2=2(1\mathrm{/}\sqrt{2})(2\mathrm{/}{a}^3)=1$. The one-sided derivatives of $f$ at zero are $1$ and $-1$, unlike the smooth standard Gaussian density; since both densities are continuous, this law is non-Gaussian.

The matrix $Q$ is a signed permutation, so $Q^{T}S=(S_2,-S_1{)}^T$. The two coordinates remain independent, centered, and unit variance by step 1.1; the even Laplace density is unchanged by sign reversal, so both remain non-Gaussian.

Since $Q^{T}Q=I_2$, $(AQ)(Q^{T}S)=AS$ exactly. Thus the observed law is unchanged while the source order/sign representation changes.

The displayed densities factor as products of the one-dimensional standard Gaussian density $g(x)=(2\pi {)}^{-1\mathrm{/}2}{e}^{-x^2\mathrm{/}2}$, so the coordinates of $S$ and $\epsilon$ are independent. Symmetry gives $\mathbb{E}Z=0$ for a coordinate $Z$, and $(x{e}^{-x^2\mathrm{/}2}{)}^{\mathrm{\prime }}=e^{-x^2\mathrm{/}2}-x^2{e}^{-x^2\mathrm{/}2}$ with vanishing boundary terms gives ${\int }_{\mathbb{R}}{x}^{2}g(x)dx={\int }_{\mathbb{R}}g(x)dx=1$. Thus $\mathbb{E}S=\mathbb{E}\epsilon =0$, $\mathrm{Cov}(S)=I_2$, and $\mathrm{Cov}(\epsilon )=I_3$. The given independence of $S$ and $\epsilon$ makes their cross-covariance zero.

Direct multiplication gives $R^{T}R=I_2$ and $(A{R}^T)(RS)=AS$, so the transformed representation yields the same $X$ for every outcome. By [F1], its covariance is unchanged; here $A{A}^T+I_3=\left(\begin{array}{ccc}2& 0& 1\\ 0& 2& 1\\ 1& 1& 3\end{array}\right)$.

The density of $RS$ is $(2\pi {)}^{-1}\exp (-\parallel R^{T}s{\parallel }^2\mathrm{/}2)\mathrm{\mid }\det R\mathrm{\mid }=(2\pi {)}^{-1}\exp (-\parallel s{\parallel }^2\mathrm{/}2)$ because $R$ is orthogonal and $\mathrm{\mid }\det R\mathrm{\mid }=1$. Thus $RS$ again has independent standard Gaussian coordinates; it remains independent of $\epsilon$, and [F2] together with step 1.1 shows both parameterizations satisfy the model. Since $R$ has four nonzero entries, it is not a signed permutation.

Each row of $W_{\alpha }$ sums to $1$, while each entry of $H$ is $1$, so $(W_{\alpha }H{)}_{ij}=\alpha +(1-\alpha )=1$ for every cell. This verifies the common positive rate matrix.

Under the transformations in [F2], each rank-one component matrix $W_{\cdot k}{H}_{k\cdot }$ is unchanged except for reordering. At $\alpha =1\mathrm{/}2$ the multiset of component matrices is $\{\frac{1}{2}X,\frac{1}{2}X\}$, whereas at $\alpha =1\mathrm{/}3$ it is $\{\frac{1}{3}X,\frac{2}{3}X\}$. These multisets differ, so the factorizations are not related by scale or permutation.

Initially every rate is $2$. For each $i,k$, the $W$ numerator is $0\mathrm{/}2+1\mathrm{/}2=1\mathrm{/}2$ and the denominator is $1+1=2$, so $W_{ik}^{(1)}=1\mathrm{/}4$. The recomputed intermediate rates are all $1\mathrm{/}2$.

For each $k,j$, the $H$ numerator is $\frac{1}{4}(0\mathrm{/}(1\mathrm{/}2))+\frac{1}{4}(1\mathrm{/}(1\mathrm{/}2))=1\mathrm{/}2$ and the denominator is $1\mathrm{/}4+1\mathrm{/}4=1\mathrm{/}2$, hence $H_{kj}^{(1)}=1$. Every division is by a positive rate as guaranteed by [F3], and a zero-count cell contributes $0\mathrm{/}(1\mathrm{/}2)=0$, not an indeterminate quotient.