Plug-and-play MMSE denoisers and stationarity — Examples and Counterexamples

Let $a,\sigma ,\gamma >0$ and let the scalar prior put mass $1\mathrm{/}2$ at each of $-a$ and $a$. Under the Gaussian likelihood kernel ${\rho }_{\sigma }(z-u)=\exp (-(z-u{)}^2\mathrm{/}(2{\sigma }^2))$, its exact posterior-mean map is $D_{\sigma }(z)=a\tanh \text{\hspace{0.17em}⁣}\left(\frac{az}{{\sigma }^2}\right),\tanh (s)=\frac{\exp (s)-\exp (-s)}{\exp (s)+\exp (-s)}\mathrm{.}$ The normalizing constant of the Gaussian likelihood cancels in the posterior. For every $z$, $D_{\sigma }(z)$ uniquely minimizes the posterior squared-error risk and, with the potential $h_{\sigma ,\gamma }$ constructed from this prior, is the unique global minimizer of $x\mapsto (x-z{)}^2\mathrm{/}2+\gamma h_{\sigma ,\gamma }(x)$.

Tweedie identity, positive Jacobian, and the MMSE proximal potential

Assume the Axiom of Choice. Let $\epsilon$ have the standard normal law, let $Y=\theta +\epsilon$, and let $g(x)=(x-Y{)}^2\mathrm{/}2$. For any noise parameter $\sigma >0$, deterministic map ${\tilde{D}}_{\sigma }:\mathbb{R}\to \mathbb{R}$, step $\gamma >0$, and initialization $x_0\in \mathbb{R}$, define $z_t=x_{t-1}-\gamma (x_{t-1}-Y)$ and $x_t={\tilde{D}}_{\sigma }(z_t)$. The false claim is that Gaussian measurement noise forces $z_t-\theta$ to have a Gaussian law at every iteration $t\ge 1$, for every such denoiser, step, and initialization.

Plug-and-play ISTA with a denoiser

Standard normal and normal laws

Let $a,\sigma >0$ and use the scalar two-point prior assigning mass $1\mathrm{/}2$ to each of $-a$ and $a$, whose exact Gaussian posterior-mean map is $D_{\sigma }(z)=a\tanh (az\mathrm{/}{\sigma }^2)$, where $\tanh (s)=(\exp (s)-\exp (-s))\mathrm{/}(\exp (s)+\exp (-s))$. For any tolerance $\eta >0$, choose $b$ with $0<\mathrm{\mid }b\mathrm{\mid }<\eta$ and define the one-pixel convolutional neural network (CNN) map ${\tilde{D}}_{\sigma }(z)=a\tanh (az\mathrm{/}{\sigma }^2)+b\mathrm{.}$ This map is implemented by a $1\times 1$ convolution, a $\tanh$ activation, output scaling, and an output bias $b$. It satisfies ${\sup }_{z\in \mathbb{R}}\mathrm{\mid }{\tilde{D}}_{\sigma }(z)-D_{\sigma }(z)\mathrm{\mid }=\mathrm{\mid }b\mathrm{\mid }<\eta$, while ${\tilde{D}}_{\sigma }(0)\mathrm{\ne }{D}_{\sigma }(0)$. It is approximate and does not meet the exact-denoiser hypothesis of the stationarity theorem; this example makes no claim that it lacks some other objective.

Tweedie identity, positive Jacobian, and the MMSE proximal potential

Plug-and-play ISTA with a denoiser

Exact-MMSE PnP-ISTA has monotone objective and vanishing gradient

An arbitrary denoiser does not inherit the exact-MMSE objective

For this one-channel, one-pixel input, a $1\times 1$ convolution with weight $w$ is scalar multiplication $z\mapsto wz$. The architecture applies weight $a\mathrm{/}{\sigma }^2$, then $\tanh$, then output scaling by $a$ and bias addition by $b$. The arrows depict function composition, and the sequence computes exactly ${\tilde{D}}_{\sigma }(z)=a\tanh (az\mathrm{/}{\sigma }^2)+b$.

\begin{tikzpicture}[x=1.45cm,y=0.8cm,>=stealth]
  \node (input) at (0,0) {$z$};
  \node[draw,rounded corners,minimum width=2.3cm,minimum height=0.9cm] (conv) at (2,0) {$1\times1$ conv: $a/\sigma^2$};
  \node[draw,rounded corners,minimum width=1.4cm,minimum height=0.9cm] (act) at (4.5,0) {$\tanh$};
  \node[draw,rounded corners,minimum width=2cm,minimum height=0.9cm] (out) at (7,0) {scale $a$, bias $b$};
  \node (result) at (9.2,0) {$\widetilde D_\sigma(z)$};
  \draw[->] (input) -- (conv);
  \draw[->] (conv) -- node[above] {$az/\sigma^2$} (act);
  \draw[->] (act) -- node[above] {$\tanh(az/\sigma^2)$} (out);
  \draw[->] (out) -- (result);
\end{tikzpicture}