An unseen target action cannot be importance sampled from behavior data

Assume the Axiom of Choice for the policy path laws and fix $0\le \gamma <1$. Let the state set be $\{s,d\}$, with $A(s)=\{a,b\}$ and $A(d)=\{c\}$. Action $a$ at $s$ gives reward $0$ and transitions to terminal state $d$; action $b$ gives reward $1$ and also transitions to $d$. At $d$, action $c$ gives reward $0$ and returns to $d$, but episodes stop upon first reaching $d$. Let the behavior policy choose $a$ with probability $1$ at $s$ and the target policy choose $b$ with probability $1$. The formal ordinary first-visit importance-sampling estimator is evaluated here outside the Monte Carlo algorithm's target-support admissibility condition, to show why that condition is needed. Every behavior episode observes $a$, whose action ratio is $\pi (a\mid s)\mathrm{/}\mu (a\mid s)=0$, so the estimate is always $0$, while the target value is $1$. The target-support condition fails because $\mu (b\mid s)=0<\pi (b\mid s)$.

The finite joint reward-transition model.

The behavior-to-target action ratio and support condition.

The ordinary first-visit importance-sampling estimator.

Sutton and Barto, Reinforcement Learning: An Introduction.

Yang, Optimal Control and Reinforcement Learning, Chapter 2.