Exp4 expert reward estimates are conditionally unbiased

For the fixed instance, there are finitely many action histories. The weights and finite estimates are functions of those histories and the current action, so they are measurable with finite ranges and hence integrable. Fix a positive-probability pre-action history; on it, $Q_t$ and $E(t)$ are fixed. If $P_{t,i}=0$, then $0={\sum }_r{Q}_{t,r}E(t{)}_{r,i}$ is a sum of nonnegative terms and every $Q_{t,r}>0$; hence $E(t{)}_{r,i}=0$ for every expert $r$. Such a coordinate contributes nothing to any expert reward estimate.

For $P_{t,i}>0$ on this history, the estimate is $(1-x_{t,i})\mathrm{/}{P}_{t,i}$ when $A_t=i$ and zero otherwise. Thus its conditional mean, computed directly from [F3], is $P_{t,i}(1-x_{t,i})\mathrm{/}{P}_{t,i}=1-x_{t,i}$, regardless of other arms' probabilities. For fixed expert $m$, its row sums to one, so ${\tilde{X}}_{t,m}=1-{\sum }_{i}E(t{)}_{m,i}{\widehat{Y}}_{t,i}$. Distributing the finite conditional sum and omitting the zero coefficients from step 1.1 gives $\mathbb{E}[{\tilde{X}}_{t,m}\mid {\mathcal{F}}_{t-1}]=1-{\sum }_{i:P_{t,i}>0}E(t{)}_{m,i}(1-x_{t,i})={\sum }_{i=1}^{k}E(t{)}_{m,i}{x}_{t,i}$. These values on the finitely many positive-probability history atoms satisfy the integral characterization [F5] on every history event; values on null histories do not affect the conditional-expectation identity.