Benchmark risk, compute budgets, and uncertainty records

Fix measurable spaces $(X,\mathcal{X})$ and $(Y,\mathcal{Y})$, a metric target space $(Q,d_Q)$ with its Borel sigma-algebra $\mathcal{Q}$, and a nonempty truth class $U\subseteq X$ with inherited sigma-algebra $\mathcal{U}=\{A\cap U:A\in \mathcal{X}\}$. Let $q:U\to Q$ and $F:U\to Y$ be measurable target and forward maps. The admissible state set is a declared nonempty $\mathcal{U}$-measurable subset $C\subseteq U$. The observation model is a specified probability kernel $K(dy\mid x)$ from $(U,\mathcal{U})$ to $(Y,\mathcal{Y})$; in an additive-noise model, when $Y$ has a measurable addition operation, it is the law of $F(x)+\epsilon$ for a declared noise law of $\epsilon$, while nonadditive or model-dependent observation noise is encoded directly by $K$.

The forward solver, when a numerical evaluation is used, is the specified procedure for evaluating this model, including its discretization, tolerances, finite call budget, and any model-specific admissibility test; exact evaluation is the special case with no numerical-solver error. A failed admissibility test or solver call is recorded with its failure status and is not silently omitted from the benchmark results.

A benchmark specification also declares a training law $P_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}$ on a named measurable space for all random variables used to fit a method or select its hyperparameters; those variables may be paired or unpaired. It declares a test law $P_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}$ on $(Y\times Q,\mathcal{Y}\otimes \mathcal{Q})$ for evaluation pairs $(Y,q(X))$, and any separate validation law used for tuning. The training and test laws need not agree. Each candidate method is identified by its fitting procedure and by its deployed measurable deterministic reconstruction rule $g:Y\to Q$ or randomized reconstruction kernel $L(da\mid y)$ from $Y$ to $Q$. Its population test risk is the risk under $P_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}$ as defined by the inverse-risk functional; a reported finite-sample score must additionally name its test sample and empirical aggregation rule. No risk transfer or generalization bound follows from the benchmark specification.

Before comparison, the specification fixes a nonnegative product-measurable loss $\mathrm{\ell }:Q\times Q\to [0,+\mathrm{\infty }]$, an uncertainty diagnostic, and a finite compute-budget vector. An uncertainty diagnostic is a declared measurable statistic of the method outputs and, when applicable, target labels, together with its target quantity; for example, if a reported set-valued estimate $C(y)\subseteq Q$ has a product-measurable membership event $\{(y,a):a\in C(y)\}\in \mathcal{Y}\otimes \mathcal{Q}$, its test coverage is $P_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}(\{(y,a)\in Y\times Q:a\in C(y)\})$. The budget vector records the applicable maximum counts of forward-model queries, adjoint queries, gradient evaluations, ODE right-hand-side evaluations, solver or sampling steps, and elapsed runtime; inapplicable counts are marked as not applicable, not zero. It also records the hardware, numerical precision, software/model versions, random seeds, tuning data, and stopping tolerances needed to interpret those counts. Each method's result records its metric values, uncertainty diagnostic, realized compute counts, solver status, and failed runs. A comparison is tied to the declared test law, test sample, loss, solver, admissibility rules, and tuning protocol; when budgets or other conditions differ, those differences are reported as part of the comparison. The specification defines an evaluation protocol, not a theorem ranking algorithms.

The arrows label declared information flow into fitting, observation generation, and evaluation. The diagram asserts no equality between $P_{\mathrm{t}\mathrm{r}\mathrm{a}\mathrm{i}\mathrm{n}}$ and $P_{\mathrm{t}\mathrm{e}\mathrm{s}\mathrm{t}}$ and no risk-transfer result.

Risk under a specified joint evaluation law

Point reconstruction rules and randomized posterior kernels

Separate forward operators, observation laws, and prior support

A model-specific finite solver/admissibility contract

Bal, Introduction to Inverse Problems, §1.1 and §§8.2–8.3, full lecture notes

Zheng et al., InverseBench, ICLR 2025, full paper and appendices