Scores and Information Bounds

Given: AC; $\mathrm{\Theta }\subseteq {\mathbb{R}}^d$ open; a fixed sigma-finite dominating measure $\mu$; a selected density family $p_{\eta }$; a fixed support $S$ and common null set $N$ on a neighborhood $U$; pointwise continuous first and second partial derivatives on $S\setminus N$; integrable uniform envelopes $g_1,g_2$ for those derivatives; and a finite second moment of the score.

The Axiom of Choice: AC says every family of nonempty sets has a choice function; here it supplies the simultaneous family of density representatives.

Dominated statistical experiments: every sampling law $P_{\eta }$ is dominated by the fixed sigma-finite positive measure $\mu$.

Likelihood densities: $p_{\eta }$ is a selected measurable Radon–Nikodym representative of $d{P}_{\eta }\mathrm{/}d\mu$.

A sigma-finite signed measure that is absolutely continuous with respect to a sigma-finite positive measure has a unique almost-everywhere density: $P_{\eta }(A)={\int }_A{p}_{\eta }d\mu$ for every measurable $A$.

Probability measures and probability spaces: $P_{\eta }(\mathcal{X})=1$.

The measure with density $f$ relative to $\mu$ and Integrating against a density agrees with integrating the product: if $P_{\eta }=p_{\eta }d\mu$, then ${\mathbb{E}}_{\eta }f(X)=\int f{p}_{\eta }d\mu$ for nonnegative $f$. For real integrable $f$, apply this identity to $f^{+}$ and $f^{-}$; applying it first to $\mathrm{\mid }f\mathrm{\mid }$ shows $\mathrm{\mid }f\mathrm{\mid }{p}_{\eta }$ is integrable, so the signed identity follows from the definition of the signed integral.

The mean value theorem, as the case $g(x)=x$ of Cauchy's: for $f$ continuous on $[a,b]$ with $a<b$ and differentiable on $(a,b)$ there is $c\in (a,b)$ with $f(b)-f(a)=f^{\mathrm{\prime }}(c)(b-a)$: for a differentiable function on an interval, the difference quotient between two endpoints equals its derivative at an intermediate point.

Dominated convergence: pointwise almost-everywhere convergence under one integrable majorant implies convergence of the integrals.

Score and Fisher information: on $p_{\theta }>0$, the score is ${\mathrm{\nabla }}_{\theta }\log p_{\theta }$ and the information matrix has entries ${\mathbb{E}}_{\theta }[S_{\theta ,i}{S}_{\theta ,j}]$.

The chain rule, in one line from Carathéodory: if $g$ is differentiable at $c$ and $f$ is differentiable at $g(c)$, then $f\circ g$ is differentiable at $c$ with $(f\circ g{)}^{\mathrm{\prime }}(c)=f^{\mathrm{\prime }}(g(c))g^{\mathrm{\prime }}(c)$, Sums, scalar multiples, products and quotients: $(f+g{)}^{\mathrm{\prime }}(c)=f^{\mathrm{\prime }}(c)+g^{\mathrm{\prime }}(c)$, $(\alpha f{)}^{\mathrm{\prime }}(c)=\alpha f^{\mathrm{\prime }}(c)$, $(fg{)}^{\mathrm{\prime }}(c)=f^{\mathrm{\prime }}(c)g(c)+f(c)g^{\mathrm{\prime }}(c)$, and $(f\mathrm{/}g{)}^{\mathrm{\prime }}(c)=(f^{\mathrm{\prime }}(c)g(c)-f(c)g^{\mathrm{\prime }}(c))\mathrm{/}g(c{)}^2$ when $g(c)\mathrm{\ne }0$, and The natural logarithm has derivative 1/x and equals the integral from 1 to x of 1/t: on the positive support, ${\mathrm{\partial }}_j\mathrm{\ell }=({\mathrm{\partial }}_{j}p)\mathrm{/}p$ and ${\mathrm{\partial }}_{ij}\mathrm{\ell }=({\mathrm{\partial }}_{ij}p)\mathrm{/}p-({\mathrm{\partial }}_{i}p)({\mathrm{\partial }}_{j}p)\mathrm{/}{p}^2$.

Expectation of a nonnegative or integrable random variable and Integrable real and complex functions, and their integrals: expectations are integrals and signed expectations require integrability.

The Lebesgue integral is linear on $L^1(\mu )$: for integrable functions, the integral commutes with differences and scalar multiplication.

Given: AC; an open scalar parameter interval and a fixed $\theta \in \mathrm{\Theta }$; a measurable real estimator $T$ with ${\mathbb{E}}_{\eta }T=\psi (\eta )$ locally, finite ${\mathbb{E}}_{\theta }{T}^2$, and differentiable $\psi$; a score $S_{\theta }$ with ${\mathbb{E}}_{\theta }{S}_{\theta }=0$, $I(\theta )={\mathbb{E}}_{\theta }{S}_{\theta }^2\in (0,\mathrm{\infty })$; and the score-covariance identity ${\mathrm{Cov}}_{\theta }(T,S_{\theta })={\psi }^{\mathrm{\prime }}(\theta )$. Alternatively, assume the sufficient common-support density and integrable-envelope route stated above.

The Axiom of Choice: AC says every family of nonempty sets has a choice function; it supplies the density representatives used by the score definition.

Score and Fisher information: on positive support, $S_{\theta }={\mathrm{\partial }}_{\theta }\log p_{\theta }$ and $I(\theta )={\mathbb{E}}_{\theta }{S}_{\theta }^2$.

Mean-zero score and the information identity: under the displayed common-support and derivative-envelope hypotheses, the score has mean zero and the information identity holds.

Statistics and estimators: an estimator is a measurable statistic with the target measurable space specified.

Moments, variance, and covariance on a probability space: for square-integrable real random variables, variance is $\mathbb{E}[(T-\mathbb{E}T{)}^2]$ and covariance is $\mathbb{E}[(T-\mathbb{E}T)(S-\mathbb{E}S)]$.

The space $L^p(\mu )$ as the quotient by null functions and Measure-null sets and almost-everywhere statements relative to a measure: $L^2(P_{\theta })$ consists of square-integrable real functions modulo equality $P_{\theta }$-almost everywhere.

The Lebesgue integral is linear on $L^1(\mu )$ and A nonnegative measurable function has integral $0$ exactly when it vanishes almost everywhere: expectations are linear on integrable functions, and a nonnegative function with integral zero vanishes almost everywhere.

Real and complex inner product spaces, with the inner product linear in the first argument: a real inner product is a symmetric bilinear positive-definite pairing.

Cauchy–Schwarz: $\mathrm{\mid }⟨u,v⟩\mathrm{\mid }\le \parallel u\parallel \parallel v\parallel$, with equality exactly for linearly dependent vectors: in a real inner product space, Cauchy–Schwarz holds, with equality exactly when the two vectors are linearly dependent.

The mean value theorem, as the case $g(x)=x$ of Cauchy's: for $f$ continuous on $[a,b]$ with $a<b$ and differentiable on $(a,b)$ there is $c\in (a,b)$ with $f(b)-f(a)=f^{\mathrm{\prime }}(c)(b-a)$ and Dominated convergence: a differentiable parameter function has its difference quotient bounded by a derivative bound on the segment, and an integrable majorant permits the limit to pass through the integral.

Likelihood densities: $p_{\eta }$ is a selected RN density representative of the sampling law.

A sigma-finite signed measure that is absolutely continuous with respect to a sigma-finite positive measure has a unique almost-everywhere density: the selected RN density satisfies $P_{\eta }(A)={\int }_A{p}_{\eta }d\mu$.

The measure with density $f$ relative to $\mu$ and Integrating against a density agrees with integrating the product and Integrable real and complex functions, and their integrals: for the density law $P_{\eta }=p_{\eta }d\mu$, the density identity holds for nonnegative $g$. For real integrable $g$, applying the nonnegative identity to $g^{+}$ and $g^{-}$ gives the signed identity; applying it to $\mathrm{\mid }g\mathrm{\mid }$ verifies that $g{p}_{\eta }$ is integrable.

Unbiased estimators and UMVU estimators: unbiasedness means integrability and ${\mathbb{E}}_{\eta }T=\psi (\eta )$ at every parameter; finite-variance unbiasedness also requires $T\in L^2(P_{\eta })$ at every parameter.

Given: AC; an open $d$-dimensional parameter set; a measurable ${\mathbb{R}}^k$-valued estimator $T$ with finite second moments and mean $\psi (\theta )$; a square-integrable score vector $S_{\theta }$ with mean zero and nonsingular $I(\theta )={\mathbb{E}}_{\theta }[S_{\theta }{S}_{\theta }^{\mathsf{T}}]$; the vector derivative identity; and the stated quadratic-form definition of $⪰$.

The Axiom of Choice: AC supplies the arbitrary-index family of selected density representatives used by the score.

Score and Fisher information: the score is the gradient of the log density and $I(\theta )$ has entries ${\mathbb{E}}_{\theta }[S_{\theta ,i}{S}_{\theta ,j}]$.

Mean-zero score and the information identity: under its regularity hypotheses, the score mean is zero and the information matrix is its second-moment matrix.

Statistics and estimators: an estimator is a measurable statistic with its target measurable space specified.

Moments, variance, and covariance on a probability space: covariance is the expectation of centered coordinate products; for a vector, this defines the covariance matrix entrywise.

Expectation of a nonnegative or integrable random variable: expectation is integration against the sampling probability measure.

Unbiased estimators and UMVU estimators, applied to each real coordinate: if $T_a$ is finite-variance unbiased for ${\psi }_a$, then ${\mathbb{E}}_{\eta }{T}_a={\psi }_a(\eta )$ and ${\mathbb{E}}_{\eta }{T}_a^2<\mathrm{\infty }$ at every parameter.

The Lebesgue integral is linear on $L^1(\mu )$: expectation is linear on integrable real random variables. Finite second moments imply first moments on a probability space (use $\mathrm{\mid }z\mathrm{\mid }\le (1+z^2)\mathrm{/}2$), and all products of coordinates in the covariance expansion are integrable by $2\mathrm{\mid }YZ\mathrm{\mid }\le Y^2+Z^2$.