Generalized linear models and Fisher scoring — Examples

Given: The scalar design $X=[1]$, response $y_1=1$, current $\beta =0$, and the Bernoulli-logit family and link in the Example.

Generalized linear model with explicit conditioning: the canonical link is $g=\tau$, equivalently ${\theta }_i={\eta }_i$.

Canonical-link score and Hessian: under a common scale the score is $X^{\mathsf{T}}(y-\mu )\mathrm{/}a(\varphi )$.

General-link score and Fisher information: the expected score outer-product information has weights $w_i={\dot{\mu }}_i^2\mathrm{/}(a(\varphi )V_i)$.

One minimum-norm IRLS step: the working response and weights are $z_i={\eta }_i+(y_i-{\mu }_i)\mathrm{/}{\dot{\mu }}_i$ and $w_i={\dot{\mu }}_i^2\mathrm{/}(a(\varphi )V_i)$.

IRLS equals the Fisher-scoring step when information is invertible: when information is invertible, ${\beta }_{\mathrm{n}\mathrm{e}\mathrm{w}}=\beta +I(\beta {)}^{-1}U(\beta )$.

The Bernoulli-logit component mass on $\{0,1\}$ relative to counting measure is $f(y;\eta )=\exp \{y\eta -\log (1+e^{\eta })\}$. It sums to one; its support is fixed, its second moment is finite, and its first two $\eta$ derivatives are bounded on the finite support. (Bernoulli random variables and binomial random variables as sums of independent Bernoulli trials, The exponential function is smooth and $(\exp {)}^{\mathrm{\prime }}=\exp$, The natural logarithm has derivative 1/x and equals the integral from 1 to x of 1/t)

Given: The scalar design $X=[1]$, response $y_1=2$, current $\beta =0$, and the Poisson-log family and link in the Example.

Generalized linear model with explicit conditioning: the canonical link is $g=\tau$, equivalently ${\theta }_i={\eta }_i$.

Mean and variance from the cumulant function: the response mean is $b^{\mathrm{\prime }}({\theta }_i)$ and its variance is $a(\varphi )b^{\mathrm{\prime }\mathrm{\prime }}({\theta }_i)$.

General-link score and Fisher information: the expected score outer-product information has weights $w_i={\dot{\mu }}_i^2\mathrm{/}(a(\varphi )V_i)$.

One minimum-norm IRLS step: the working response and weights are $z_i={\eta }_i+(y_i-{\mu }_i)\mathrm{/}{\dot{\mu }}_i$ and $w_i={\dot{\mu }}_i^2\mathrm{/}(a(\varphi )V_i)$.

IRLS equals the Fisher-scoring step when information is invertible: when information is invertible, ${\beta }_{\mathrm{n}\mathrm{e}\mathrm{w}}=\beta +I(\beta {)}^{-1}U(\beta )$.

The Poisson response has support ${\mathbb{N}}_0$, counting base measure, and $f(k;\theta )=e^{-e^{\theta }}{e}^{k\theta }\mathrm{/}k!$. The exponential series and addition formula normalize the masses. Its first two $\theta$ derivatives are $f(k-e^{\theta })$ and $f\{(k-e^{\theta }{)}^2-e^{\theta }\}$. On any compact interval $[u,v]$ these are bounded by the summable sequence $e^{vk}\{1+(k+e^v{)}^2+e^v\}\mathrm{/}k!$; the ratio test gives summability. Reindexing gives $\mathbb{E}[Y^2]={\lambda }^2+\lambda <\mathrm{\infty }$ for $\lambda =e^{\theta }$. Thus the fixed-support, finite-moment and derivative-envelope hypotheses hold, and the component score has finite second moment. (The real exponential function and the number $e$ by a power series, The exponential addition formula $\exp (x+y)=\exp (x)\exp (y)$, The factorial $n!$ and the falling factorial $n^{\underset{‾}{k}}$, defined by recursion in $\mathbb{N}$, The exponential function is smooth and $(\exp {)}^{\mathrm{\prime }}=\exp$)

Given: The design column $X=(-1,1{)}^{\mathsf{T}}$, responses $(0,1)$, the Bernoulli cumulant $b(\eta )=\log (1+e^{\eta })$, and $\beta \in \mathbb{R}$.

Generalized linear model with explicit conditioning: the canonical link is the choice $g=\tau$, equivalently ${\theta }_i={\eta }_i$.

Canonical-link score and Hessian: the canonical score is $X^{\mathsf{T}}(y-\mu )$ when $a(\varphi )=1$.

Strict concavity and uniqueness of a finite stationary maximizer: positive $b^{\mathrm{\prime }\mathrm{\prime }}$ and full column rank imply strict concavity, but the conclusion does not assert existence of a finite maximizer.

The Bernoulli-logit component mass on $\{0,1\}$ relative to counting measure is $f(y;\eta )=\exp \{y\eta -\log (1+e^{\eta })\}$. It sums to one; the support is fixed, the response is bounded, and its first two $\eta$ derivatives are locally bounded because $f^{\mathrm{\prime }}=f(y-p)$ and $f^{\mathrm{\prime }\mathrm{\prime }}=f\{(y-p{)}^2-p(1-p)\}$ with $p=e^{\eta }\mathrm{/}(1+e^{\eta })$. (Bernoulli random variables and binomial random variables as sums of independent Bernoulli trials, The exponential function is smooth and $(\exp {)}^{\mathrm{\prime }}=\exp$, The natural logarithm has derivative 1/x and equals the integral from 1 to x of 1/t)

Given: The two-point count law in the Example and a comparison Poisson GLM with $a(\varphi )=1$, $b(\theta )=e^{\theta }$, and $\theta =\log 10$.

Generalized linear model with explicit conditioning: a GLM pairs a response family with a link $g({\mu }_i)={\eta }_i=x_i^{\mathsf{T}}\beta$ and a natural parameter map satisfying $b^{\mathrm{\prime }}({\theta }_i)={\mu }_i$.

Mean and variance from the cumulant function: for the stated family, mean is $b^{\mathrm{\prime }}(\theta )$ and variance is $a(\varphi )b^{\mathrm{\prime }\mathrm{\prime }}(\theta )$.

For the comparison Poisson response, the support is ${\mathbb{N}}_0$, the base measure is counting measure, and $f(k;\theta )=e^{-e^{\theta }}{e}^{k\theta }\mathrm{/}k!$. The exponential series and addition formula normalize these masses. Its first two parameter derivatives are ${\mathrm{\partial }}_{\theta }f(k;\theta )=f(k;\theta )(k-e^{\theta })$ and ${\mathrm{\partial }}_{\theta }^{2}f(k;\theta )=f(k;\theta )\{(k-e^{\theta }{)}^2-e^{\theta }\}$. On any compact interval $[u,v]$ they are dominated by the summable sequence $e^{vk}\{1+(k+e^v{)}^2+e^v\}\mathrm{/}k!$; the ratio test gives summability. Reindexing the same exponential series gives $\mathbb{E}[Y^2]={\lambda }^2+\lambda <\mathrm{\infty }$ for $\lambda =e^{\theta }$, so the family satisfies the fixed-support, moment, and derivative-envelope hypotheses. (The real exponential function and the number $e$ by a power series, The exponential addition formula $\exp (x+y)=\exp (x)\exp (y)$, The factorial $n!$ and the falling factorial $n^{\underset{‾}{k}}$, defined by recursion in $\mathbb{N}$, The exponential function is smooth and $(\exp {)}^{\mathrm{\prime }}=\exp$)

Given: The Bernoulli cumulant $b(\theta )=\log (1+e^{\theta })$, $x_1=1$, identity link $\eta =\mu =\beta$, and the closure boundary $\mu \downarrow 0$.

Generalized linear model with explicit conditioning: ${\mu }_i=g^{-1}({\eta }_i)$ and ${\theta }_i=\tau ({\mu }_i)$, where $\tau$ satisfies $b^{\mathrm{\prime }}(\tau (\mu ))=\mu$.

General-link score and Fisher information: $w={\dot{\mu }}^2\mathrm{/}(a(\varphi )V(\mu ))$ where $V=b^{\mathrm{\prime }\mathrm{\prime }}(\theta )$.

One minimum-norm IRLS step: the scoring step requires finite positive weights and a finite nonzero link derivative at the current iterate.

In this Bernoulli GLM, $Y_i\in \{0,1\}$ and $p(\theta )=e^{\theta }\mathrm{/}(1+e^{\theta })$; relative to counting measure the mass is $f(y;\theta )=\exp \{y\theta -\log (1+e^{\theta })\}$. It is normalized by summing over $y=0,1$. Writing the natural-parameter derivatives as $f^{\mathrm{\prime }}=f(y-p)$ and $f^{\mathrm{\prime }\mathrm{\prime }}=f\{(y-p{)}^2-p(1-p)\}$, the first two derivatives have constant integrable bounds on the two-point support, and $\mathbb{E}[Y^2]\le 1$. Thus the fixed-support, finite-moment, derivative-envelope, and score-moment hypotheses used above hold. (Bernoulli random variables and binomial random variables as sums of independent Bernoulli trials, The exponential function is smooth and $(\exp {)}^{\mathrm{\prime }}=\exp$, The natural logarithm has derivative 1/x and equals the integral from 1 to x of 1/t)

Given: The two-row design and responses in the Example, the Bernoulli-logit family, current $\beta =0$, and $a(\varphi )=1$.

General-link score and Fisher information: $I=X^{\mathsf{T}}\mathrm{diag}(w_i)X$ and $U=X^{\mathsf{T}}({\dot{\mu }}_i(y_i-{\mu }_i)\mathrm{/}(a{V}_i){)}_i$.

One minimum-norm IRLS step: IRLS minimizes the weighted least-squares objective and applies the Moore-Penrose minimum-norm convention if minimizers are not unique.

Each Bernoulli-logit response has the counting-measure mass $f(y;\eta )=\exp \{y\eta -\log (1+e^{\eta })\}$ on $\{0,1\}$. It is normalized, has fixed support and finite moments, and its first two $\eta$ derivatives are locally bounded on this finite support. (Bernoulli random variables and binomial random variables as sums of independent Bernoulli trials, The exponential function is smooth and $(\exp {)}^{\mathrm{\prime }}=\exp$, The natural logarithm has derivative 1/x and equals the integral from 1 to x of 1/t)