LemmaStatement: AI-adaptedProof: AI-adaptedPipeline-generatedjudge pass (gpt-6-luna)

How statement and proof provenance work

The first chip identifies the source of the statement or construction; the second identifies the source of its local proof or verification.

Literature-sourced: the exact statement appears in a cited source; only wording and notation differ.
AI-adapted: a semantically identical restatement of literature-sourced material, modulo indexing, notation, and boundary cases adopted by the library.
AI-generated: a genuinely novel statement formulated by AI, with no source for the claim itself.

These labels describe origin, not correctness: citations and verification chips remain separate evidence.

Second moments of symmetric rectifiers

Statement

Let $Z$ be a real random variable whose law is symmetric about zero, meaning that $Z$ and $- Z$ have the same distribution. Write $p_{0} = P (Z = 0)$ . Define the ReLU function $ρ (t) = \max (t, 0)$ and, for $a \in R$ , define $ϕ_{a} (t) = t$ for $t > 0$ and $ϕ_{a} (t) = a t$ for $t \leq 0$ . Then $P (Z > 0) = P (Z < 0) = (1 - p_{0}) / 2$ . If $E [Z^{2}] < \infty$ , the following ReLU moment identities hold: $E [ρ (Z)] = \frac{1}{2} E [∣ Z ∣], E [ρ (Z)^{2}] = \frac{1}{2} E [Z^{2}] .$ The corresponding PReLU identities are $E [ϕ_{a} (Z)] = \frac{1 - a}{2} E [∣ Z ∣], E [ϕ_{a} (Z)^{2}] = \frac{1 + a^{2}}{2} E [Z^{2}] .$ In particular, $Var (ρ (Z)) = \frac{1}{2} E [Z^{2}] - \frac{1}{4} (E [∣ Z ∣])^{2} .$ For any specified value $c \in R$ at the kink, put $η_{a, c} (t) = 1_{t > 0} + a 1_{t < 0} + c 1_{t = 0}$ . Then, without a moment hypothesis on $Z$ , $E [η_{a, c} (Z)^{2}] = \frac{1 + a^{2}}{2} (1 - p_{0}) + c^{2} p_{0} .$ The zero-at-kink ReLU backpropagation gate is $η_{0, 0}$ , with second moment $(1 - p_{0}) / 2$ . It agrees almost surely with the ordinary ReLU derivative if $p_{0} = 0$ . A specified value at zero is a backpropagation convention, and is not an ordinary derivative there unless $a = 1$ and $c = 1$ .

Facts & Assumptions

Given: A symmetric real random variable $Z$ ; finite second moment for the activation-moment conclusions; and the specified finite gate values $a, c$ .

[F1]

Expectations of measurable functions of a random element are integrals against its law. (Change of variables for expectation)

[F2]

Expectation is the Lebesgue integral on the probability space. (Expectation of a nonnegative or integrable random variable)

[F3]

Variance is $Var (Y) = E [(Y - E [Y])^{2}]$ for square-integrable $Y$ . (Moments, variance, and covariance on a probability space)

[F4]

A square-integrable random variable has integrable products by Cauchy–Schwarz; in particular $E [∣ Z ∣] \leq (E [Z^{2}])^{1 / 2}$ . (Cauchy-Schwarz for random variables)

[F5]

Expectation is linear in scalar multiples and binary sums of integrable random variables; repeated use gives the finite-sum rule. (Linearity, monotonicity, and the modulus bound for expectation)

Proof

1.1givenF1F2

Symmetry gives $P (Z > 0) = P (Z < 0)$ , and these probabilities sum to $1 - p_{0}$ , so each is $(1 - p_{0}) / 2$ . The gate takes the values $1, a, c$ on the positive, negative, and zero events, respectively, so its squared expectation is $((1 + a^{2}) / 2) (1 - p_{0}) + c^{2} p_{0}$ . For the remaining moment claims assume $E [Z^{2}] < \infty$ . Applying [F1] and [F2] to $t \mapsto t^{2} 1_{t > 0}$ and $t \mapsto t^{2} 1_{t < 0}$ shows that the positive-side and negative-side second moments are equal. Their sum is $E [Z^{2}]$ , since the contribution at zero is zero, so each is $\frac{1}{2} E [Z^{2}]$ .

1.2givenF1F4F5algebra

By [F1], symmetry gives $E [Z] = E [- Z]$ , while scalar linearity [F5] gives $E [- Z] = - E [Z]$ , hence $E [Z] = 0$ . The expectation $E [∣ Z ∣]$ is finite by [F4]. The positive and negative parts of $Z$ have equal expectations by symmetry and sum to $∣ Z ∣$ , so linearity [F5] gives each expectation as $\frac{1}{2} E [∣ Z ∣]$ . Therefore $E [ρ (Z)] = \frac{1}{2} E [∣ Z ∣]$ and $E [ϕ_{a} (Z)] = E [Z 1_{Z > 0}] + a E [Z 1_{Z < 0}] = \frac{1 - a}{2} E [∣ Z ∣]$ , using [F5] for the sum and scalar factor.

2.1step 1.1algebra

The identities $ρ (Z)^{2} = Z^{2} 1_{Z > 0}$ and $ϕ_{a} (Z)^{2} = Z^{2} 1_{Z > 0} + a^{2} Z^{2} 1_{Z < 0}$ now give the two asserted second moments from step 1.1.

3.1step 2.1step 1.2F3F5algebra∎

Expanding the variance definition [F3] using expectation linearity [F5] gives $Var (ρ (Z)) = E [ρ (Z)^{2}] - (E [ρ (Z)])^{2}$ . Substitution of steps 2.1 and 1.2 proves the stated centered-variance identity and completes all formulas.

Depends on

Used by

Dependency tree · two levels

22 results within two dependency steps of this one, each drawn at its shortest distance from it. An arrow runs from a result to what uses it, so the chart reads left to right and ends at this result, which carries a heavier outline. Every node is a link to that result. Click elsewhere on the chart to enlarge it.

Sources

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, Delving Deep into Rectifiers, §2.2 (standard reference, not scraped)
Simon J. D. Prince, Understanding Deep Learning, §7.5.1–7.5.2 (standard reference, not scraped)
Johns Hopkins University, Handout #8: Neural Network Training and Tokenization, §3 (standard reference, not scraped)