How statement and proof provenance work
The first chip identifies the source of the statement or construction; the second identifies the source of its local proof or verification.
- Literature-sourced: the exact statement appears in a cited source; only wording and notation differ.
- AI-adapted: a semantically identical restatement of literature-sourced material, modulo indexing, notation, and boundary cases adopted by the library.
- AI-generated: a genuinely novel statement formulated by AI, with no source for the claim itself.
These labels describe origin, not correctness: citations and verification chips remain separate evidence.
Reverse-mode work is a fixed multiple of primal work under explicit primitive-cost bounds
Statement
For a finite graph on a fixed strict branch, fix a finite set of scalar/vector primitive types with at most parent slots per operation and bounded dimensions. Suppose each operation has primal work , its local vector-Jacobian product costs at most operations, and accumulating each parent-slot contribution costs at most operations. Assume each invocation's retained local tape and peak pullback scratch use at most a fixed number of scalar entries. Let be the scalar storage in the primal tape and input-gradient buffer, counting all input and node-value coordinates, per-node and per-parent metadata, branch labels, and retained local-tape entries. Then the exact reverse-mode algorithm for a scalar output uses at most operations for the primal evaluation, reverse sweep, and parent accumulations, and stores scalar entries. Opaque solver calls are excluded unless their primal and derivative costs satisfy the same hypotheses.
Facts & Assumptions
Given: A finite graph and the stated primitive-cost model, with .
Finite straight-line computational programs and real-arithmetic tapes: there are finitely many topologically ordered nodes and the primal tape stores their values and parent slots.
Reverse-mode automatic differentiation for a scalar graph output: reverse mode evaluates each primitive once in the primal pass, then visits each node once and applies one local vector-Jacobian product per node while accumulating a contribution for each parent slot.
The primitive set has fixed finite type and bounded dimensions. Local primal, pullback, and parent-accumulation costs obey the constants in the statement; an opaque solver call has no uncharged derivative cost.
The operation count is arithmetic work; allocating or zero-initializing the cotangent buffers is counted in storage rather than arithmetic operations.
Each primitive invocation's retained local tape and peak pullback scratch use a uniformly bounded number of scalar entries; parent-slot temporary vectors have bounded dimensions because the primitive types and arities are fixed.
Proof
The primal pass evaluates each node once, so its cost is . The reverse pass makes one local pullback call at node , costing at most , and handles at most parent slots, costing at most arithmetic operations. Since , this node's reverse arithmetic work is at most . Summing over all nodes gives total arithmetic work at most ; cotangent-buffer initialization is storage under [A2].
The primal tape, per-node local tapes, metadata, and input-gradient entries are counted in . Reverse cotangents require one vector of the corresponding dimension per node; all these dimensions are among the node/input-value coordinates counted in . At each node, [A3] bounds local scratch and parent-slot temporaries by a constant, so their peak storage is . Hence total storage is ; the bound says nothing about an opaque solver whose local tape or pullback cost was excluded in [A1].
Remarks
Baur–Strassen proves a separate nonscalar arithmetic-circuit complexity theorem over an infinite field. The local bound above is proved directly for a fixed smooth primitive set with explicit primal/pullback cost ratios; it does not extend the arithmetic-circuit theorem to arbitrary opaque procedures or claim that every differentiable primitive has a cheap derivative.
Depends on
Used by
Nothing in the library uses this result yet.
Dependency tree · two levels
4 results within two dependency steps of this one, each drawn at its shortest distance from it. An arrow runs from a result to what uses it, so the chart reads left to right and ends at this result, which carries a heavier outline. Every node is a link to that result. Click elsewhere on the chart to enlarge it.
Sources
- MIT OpenCourseWare 18.S096, Matrix Calculus for Machine Learning and Beyond, complete lecture notes, Chapter 8 §§8.3–8.4 (standard reference, not scraped)
- W. Baur and V. Strassen, The Complexity of Partial Derivatives, Theoretical Computer Science 22 (1983), §§2 and 4 (standard reference, not scraped)