Alphabeta Math

Neural Networks and Learning Dynamics

56 pages in 5 parts

This collection studies neural architectures, their approximation properties and the mathematical behavior of training. It begins with feedforward and ReLU networks, convolution and equivariance, graph message passing, recurrent and state-space models, attention and neural operators. Architecture-specific results distinguish representational capacity from trainability and predictive performance. Selected risk, complexity and kernel results from Statistical Learning and Classical Models provide prerequisites without requiring that entire collection to be read first.

Automatic differentiation leads to nonconvex gradient descent, stochastic training, momentum, adaptive methods and initialization. Each algorithm retains its precise update convention, smoothness assumptions and cost model. Exact differentiation, a finite execution trace and convergence of an optimization procedure are separate claims. The later efficient-training part covers implicit differentiation, tiled attention and low-rank parameter updates.

Training theory connects implicit bias, interpolation, double descent, neural tangent kernels and mean-field learning. The sample-covariance prerequisite pairs belong with the interpolation results they support. Wasserstein geometry and gradient flows similarly precede mean-field dynamics. Norm, margin, compression and stability arguments then address generalization under their distinct hypotheses. These advanced branches can be read selectively. Representation Learning and Generative Models uses the relevant neural foundations; preference optimization in Online and Reinforcement Learning also draws on differentiation.

Pathway

The parts run in order. Everything a page needs from this group has been read by the time you reach it, and the level on each row is how many dependency steps into the group that page sits.

  1. Part 1 · Architectures and expressivity

    7 pages

    Begin with feedforward approximation, then study depth, equivariance, graph message passing, recurrence, attention and neural operators. Read the architecture and topology assumptions literally. Expressivity results do not by themselves establish an effective training method or a generalization guarantee.

  2. Part 2 · Differentiation, optimization and initialization

    6 pages · after Part 1

    Backpropagation supplies the differentiation interface for gradient-based training. Continue through nonconvex descent, stochastic gradients, momentum and adaptive methods, with initialization as a separate signal-propagation problem. Preserve each algorithm convention and its stated finite-iteration or asymptotic guarantee.

  3. Part 3 · Implicit bias, interpolation and kernel limits

    6 pages · after Parts 1 and 2

    Study implicit bias and overparameterized interpolation alongside their linear and kernel prerequisites. Read the Marchenko-Pastur and sample-covariance resolvent pairs before double descent. Neural tangent kernels describe a particular training regime and should not be treated as a universal account of feature learning.

  4. Part 4 · Mean-field dynamics and generalization

    6 pages · after Part 1

    Read Wasserstein geometry before the gradient-flow existence and stability pair, then apply that interface to mean-field training. Norm, margin, compression and stochastic stability provide separate generalization arguments. Their classical risk and federated-learning prerequisites remain in Statistical Learning and Classical Models.

  5. Part 5 · Efficient training and adaptation

    3 pages · after Parts 1 and 2

    Bilevel learning follows differentiation and its implicit-function prerequisites. Tiled attention and low-rank adaptation follow the attention and backpropagation pages. These methods distinguish exact algebraic identities from approximate solves, finite-precision effects and empirical claims about adaptation quality.