Alphabeta Math

Online and Reinforcement Learning

42 pages in 5 parts

Online learning studies sequences of decisions, feedback and updates. Its guarantees compare cumulative performance with an explicit benchmark, often through regret. Reinforcement learning adds a state whose future evolution can depend on the chosen actions. This collection connects these settings through their information and decision structures. Online refers to sequential interaction, not an internet connection. Probability, martingales, optimization and selected classical prediction methods supply external prerequisites.

The pathway starts with online protocols, Hedge, gradient methods, mirror descent and confidence sequences under adaptive sampling. It then develops stochastic, adversarial and contextual bandits, where the learner receives only the feedback allowed by the protocol. Confidence statements must retain their sampling and stopping assumptions. Regret and sample complexity describe different guarantees and are not interchangeable.

Finite MDPs and Bellman operators lead to tabular prediction, control and exploration. Later pages cover policy gradients, function approximation, model-based planning, offline evaluation, partial observability, imitation and constrained policies. Their assumptions determine which conclusions survive approximation or distribution mismatch. The final part connects preference learning, direct preference optimization and verifiable generation with policy objectives. Most of the collection can be read after the relevant classical and probability foundations; the preference branch additionally uses differentiation from Neural Networks and Learning Dynamics. Generative modeling is a parallel pathway rather than a prerequisite for every decision method.

Pathway

The parts run in order. Everything a page needs from this group has been read by the time you reach it, and the level on each row is how many dependency steps into the group that page sits.

  1. Part 1 · Online protocols, regret and adaptive confidence

    4 pages

    Begin with the decision and feedback protocol, then read Hedge, online gradient descent and mirror descent. Confidence sequences provide a complementary route to uncertainty under adaptive sampling. Keep the comparator class, stopping convention and feedback model explicit.

  2. Part 2 · Bandit feedback and contextual decisions

    4 pages · after Part 1

    Move from stochastic finite-armed bandits to adversarial and contextual feedback. Contextual expert methods follow importance weighting; linear contextual bandits use the earlier linear-prediction interface. The learner observes only the feedback specified by each protocol.

  3. Part 3 · MDPs, Bellman operators and tabular learning

    4 pages

    Read finite MDPs and Bellman operators before tabular prediction and control. Optimistic exploration and average-reward methods add their own horizon, communication and confidence assumptions. Planning with a known model and learning through interaction are distinct operations.

  4. Part 4 · Policies, planning, offline learning and safety

    7 pages · after Parts 1 and 3

    Continue through policy gradients, approximation limits, model-based planning and offline evaluation. Partial observability and imitation modify the information or training protocol; constrained MDPs add explicit policy-safety conditions. Read each tabular or Bellman prerequisite before its extension rather than assuming deep approximation preserves the guarantee.

  5. Part 5 · Preferences and verifiable generation

    2 pages · after Part 4

    Preference learning and direct preference optimization use classical calibration and classification together with neural differentiation. Verifiable generation and group-relative updates follow policy-gradient and offline-evaluation interfaces. These objective and sampling results do not by themselves establish broad model alignment or reliable verification outside the stated setting.