Reinforcement learning

My notes tracing reinforcement learning from Sutton and Barto's first principles through to the policy alignment methods that train reasoning LLMs. Seven chapters, in the order the ideas actually build on each other: the problem, then the formal framework, then two different ways of solving it, then what happens when the state space is a sequence of tokens.

Written for me, so the emphasis is on why each method exists rather than on covering everything. Entries in grey are notes I have not written yet.

The problem

• The RL problem: what are we even doing?the agent-environment loop, the four elements, and a tic-tac-toe agent that learns without a model
• Bandits and the exploration problemexploration versus exploitation with everything else stripped away; ε-greedy and the 10-armed testbed

The formal framework

• MDPs, value functions and Bellman equationsthe Markov property, returns and discounting, and the Bellman equation derived in full
• Partially observable MDPs

Solving it when you know the world

• Dynamic programmingpolicy evaluation, improvement and iteration, value iteration, and why all of it fails at scale

Solving it from experience

• Learning from experience: MC, TD and controlMonte Carlo against temporal difference, SARSA and Q-learning, then function approximation
• Eligibility traces and TD(λ)
• Deep Q-networks

Optimising the policy directly

• Policy gradientsthe policy gradient theorem, REINFORCE with a baseline, actor-critic, GAE, and the path from TRPO to PPO
• Deterministic policy gradients and SAC

RL for language models

• RL meets LLMs: PPO, GRPO and reasoninggeneration as an MDP, reward modelling and RLHF, PPO's four-model memory bill, and how GRPO removes the critic
• DPO and the direct-preference family
• Process reward models

Reference

• Concept maphow the chapters connect
• Method evolution treewhich algorithm descends from which, and what each one bought

Prerequisites: calculus (derivatives and the chain rule), linear algebra, probability (expectations and distributions), and basic Python.