Build Agents That Learn by Trial and Error with Python, Gymnasium, and PyTorch

Read before you buy: the PDF here is a free sample — the first 20 pages of the 434-page book (payments, sales and reviews on this demo site are sample data). Get the full e-book at thenexbooks on Gumroad.

Understand reinforcement learning properly — and build it yourself, one runnable step at a time.

Reinforcement learning teaches programs to make good decisions by trial and error: instead of writing the rules, you describe what you want and let an agent discover how to get it. The same idea has mastered games, controls simulated robots, and helps align large language models. It also has a reputation for dense notation, fragile training runs, and tutorials that jump straight to neural networks.

This book takes the opposite path. Every idea is explained in plain words first, then written as a formula, then turned into short Python code that you run yourself. We start with a problem small enough to solve exactly, so you can always check whether an algorithm really learned — and only then move on to deep reinforcement learning with PyTorch.

What's inside

• What reinforcement learning is, when it is the right tool, and when it is not
• Python, NumPy, Gymnasium, and PyTorch set up for RL, with reproducible seeds and CPU-friendly runs
• Multi-armed bandits: ε-greedy, optimistic starts, UCB, Thompson sampling, and regret
• Markov decision processes, returns, discounting, and the Bellman equations in plain language
• Building and testing a custom Gymnasium environment from scratch
• Dynamic programming: policy evaluation, policy iteration, and value iteration with exact results
• Monte Carlo methods, TD learning, SARSA, Q-learning, Expected SARSA, Double Q-learning, and n-step methods
• Function approximation, PyTorch essentials, and deep Q-networks with replay buffers and target networks
• Double DQN, debugging techniques, and evaluating agents honestly across seeds
• Policy gradients: REINFORCE, baselines, actor-critic, and PPO written from scratch
• Continuous actions with Gaussian policies, plus DDPG, TD3, and SAC explained
• Vector environments, wrappers, normalization, and hyperparameter tuning
• Reward design and shaping, reward hacking, sparse rewards, exploration, and curriculum learning
• Model-based RL with Dyna-Q, imitation learning, and offline RL
• How RLHF, reward models, and preference optimization shape language models
• Safe reinforcement learning, sim-to-real transfer, serving a policy, monitoring, and staged rollout

One running example, start to finish. You train Tally, a small warehouse robot at Thornbury Fulfillment, a fictional fulfillment center. Tally must carry totes from the loading dock to the packing station without driving onto a wet floor, on a floor slick enough that it sometimes slides sideways. The warehouse is small enough to solve exactly, so you see real lessons with real numbers: a slippery floor changes the best route, some algorithms learn risky shortcuts while others learn caution, a badly designed reward makes the robot circle forever, and learning only from logs fails when the logs never show the rare accident.

Current and honest. The code uses Python, NumPy, Gymnasium 1.x, and PyTorch, and every training run is small enough to finish in minutes on an ordinary CPU with fixed seeds — no GPU required. Every printed result comes from running the book's code. The book contains no invented benchmarks or statistics. Every chapter includes runnable code, Tips, Notes, Warnings, exercises, and Key Takeaways, and the appendices give you a cheat sheet, templates and reference tables, a production checklist, and a glossary.

Format: PDF, 434 pages (A4), linked table of contents and bookmarks. English. First Edition, 2026. Published by thenexbooks. This book is independent and is not affiliated with, sponsored by, or endorsed by the Python Software Foundation, the PyTorch Foundation, The Linux Foundation, or the Farama Foundation.

Table of Contents

Part I — Foundations

• Ch 1 · What Reinforcement Learning Is: Agents, Rewards, and Learning by Trial and Error
• Ch 2 · Your RL Toolkit: Python, NumPy, Gymnasium, and PyTorch
• Ch 3 · Multi-Armed Bandits: Exploration versus Exploitation
• Ch 4 · Markov Decision Processes: States, Actions, Rewards, and Returns
• Ch 5 · Building the Thornbury Warehouse: A Custom Gymnasium Environment

Part II — Planning and Tabular Learning

• Ch 6 · Bellman Equations: Values, Q-Values, and Optimality
• Ch 7 · Dynamic Programming: Policy Evaluation, Policy Iteration, and Value Iteration
• Ch 8 · Monte Carlo Methods: Learning from Complete Episodes
• Ch 9 · Temporal-Difference Learning: TD(0), SARSA, and Q-Learning
• Ch 10 · Beyond One Step: Expected SARSA, Double Q-Learning, and n-Step Returns

Part III — Function Approximation and Deep Q-Learning

• Ch 11 · Function Approximation: From Tables to Features
• Ch 12 · PyTorch for Reinforcement Learning: Networks, Losses, and Optimizers
• Ch 13 · Deep Q-Networks: Replay Buffers and Target Networks
• Ch 14 · Making DQN Work: Double DQN, Dueling Networks, Prioritized Replay, and Debugging
• Ch 15 · Evaluating Agents Honestly: Seeds, Learning Curves, and Fair Comparisons

Part IV — Policy Gradient Methods

• Ch 16 · Policy Gradients and REINFORCE
• Ch 17 · Baselines and Actor-Critic Methods
• Ch 18 · Proximal Policy Optimization from Scratch
• Ch 19 · Continuous Actions: Gaussian Policies, DDPG, TD3, and SAC
• Ch 20 · Practical Training: Vectorized Environments, Normalization, Wrappers, and Hyperparameters

Part V — Designing RL Systems

• Ch 21 · Reward Design and Shaping: Getting the Behavior You Asked For
• Ch 22 · Exploration, Sparse Rewards, and Curriculum Learning
• Ch 23 · Model-Based RL and Planning: Dyna-Q and Search
• Ch 24 · Learning from Logged Data: Imitation and Offline RL

Part VI — RL in the Real World

• Ch 25 · RL for Language Models: RLHF, Reward Models, and Preference Optimization
• Ch 26 · Safe Reinforcement Learning: Constraints, Shielding, and Safety Testing
• Ch 27 · From Simulation to Deployment: Sim-to-Real, Libraries, Serving, and Monitoring
• Ch 28 · Case Study: Taking Tally from Gridworld to a Pilot Rollout

Appendices
A · Reinforcement Learning Cheat Sheet
B · Templates and Reference
C · Production Checklist
D · Glossary

Book details