Brief
In reinforcement learning (RL) nobody tells the agent the right answer. It acts in an environment, gets rewards, and has to work out which earlier actions earned them. You'll start with CartPole, where a cart must keep a pole balanced upright, and finish with an agent that plays a Snake game you wrote yourself.
The central idea is the action value : the total discounted reward the agent expects if it takes action in state and acts well afterwards. Q-learning improves that estimate after every step:
Here is the learning rate and , the discount factor, says how much future reward counts. A table of Q-values works when there are few states. CartPole's state is four real numbers, so you'll first chop them into bins, then replace the table with an MLP.
The MLP version is called DQN. Two tricks keep it stable. A replay buffer stores past steps and trains on random batches of them. A target network, a lagging copy of the Q-network, computes so the targets don't shift with every update. Exploration comes from epsilon-greedy: take a random action with probability , and shrink over training.
Install Gymnasium with pip install gymnasium and create the environment with gymnasium.make("CartPole-v1").
Each step the pole stays up earns a reward of 1, and an episode stops after 500 steps, so 500 is a perfect score.
Gymnasium counts an average of 475 as solved.
DATA · Gymnasium
Objectives
Read-only. Log in to update progress.
VICTORY CONDITIONS
- Mean return of at least 475 over 100 greedy episodes of CartPole-v1, recorded with `python tools/mlarena.py record BL-19 "<mean return>"`.
- Learning curves from 3 seeds, so the result isn't one lucky run.
- Your Snake agent's mean score over 100 games beats both the random agent and the steps-towards-the-food rule.
Side quests
Terms in plain words
- Return
- The total reward from one episode. With discounting, rewards further in the future count for a little less.
- Q-value
- The return an agent expects if it takes a given action in a given state and plays well from then on.
- Epsilon-greedy
- Taking a random action with a small probability epsilon and the best-looking action otherwise, so the agent keeps exploring.
- Replay buffer
- A store of past steps that the agent trains on in random batches, which breaks up runs of very similar steps.
- Target network
- A copy of the Q-network that is updated only now and then and is used to compute training targets, which keeps them from chasing themselves.