← All Base Form ML projects
BL-19NOT STARTEDBASE FORM · DEEP LEARNINGMLPYTHON

Game-playing agent

Teach an agent to balance a pole, then to play a Snake game you wrote, with Q-learning and DQN.

EST.8 h
REWARD+100 XP
AFTERMnih et al.

ML-ARENA BENCHMARKS

YOUR NUMBER VS THE INDUSTRY'S
METRICINDUSTRYMINE
average return on CartPole-v1
GOAL · ≥ 475 over 100 episodes
Gymnasium solved threshold: 475—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

In reinforcement learning (RL) nobody tells the agent the right answer. It acts in an environment, gets rewards, and has to work out which earlier actions earned them. You'll start with CartPole, where a cart must keep a pole balanced upright, and finish with an agent that plays a Snake game you wrote yourself.

The central idea is the action value Q(s,a)Q(s, a): the total discounted reward the agent expects if it takes action aa in state ss and acts well afterwards. Q-learning improves that estimate after every step:

Q(s,a)←Q(s,a)+α[r+γmax⁡a′Q(s′,a′)−Q(s,a)]Q(s, a) \leftarrow Q(s, a) + \alpha \big[ r + \gamma \max_{a'} Q(s', a') - Q(s, a) \big]

Here α\alpha is the learning rate and γ\gamma, the discount factor, says how much future reward counts. A table of Q-values works when there are few states. CartPole's state is four real numbers, so you'll first chop them into bins, then replace the table with an MLP.

The MLP version is called DQN. Two tricks keep it stable. A replay buffer stores past steps and trains on random batches of them. A target network, a lagging copy of the Q-network, computes r+γmax⁡a′Q(s′,a′)r + \gamma \max_{a'} Q(s', a') so the targets don't shift with every update. Exploration comes from epsilon-greedy: take a random action with probability ε\varepsilon, and shrink ε\varepsilon over training.

Install Gymnasium with pip install gymnasium and create the environment with gymnasium.make("CartPole-v1"). Each step the pole stays up earns a reward of 1, and an episode stops after 500 steps, so 500 is a perfect score. Gymnasium counts an average of 475 as solved.

OVERVIEWFig. 1
ENVIRONMENT CartPole or Snake reset(), step(a) AGENT Q-network epsilon-greedy over Q(s, ·) state s action a REPLAY BUFFER (s, a, r, s′, terminated) every step is stored random batch LOSS Huber(Q, target) update TARGET NETWORK r + γ max Q′(s′, ·) copy weights every N steps
The DQN loop. The agent acts, every step goes into the replay buffer, and random batches from the buffer train the Q-network against targets from the slower target network. Why it's here: Objective 3 has four moving parts. This shows how data and weights move between them.
HINT · OBJECTIVE 3
Stuck? Hints stay hidden until you ask.
HINT · OBJECTIVE 4
Stuck? Hints stay hidden until you ask.

DATA · Gymnasium

Objectives

0 / 6 CLEARED

Read-only. Log in to update progress.

VICTORY CONDITIONS

  1. Mean return of at least 475 over 100 greedy episodes of CartPole-v1, recorded with `python tools/mlarena.py record BL-19 "<mean return>"`.
  2. Learning curves from 3 seeds, so the result isn't one lucky run.
  3. Your Snake agent's mean score over 100 games beats both the random agent and the steps-towards-the-food rule.

Side quests

Switch to Double DQN, where the online network picks the next action and the target network scores it, and compare curves.
Solve CartPole with REINFORCE, a policy-gradient method with no Q-values at all.
Save a GIF of your best Snake game.

Terms in plain words

Return
The total reward from one episode. With discounting, rewards further in the future count for a little less.
Q-value
The return an agent expects if it takes a given action in a given state and plays well from then on.
Epsilon-greedy
Taking a random action with a small probability epsilon and the best-looking action otherwise, so the agent keeps exploring.
Replay buffer
A store of past steps that the agent trains on in random batches, which breaks up runs of very similar steps.
Target network
A copy of the Q-network that is updated only now and then and is used to compute training targets, which keeps them from chasing themselves.