← All Base Form ML projects
BL-18NOT STARTEDBASE FORM · DEEP LEARNINGMLPYTHON

Sprite generator

Train a small VAE, then a tiny diffusion model, that draw new pixel-art sprites.

EST.10 h
REWARD+100 XP
AFTERoriginal

ML-ARENA BENCHMARKS

YOUR NUMBER VS THE INDUSTRY'S
METRICINDUSTRYMINE
reconstruction loss; sample grid
GOAL · within 5%
PyTorch reference VAE—
no results yet
YOUR RESULTS

Your recorded results appear here.

Brief

A generative model learns what its training data looks like well enough to make new examples. You'll build two kinds, each small enough to train on a laptop, and use them to draw pixel-art sprites.

A variational autoencoder (VAE) has an encoder that maps an image to a distribution over a short latent code zz, and a decoder that maps a code back to an image. The encoder outputs a mean μ\mu and a log-variance log⁡σ2\log \sigma^2. To keep sampling differentiable, you draw z=μ+σ⊙ϵz = \mu + \sigma \odot \epsilon with ϵ∼N(0,I)\epsilon \sim \mathcal{N}(0, I). This is the reparameterisation trick. The loss is reconstruction error plus a KL term that keeps the codes close to N(0,I)\mathcal{N}(0, I), so that afterwards you can sample a fresh code and decode a new sprite.

A diffusion model takes another route. It adds Gaussian noise to a training image until it's pure noise, and trains a network to predict the noise that was added. A closed form lets you jump straight to any step tt: xt=αˉt x0+1−αˉt ϵx_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1 - \bar\alpha_t}\, \epsilon, where αˉt\bar\alpha_t shrinks from 1 towards 0 as tt grows. To generate, you start from noise and remove a little at each step. It's slower to sample than a VAE, but the pictures are usually sharper.

For data, use a free sprite pack. Kenney (kenney.nl) publishes CC0 packs such as Tiny Dungeon and 1-Bit Pack, and OpenGameArt.org hosts the Liberated Pixel Cup (LPC) character sprites, which need attribution. Slice the sheets into 16 × 16 or 32 × 32 tiles. A few hundred tiles is a small set, so stretch it with horizontal flips and colour shifts.

VISUALFig. 1
sprite x Encoder μ log σ² z = μ + σ ⊙ ε ε ~ N(0, I) Decoder reconstruction KL( N(μ, σ²) ‖ N(0, I) ) keeps codes near N(0, I) reconstruction loss: compare x with its reconstruction
A VAE. The encoder outputs a mean and a log-variance, the code is sampled with outside noise, and the loss has two parts: reconstruction and KL. Why it's here: Objective 3 adds three pieces to your autoencoder at once. This shows where each one goes.
HINT · OBJECTIVE 3
Stuck? Hints stay hidden until you ask.
HINT · OBJECTIVE 5
Stuck? Hints stay hidden until you ask.

DATA · a public sprite set

Objectives

0 / 6 CLEARED

Read-only. Log in to update progress.

VICTORY CONDITIONS

  1. Your VAE's validation reconstruction loss is within 5% of the reference VAE, or lower, recorded with `python tools/mlarena.py record BL-18 "<yours> vs <reference>"`.
  2. The README shows an 8 × 8 grid of VAE samples and an 8 × 8 grid of diffusion samples.
  3. The copy check shows new sprites, not near-copies of training tiles.

Side quests

Pick two sprites, encode them, and decode 8 codes on the straight line between them.
Condition the diffusion model on a label, such as the sprite's category, and generate each one on demand.

Terms in plain words

Generative model
A model that learns what its training data looks like well enough to produce new examples of it.
Latent code
The short list of numbers a VAE squeezes each image into. Decoding a new code gives a new image.
Reparameterisation trick
Writing a random sample as mean plus spread times fixed noise, so gradients can flow through the mean and the spread.
KL divergence
A measure of how different two probability distributions are. In a VAE it keeps the codes close to a standard normal.
Diffusion model
A model trained to remove a little noise at a time. Starting from pure noise and repeating that step many times produces an image.